---
title: "Optimal Binning and Weight of Evidence: A Practical Guide"
author: "José Evandeilton Lopes"
date: "`r Sys.Date()`"
output:
  rmarkdown::html_vignette:
    toc: true
    toc_depth: 3
vignette: >
  %\VignetteIndexEntry{Optimal Binning and Weight of Evidence: A Practical Guide}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.width = 7.5,
  fig.height = 5,
  fig.align = "center",
  warning = FALSE,
  message = FALSE
)
options(width = 100, digits = 4)
```

This vignette is the working reference for analysts who bin variables for a
living. It covers what the package computes, how to read what it returns, and
how to decide what to keep — using a benchmark dataset that ships with the
package, so every result below is reproducible with no extra downloads.

The companion vignette,
[An Industrial Scorecard Pipeline](industrial-pipeline.html), takes the same
machinery to a wide production base with out-of-time validation, a `recipes`
pipeline and deployment artefacts.

```{r library}
library(OptimalBinningWoE)
```

# Why bin

A scorecard needs predictors that a credit committee, a regulator and a
production database can all agree on. Binning buys three things a raw
continuous variable does not give you:

- **A bounded, monotone transformation.** Outliers and skew stop mattering; the
  relationship with risk becomes a step function you can plot on one page.
- **Missing values as first-class citizens.** A missing income is a level, not
  an imputation problem.
- **A linear-in-the-logit predictor.** After the WoE transform, logistic
  regression coefficients are directly interpretable and the model is a table.

The cost is resolution: binning discards within-bin variation. The whole craft
is choosing cut points that pay for that loss.

# The quantities

## Weight of Evidence

For bin $i$ with $n_{i,1}$ events and $n_{i,0}$ non-events, out of population
totals $N_1$ and $N_0$:

$$
\mathrm{WoE}_i \;=\; \ln\!\left(\frac{n_{i,1}/N_1}{n_{i,0}/N_0}\right)
\;=\; \ln \theta_i + \ln\frac{N_0}{N_1},
\qquad \theta_i = \frac{n_{i,1}}{n_{i,0}}
$$

WoE is the bin log-odds shifted by a constant. Positive means riskier than the
portfolio average, negative means safer, and the scale is comparable across
variables — which is why WoE, not the raw category, is what enters the model.

## Information Value

$$
\mathrm{IV} \;=\; \sum_{i=1}^{k}
\left(\frac{n_{i,1}}{N_1} - \frac{n_{i,0}}{N_0}\right) \mathrm{WoE}_i
$$

This is the symmetrised Kullback–Leibler divergence (Jeffreys divergence)
between the event and non-event distributions across bins. It is the standard
single-number summary of predictive strength, and the package grades it with
the bands from Siddiqi (2006):

| IV | Band | Reading |
|---|---|---|
| $< 0.02$ | Unpredictive | drop |
| $[0.02,\ 0.10)$ | Weak | keep only if it adds diversity |
| $[0.10,\ 0.30)$ | Medium | the workhorses of a scorecard |
| $[0.30,\ 0.50)$ | Strong | strong, verify it is not a proxy |
| $\ge 0.50$ | Suspicious | almost always leakage |

IV never decreases when you split a bin, so it cannot be used on its own to
choose the number of bins — that is what the optimisation algorithms are for.

## Monotonicity

Both quantities a scorecard cares about are strictly increasing functions of
the bin odds $\theta_i$:

$$
\pi_i = \frac{n_{i,1}}{n_{i,1}+n_{i,0}} = \frac{\theta_i}{1+\theta_i},
\qquad
\mathrm{WoE}_i = \ln\theta_i + \ln\frac{N_0}{N_1}
$$

So "the event rate is monotone across bins" and "the WoE is monotone across
bins" are the same statement, and checking either one suffices. For a numerical
variable this is a genuine constraint — the bin order is the interval order.
For a nominal variable it is not: the bins can always be relabelled in WoE
order, so monotonicity there is free.

# The data

The package bundles the Statlog (German Credit) benchmark: 1000 loan
applications, 7 numerical and 13 categorical attributes, from the UCI Machine
Learning Repository.

```{r data}
german <- read.csv(
  gzfile(system.file("extdata", "germancredit.csv.gz",
                     package = "OptimalBinningWoE")),
  stringsAsFactors = FALSE
)

# credit_risk is 1 for a good customer; the event we model is default
german$default <- 1L - german$credit_risk
german$credit_risk <- NULL

dim(german)
table(german$default)
```

```{r data-glimpse}
str(german[, c("duration", "amount", "age", "purpose", "savings")])
```

# One variable, end to end

`obwoe()` is the single entry point. Point it at a data frame and a target and
it detects each variable's type, dispatches the right algorithm and returns a
fitted binning.

```{r single-fit}
fit <- obwoe(german, target = "default", feature = "duration",
             min_bins = 3, max_bins = 6)
fit
```

The per-variable result carries everything the transformation needs.

```{r single-result}
res <- fit$results$duration
data.frame(
  bin   = res$bin,
  count = res$count,
  pos   = res$count_pos,
  rate  = round(res$count_pos / res$count, 4),
  woe   = round(res$woe, 4),
  iv    = round(res$iv, 4)
)
```

Two things to read here. The event rate climbs from 10% to 54% across the
bins — longer loans default more, and the ordering never reverses. And the IV
column shows where the information sits: the two extreme bins carry 73% of it
between them, while the largest bin, holding 40% of the portfolio, contributes
almost nothing.

`cutpoints` is the authoritative boundary vector; bin labels are for humans.

```{r single-cutpoints}
res$cutpoints
```

Intervals are half-open on the right, $(a,\,b]$, throughout the package: a loan
of exactly 10 months falls in the bin that ends at 10.

```{r single-plot, fig.height=4.5}
plot(fit, type = "woe", feature = "duration")
```

# Reading the gains table

`obwoe_gains()` turns a fitted binning into the table a credit analyst actually
reviews: cumulative capture, KS, lift.

```{r gains}
gains <- obwoe_gains(fit, feature = "duration", sort_by = "woe")
gains
```

The three headline numbers:

- **KS** is the largest gap between the cumulative event and non-event
  distributions, $\max_i |F_1(i) - F_0(i)|$. It answers "at the best cut, how
  much better than random is this variable at separating the two populations".
- **Gini** is $2\,\mathrm{AUC}-1$ on the binned score.
- **Lift** in a bin is its event rate over the portfolio rate — the operational
  number for a cutoff policy.

```{r gains-plot, fig.height=6}
op <- par(mfrow = c(2, 2), mar = c(4, 4, 2, 1))
plot(gains, type = "cumulative")
plot(gains, type = "ks")
plot(gains, type = "lift")
plot(gains, type = "woe_iv")
par(op)
```

# All variables at once

Drop the `feature` argument and every column is binned. Numerical and
categorical variables are routed automatically.

```{r multi-fit}
model <- obwoe(german, target = "default", min_bins = 2, max_bins = 6)
model
```

```{r multi-summary}
summary(model)
```

## Screening with `obwoe_select()`

`summary()` ranks; `obwoe_select()` decides. It applies the two criteria that
govern admission in practice — IV strength and guaranteed ordering — and
returns a verdict per variable, with the reason attached.

```{r select}
sel <- obwoe_select(model)
head(sel[, c("feature", "type", "n_bins", "total_iv", "iv_class",
             "ks", "gini", "monotonic", "quality", "selected")], 10)
```

Nothing is ever dropped from the output. A base with 500 candidates returns 500
rows, so the automatic verdict can be reviewed rather than trusted.

```{r select-rejected}
sel[!sel$selected, c("feature", "total_iv", "iv_class", "reason")]
```

`status` — the checking-account balance — is the strongest variable in the file
and is rejected for it. An IV of 0.67 sits in the *Suspicious* band, where
single-variable strength is far more often a symptom of target leakage than of
a genuinely dominant predictor. Raise `iv_max` to admit it deliberately:

```{r select-admit}
admitted <- obwoe_select(model, iv_max = Inf)
admitted[admitted$feature == "status",
         c("feature", "total_iv", "quality", "selected", "reason")]
```

The policy is configurable end to end. Each rule that fires is appended to
`reason`, so a variable rejected on two counts reports both.

```{r select-policy}
strict <- obwoe_select(
  model,
  iv_min            = 0.02,      # drop the Unpredictive band
  iv_max            = 0.50,      # drop the Suspicious band
  require_monotonic = "numeric", # ordering is intrinsic only for numerics
  monotonicity      = "strict",  # no ties between adjacent bins
  min_bin_pct       = 0.05,      # every bin holds at least 5% of the base
  allow_degenerate  = FALSE,     # no bin without events or without non-events
  top_n             = 8,
  sort_by           = "ks"
)
table(strict$reason)
```

## The bin-level view

`detail = "full"` returns one row per variable *and* optimised bin, joined to
the complete gains table of each. This is the object to hand to a model
validation team.

```{r select-full}
detail <- obwoe_select(model, detail = "full")
dim(detail)

detail[detail$feature == "savings",
       c("bin", "n_categories", "count", "pos_rate", "woe", "iv", "lift")]
```

# Choosing an algorithm

The package ships 37 algorithms: 21 numerical, 16 categorical. They differ in
what they optimise, not in what they return.

| Family | Algorithms | Optimises |
|---|---|---|
| Information-theoretic | `mdlp`, `fast_mdlp`, `dmiv`, `ivb` | entropy or IV gain per split, with an MDL stopping rule |
| Statistical merging | `cm`, `fetb`, `mob` | merges neighbours whose difference fails a $\chi^2$ or Fisher test |
| Shape-constrained | `ir`, `mrblp`, `mblp`, `oslp` | best fit subject to a monotonicity constraint |
| Exact optimisation | `dp`, `milp`, `sblp`, `bb` | global optimum of IV under bin-count and size constraints |
| Metaheuristic | `sab`, `gmb`, `mba` | simulated annealing, greedy or agglomerative search |
| Unsupervised | `ewb`, `kmb`, `ubsd`, `sketch` | equal width, k-means, standard deviation, streaming quantiles |

Three questions settle the choice in practice.

**Does the model face a regulator?** Then the WoE profile must be monotone, and
a shape-constrained method (`ir`, `mrblp`, `mblp`) or `mob` earns its keep.
`ir` runs the Pool Adjacent Violators algorithm and merges the bins it pools,
so the bins it returns are monotone in their own observed event rate.

**Is the relationship expected to be non-monotone?** Utilisation and age often
are U-shaped. Use an information-theoretic method (`mdlp`, `dmiv`) or `jedi`
and let the data pick the shape.

**How large is the base?** `sketch` is designed for streaming and very large
$n$; the exact optimisers (`milp`, `bb`) are the most expensive and are best
kept for a shortlist.

```{r algo-list}
algos <- obwoe_algorithms()
table(numerical = algos$numerical, categorical = algos$categorical)
```

A comparison run costs a few lines. Here five methods on `amount`, a variable
with no clean monotone relationship:

```{r algo-compare}
compare <- function(alg) {
  f <- obwoe(german, target = "default", feature = "amount",
             algorithm = alg, min_bins = 2, max_bins = 6)
  s <- obwoe_select(f, require_monotonic = "none")
  data.frame(algorithm = alg, n_bins = s$n_bins, iv = round(s$total_iv, 4),
             ks = round(s$ks, 4), monotonic = s$monotonic)
}

do.call(rbind, lapply(c("jedi", "mdlp", "mob", "ir", "dp"), compare))
```

`amount` is a hard case, and the spread is the point. `mdlp` and `dp` find a
non-trivial partition worth `IV` ≈ 0.11; `jedi` and `mob` stop at two bins and
find almost nothing. `ir` lands in between at three bins: enforcing
monotonicity on a variable whose empirical rate violates it can only be done by
pooling, so the shape constraint is paid for in resolution. Running four or
five algorithms on the awkward variables and reading this table is cheap and
usually decisive.

# Applying the transformation

`obwoe_apply()` scores a data frame with a fitted binning. It is the function
to use on a validation sample, on a scoring run, or on next month's file.

```{r apply}
scored <- obwoe_apply(german, model, keep_original = FALSE)
head(scored[, c("default", "duration_bin", "duration_woe",
                "purpose_bin", "purpose_woe")], 4)
```

Values outside the training range fall into the extreme bins; categories never
seen during training and `NA` receive `na_woe`, which defaults to 0 — the
neutral value, since $\mathrm{WoE}=0$ means "portfolio average".

```{r apply-newdata}
two <- obwoe(german, target = "default",
             feature = c("duration", "purpose"), max_bins = 6)

new_data <- data.frame(
  duration = c(4, 10, 200, NA),
  purpose  = c("car (new)", "unseen category", "education", NA)
)
obwoe_apply(new_data, two, keep_original = TRUE)
```

A WoE-transformed frame is exactly what a logistic regression wants:

```{r apply-model}
keep <- sel$feature[sel$selected]
woe_cols <- paste0(keep, "_woe")
train <- scored[, c("default", woe_cols)]

glm_fit <- glm(default ~ ., data = train, family = binomial())
round(head(coef(summary(glm_fit)), 6), 4)
```

Coefficients on WoE predictors should all come out positive: a WoE of $+1$ means
one log-odds more risk, so a negative coefficient signals a variable fighting
the rest of the model, usually through correlation.

# Exporting to SQL

`obwoe_sql()` writes the same transformation as SQL, so the scoring can run in
the warehouse instead of in R.

```{r sql}
obwoe_sql(
  model,
  table        = "risk.applications",
  features     = c("duration", "purpose"),
  keep_columns = "application_id",
  dialect      = "postgres"
)
```

Points worth knowing: the intervals reproduce `obwoe_apply()` exactly, cut
points are written as the shortest decimal that parses back to the identical
`double`, and every expression opens with an explicit `IS NULL` branch —
because in SQL `NULL <= 5` is `NULL`, not `FALSE`, and a missing value would
otherwise fall through to `ELSE`. Fourteen dialects are supported, along with
`style = "view"`, `"cte"` and `"case"`.

```{r sql-case}
obwoe_sql(model, features = "age", style = "case", comment = FALSE)
```

# Preprocessing

`ob_preprocess()` handles the two pathologies that break binning before it
starts: missing values and extreme outliers.

```{r preprocess}
set.seed(2024)
messy <- c(rnorm(800, 5000, 2000), rep(NA, 100), runif(100, -1e4, 5e4))
y <- rbinom(1000, 1, 0.3)

prep <- ob_preprocess(
  feature         = messy,
  target          = y,
  outlier_method  = "iqr",
  outlier_process = TRUE,
  preprocess      = "both"
)

prep$report
```

The cleaned vector is in `prep$preprocess$feature_preprocessed` and goes
straight into `obwoe()`. Missing values become a sentinel that the binner keeps
as its own bin, so the information in "this field was blank" is preserved
rather than imputed away.

# Where to go next

- [An Industrial Scorecard Pipeline](industrial-pipeline.html) — a wide base,
  out-of-time validation, `recipes` and `step_obwoe()`, scorecard points,
  stability monitoring and deployment.
- `?obwoe` for the fitting interface, `?obwoe_select` for the screening rules,
  `?obwoe_sql` for the SQL contract, `?obwoe_gains` for the gains table
  definitions.

# References

Siddiqi, N. (2006). *Credit Risk Scorecards: Developing and Implementing
Intelligent Credit Scoring*. John Wiley & Sons.

Thomas, L. C., Edelman, D. B., & Crook, J. N. (2002). *Credit Scoring and Its
Applications*. SIAM.

Navas-Palencia, G. (2020). Optimal binning: mathematical programming
formulation. *arXiv:2001.08025*.

Fayyad, U. M., & Irani, K. B. (1993). Multi-interval discretization of
continuous-valued attributes for classification learning. *IJCAI*.

Hofmann, H. (1994). *Statlog (German Credit Data)*. UCI Machine Learning
Repository.

```{r session}
sessionInfo()
```