How do you check whether a classifier's predicted probabilities are calibrated?

## What calibration actually claims A binary classifier gives each row a score between 0 and 1. The model is **calibrated** if the score can be read as a probability: among all the rows scored near 0.30, about 30% really turn out positive; among the rows scored near 0.70, about 70% do. Notice what is *not* being claimed. Nothing about accuracy, nothing about which rows are ranked above which. Calibration is a property of the *numbers*; discrimination is a property of the *ordering*. A model can be excellent at one and hopeless at the other. This matters the moment the score is consumed as a number rather than as a rank. A subscription-churn model whose 0.30 bucket must really churn 30% of the time before finance sizes a retention offer is the standard case: if the true churn rate in that bucket is 12%, every expected-value calculation built on the score - offer budget, expected saved revenue, expected number of churners next month - is wrong by more than a factor of two, even though the model may rank customers beautifully. ## The reliability diagram The diagnostic is mechanical: 1. Take predictions on data the model was **not** trained on. 2. Split the score range into bins. *Equal-width* bins cut [0, 1] into, say, 10 slices of 0.1. *Equal-frequency* bins put the same number of rows in each. 3. In every bin compute the mean predicted score (the x coordinate) and the observed positive rate (the y coordinate). 4. Plot the points, add the 45-degree diagonal, and plot a histogram of bin counts underneath. Reading it: - **On the diagonal** - calibrated in that region. - **Below the diagonal** - observed rate is lower than predicted; the model is **overconfident** there, promising more positives than materialise. - **Above the diagonal** - observed rate exceeds predicted; the model is **underconfident**, understating the risk. - **A curve steeper than the diagonal, crossing it in the middle** - the classic signature of scores squeezed toward 0.5. A model that averages many weak votes, such as a forest of trees, often never emits a score below 0.2 or above 0.8, so at the low end it overshoots and at the high end it undershoots. - **A flatter S, hugging the extremes** - scores pushed toward 0 and 1; predictions of 0.97 that come true only 70% of the time. ## Expected calibration error The diagram condensed to one number: `ECE = sum over bins b of (n_b / N) * | mean_predicted_b - observed_rate_b |` That is the average vertical gap in the diagram, weighted by how many rows fall in each bin. Zero is perfect. It is useful and it is fragile, for reasons worth knowing: - **It depends on the binning.** The same model scored over 10 equal-width bins and over 50 bins gives different numbers, usually a larger one with more bins, because fine bins stop opposite-signed errors from cancelling inside a wide bin. Only compare ECE across models computed with identical bins on identical data, and report the bin count alongside the number. - **Equal-width binning collapses under a low base rate.** On an insurance-claim propensity score where almost every row lands under 0.1, nine of ten equal-width bins may hold a handful of rows each; their observed rates are pure noise, and the weighting mercifully shrinks their influence but does not make the picture readable. Equal-frequency bins fix the counts at the cost of uneven widths. - **It throws away direction.** The absolute value means a model that is overconfident at the top and underconfident at the bottom can score the same as one uniformly off in one direction, though the fixes differ. Keep the diagram. - **It is an estimate.** With a few thousand rows the per-bin observed rates carry real sampling error; do not read a difference of 0.002 as an improvement. ## Why AUC cannot answer this question ROC AUC is the probability that a random positive scores above a random negative. Apply any strictly increasing transform to every score - halve them, square them, push them through a sigmoid - and every pairwise comparison is preserved, so AUC is *exactly* unchanged while the probabilities become arbitrary. This is why a churn model can sit at AUC 0.82 and still be unusable for pricing, and equally why recalibrating a model is a safe operation for its ranking: a monotone reshaping cannot damage the ordering. ## Practical hygiene Always evaluate calibration on held-out data, at the class prevalence the model will actually meet in production. Show the counts per bin. Look at the region you actually act on - if decisions are made above 0.8, the calibration of the 0.1 bucket is a curiosity. And treat a bad diagram as diagnosis, not verdict: the shape tells you which recalibration method is likely to fix it.