When would you choose isotonic regression over Platt scaling to recalibrate a model's scores?

## The shared setup Recalibration takes an existing model as given and learns a second, one-dimensional function that maps its raw score `s` to a probability. The model is never retrained. Two properties follow immediately: the map is fitted on **labelled data the model did not train on**, and the map is **monotone**, so the ranking the model produced survives. The input `s` need not be a probability. An SVM decision value is a signed distance from the separating boundary and can be any real number; recalibration is precisely how such a score is turned into something a business can consume. This is the setting Platt scaling was introduced for. ## Platt scaling Fit a logistic curve with two parameters to the held-out scores and their labels: `p(s) = 1 / (1 + exp(a*s + b))` with `a` and `b` chosen by maximum likelihood on `(score, label)` pairs. Because a is typically negative, the curve rises with the score. **Strengths.** Two parameters is almost nothing to estimate, so the fit is stable on a few hundred rows. It is defined over the whole real line, so it extrapolates naturally beyond the scores it saw. It is smooth and strictly increasing, which means no ties are introduced and AUC is preserved exactly. **Weakness.** It can only produce a sigmoid. If the true distortion is not sigmoid-shaped - if the reliability curve wiggles, or is flat in the middle and steep at both ends - Platt scaling will straighten part of the curve and leave the rest bent. It is a strong assumption dressed up as a convenience. ## Isotonic regression Fit the non-decreasing function `g` that minimises the squared error between `g(s_i)` and the 0/1 labels, subject only to `g` never decreasing. The pool-adjacent-violators algorithm computes it: sort the rows by score, and wherever a later group has a lower mean label than an earlier one, merge the two groups and replace both by their pooled mean, repeating until the sequence is non-decreasing. The result is a **step function** - a set of intervals, each with one constant probability. **Strengths.** No shape assumption. It repairs any monotone distortion, which is what you need for a bagged model whose averaging compresses every score into 0.2 to 0.8: isotonic regression simply stretches the ends back out. Given enough data it is hard to beat. **Weaknesses.** - **Variance.** The flexibility that fixes odd shapes also fits noise. Below roughly a thousand held-out rows it tends to overfit, and the correct comparison is against Platt scaling, not against doing nothing. - **Ties.** Everything inside a step gets exactly the same probability. Because AUC counts tied pairs as half a point, collapsing many distinct scores into one value can nudge AUC down slightly - monotone but not *strictly* monotone. - **No extrapolation.** Outside the score range of the calibration set the function is clamped at its end values, and the extreme steps are estimated from the fewest rows, which is exactly where decisions are often taken. - **Blocky output.** A step function gives few distinct probabilities, which can look wrong to a consumer expecting a smooth score distribution. ## Where the calibration data comes from The single most common implementation error is fitting the calibrator on the same rows the model was trained on. In-sample scores are pushed toward the correct labels by the training process itself, so the reliability curve on those rows looks far better than reality; the calibrator then learns a near-identity map and corrects nothing, or worse, corrects in the wrong direction. Calibration looks flawless in the report and fails on the first live batch. The options, in order of data cost: 1. **A dedicated calibration split** - a third slice of data, separate from train and test. Simplest and cleanest when data is plentiful. 2. **Cross-validated calibration** - split the training data into k folds; for each fold, train a model on the other k-1 and predict the held-out fold. Every training row now has an *out-of-fold* prediction produced by a model that never saw it. Pool those predictions to fit one calibrator. Then refit the model on all the data and apply that calibrator to it. This spends no data and is the usual choice when data is tight. 3. **Reusing the test split** - workable in a pinch, but the test set is then no longer an honest estimate of anything, since it has been consumed by fitting. ## Choosing, in practice Plot the reliability diagram first; it names the distortion. A clean sigmoid bend, or a calibration set in the hundreds, or a raw decision value with unbounded range: Platt scaling. A few thousand held-out rows and a curve that no sigmoid will follow, such as a compressed forest score: isotonic regression. When in doubt, fit both on the same held-out folds and compare their calibration on a further untouched slice - the comparison is cheap, and the reliability diagram of each makes the winner obvious.