How does split conformal prediction turn a trained regression model into 90% prediction intervals?

## The problem it solves A regression model returns one number. A decision-maker usually needs a range plus a promise about how often the truth falls inside it. Split conformal prediction produces that range from any point predictor at all — a linear model, a gradient-boosted ensemble, a nearest-neighbour rule — without assuming the errors are Gaussian, symmetric, or homoscedastic, and without retraining anything. ## The recipe 1. **Split.** Partition the labelled data into a *proper training set* and a *calibration set* of size `n`. The calibration rows must never touch model fitting. 2. **Fit.** Train the model `f` on the proper training set alone. 3. **Score.** For each calibration row compute a **nonconformity score** — a number that is large when the model was surprised. For regression the default is the absolute residual `s_i = |y_i - f(x_i)|`. 4. **Quantile.** Sort the scores ascending and take `q = s_(k)` where `k = ceil((n+1)(1-alpha))`. 5. **Predict.** For a new input `x`, output `[f(x) - q, f(x) + q]`. With `n = 500` and `alpha = 0.10`, `k = ceil(501 * 0.9) = ceil(450.9) = 451`, so `q` is the 451st smallest of the 500 absolute residuals — not the 450th. That `+1` is the finite-sample correction, and it matters most when `n` is small. ## Why it works The argument is a rank argument, not a distributional one. Imagine the test row's own nonconformity score `s_new` dropped into the pile of `n` calibration scores. If all `n+1` scores are **exchangeable** — their joint distribution is unchanged by any reordering — then `s_new` is equally likely to occupy any of the `n+1` positions. The interval fails exactly when `s_new` exceeds `q`, and the probability of landing above the `k`-th of `n+1` positions is at most `alpha` by construction. Nothing in that reasoning needed a normality assumption, an asymptotic argument, or a correctly specified model. Exchangeability is weaker than independent and identically distributed: iid implies exchangeable, but not the reverse. It is the only assumption in play, and it is the one that breaks in practice. ## What the guarantee does and does not say - **Distribution-free and finite-sample.** Valid at `n = 50`, not just asymptotically. - **Model-agnostic.** A badly fitted model still yields *valid* intervals — they simply become wide. Interval width is therefore a clean read on model quality, while coverage is not. - **Marginal.** The probability is averaged over the random draw of the calibration set *and* the random test point. It is not a per-input statement, and it is not automatically true within a subgroup. - **Slightly conservative.** With continuous scores and no ties, coverage sits between `1-alpha` and `1-alpha + 1/(n+1)`. - **Random in realisation.** Once you fix one calibration set, the coverage you actually experience fluctuates around `1-alpha`; the spread shrinks as `n` grows, which is the practical reason to prefer several hundred calibration rows over a few dozen. The mechanics only *require* `n >= 9` at `alpha = 0.10`, since otherwise `ceil((n+1)(1-alpha))` exceeds `n` and no finite quantile exists — that degenerate case corresponds to an infinite-width interval. ## The score is the design decision Everything interesting lives in the choice of nonconformity score. The absolute residual gives one global `q`, so every interval has the identical width `2q` — cheap, valid, and blind to the fact that some inputs are far harder to predict than others. A **normalized** score `|y - f(x)| / sigma(x)`, where `sigma(x)` is a second model predicting the typical error magnitude at `x`, restores width that varies with the input. **Conformalized quantile regression** goes further by conformalizing a predicted band rather than a point. Swapping the score never breaks validity — only the usefulness of the interval changes. ## Variants worth naming **Full (transductive) conformal** refits the model for every candidate value of the new label, which recovers the data spent on calibration but multiplies compute by the size of the candidate grid. **Jackknife+** and **CV+** reuse all the data through leave-one-out or fold-wise residuals, buying back sample efficiency at the cost of a weaker worst-case guarantee of `1 - 2*alpha`. Split conformal stays the default in practice because it costs exactly one extra model evaluation per calibration row. ## Common way to get it wrong Computing the residual quantile on the training data. In-sample residuals are optimistically small for any model with capacity, so the interval comes out too narrow and the guarantee evaporates — the rank argument requires that the calibration rows were never seen during fitting.