When is leave-one-out cross-validation worth its n model fits instead of 5-fold?

## What leave-one-out is Leave-one-out cross-validation (LOOCV) is the extreme case of k-fold with k set to `n`, the number of rows. Row 1 is held out, a model is trained on rows 2..n and predicts row 1; row 2 is held out, a model is trained on the rest and predicts row 2; and so on for all `n` rows. You end with one out-of-fold prediction per row and no choices to make: there is exactly one possible partition, so no shuffle, no seed, and no run-to-run variation from the split. ## The case for it **Maximum training data.** Each model sees `n-1` rows. Compared with 5-fold's 80%, that all but eliminates the pessimistic bias caused by training on a fraction of the data. On a 120-patient rare-disease biomarker cohort this is the whole argument: 5-fold trains each model on 96 patients, and a flexible model on 96 high-dimensional rows is meaningfully worse than the same model on 119. When the learning curve is steep and the dataset cannot be grown, you want every row you can get into the training set. **Determinism.** No partition to draw means no partition noise. Two people running LOOCV on the same table get the same number, which removes a whole category of argument on small datasets where 5-fold results wander. **Every row gets tested.** With `n` small, a 5-fold test fold might contain 24 rows and the rare event you care about might miss some folds entirely. LOOCV guarantees each row contributes one out-of-fold prediction. ## The case against it **Compute.** `n` fits. On a 50,000-row taxi trip-duration model at 40 minutes per fit, 10-fold costs about seven hours, which is already a real scheduling decision; LOOCV costs 50,000 fits, roughly a few years of serial compute. There is no cleverness that rescues this — the answer is simply that LOOCV is a small-`n` scheme. **Correlated fits, jumpy estimate.** Any two LOOCV training sets differ by two rows out of `n`. The `n` models are therefore near-identical and their errors move together, so averaging `n` of them cancels far less noise than averaging `n` independent measurements would. The textbook summary is "nearly unbiased but high variance". Treat it as a strong tendency rather than a law — for a very stable procedure the effect is mild, and the ordering against 10-fold is not guaranteed in every study — but the mechanism is real and worth naming. **Sensitivity to single rows.** For some procedures LOOCV behaves badly in a way that is easy to state. Consider a balanced two-class dataset and a classifier that predicts the majority class of its training set. Removing one row always tips the training majority toward the *other* class, so every held-out row is predicted wrong and LOOCV reports 0% accuracy where the true rate is 50%. **Single-row test folds.** A fold of one row can produce accuracy of 0 or 1 and nothing else, and metrics that need a ranking over multiple rows — anything built from a curve over thresholds — cannot be computed per fold at all. With LOOCV you must pool all `n` out-of-fold predictions into one set and compute the metric once over that set, which also means there are no per-fold scores to look at for spread. ## The escape hatch: closed-form LOOCV For a few procedures the `n` fits are not needed, because the leave-one-out error has an algebraic shortcut. Ordinary least squares is the standard example: fit once on all rows, and the leave-one-out residual for row i equals the ordinary residual divided by `1 - h_ii`, where `h_ii` is that row's leverage — its diagonal entry in the hat matrix. Summing the squares of those adjusted residuals gives the LOOCV squared error, known as the PRESS statistic, from a single fit. Ridge-penalised least squares has a comparable identity. When such a shortcut applies, the cost objection disappears and LOOCV becomes attractive at sizes where it would otherwise be unthinkable. Say this if it applies to the procedure being discussed; do not claim it for arbitrary models, because for a boosted tree ensemble or anything trained iteratively there is no such identity. ## How to answer the decision Estimate the two numbers out loud. One: how long does a single fit take, multiplied by `n`. Two: how far apart are `n` and `n*(k-1)/k` on the learning curve — is training on 96 rows versus 119 a real difference, or is it 40,000 versus 45,000 where nothing moves? LOOCV wins only when the second gap is material and the first product is small. In most production settings, neither holds, and 5- or 10-fold is the right default.