A penalised model's CV-error curve is flat across a whole decade of lambda — why is the lambda with the lowest mean error a poor pick?

## What flatness actually means A CV-error-versus-lambda curve is a hundred-ish noisy estimates plotted side by side. Each is the mean of the per-fold scores at one lambda, and each carries a standard error that tells you how much that mean would wobble under a different partition of the rows. A **flat** stretch means the vertical spread of the means across that stretch is comparable to, or smaller than, the error bars attached to them. This is common, not exotic. It happens when predictors are correlated, so extra shrinkage merely redistributes weight among near-duplicates without changing predictions much; when there is a broad range of lambda over which the effective complexity of the fit changes slowly; and whenever the dataset is large enough that the fit is not very sensitive to modest shrinkage. ## Two separate problems, often conflated **Instability of the selection.** If the differences you are ranking on are smaller than the noise in the ranking, the ordering is arbitrary. The concrete symptom: re-run cross-validation with a different fold assignment and record where the minimum lands each time. On a flat curve it wanders across a decade. Nothing about your data changed; only the noise did. Reporting `optimal lambda = 0.0137` after that is false precision. **Optimism of the reported minimum.** Even setting stability aside, the height at the argmin is a biased estimate of that lambda's true error. Suppose every grid point had exactly the same true error; each estimate would still land above or below it by chance, and by construction you are reading off the one that landed lowest. The minimum of many noisy estimates is systematically below the truth — you have selected on the noise, and the number you quote contains that selection. It is a self-scored result: the same fold errors both chose the winner and reported its score. One nuance worth having ready: the grid points are not independent estimates. Every lambda is scored on the same folds, so a fold split that is easy for one lambda tends to be easy for all of them, and the curve shifts up or down almost as a whole. That correlation damps the optimism relative to a hundred independent draws — but it does not remove it, because the *shape* of the curve still jitters and the argmin still follows the jitter. ## How to diagnose it in a few minutes - Compare the depth of the dip with the height of the bars. If the minimum sits well inside its neighbours' bars, you have a plateau, not an optimum. - Re-run the whole tuning sweep with two or three different fold assignments and tabulate the argmin from each. If they disagree by a factor of ten, say so out loud rather than picking one. - Do the same for the one-standard-error selection. It typically moves by one grid point or not at all, which is the point of using it. ## What flatness does not mean It does not mean the penalty is doing nothing. The plateau is the interior of the curve; the ends still tell a story — error climbs as lambda goes large enough to crush the model toward a constant prediction, and usually climbs on the small-lambda side too. Nor does it mean your grid is too coarse: refining a grid inside a plateau resolves noise, not signal, and gives you a more precisely stated arbitrary answer. ## What to do instead Stop asking the noise to choose. Convert the plateau into an explicit tie set — every lambda whose mean is within one standard error of the best — and break the tie on a criterion that does not depend on the fold split. Parsimony is the usual criterion, and on a lambda axis that means the largest lambda in the set. Three things improve at once: the selection is reproducible across re-splits, the deployed model is simpler, and you can state the rule you used before running the sweep rather than after seeing the plot. When you report, report the reading rather than a single decimal: the selected lambda, the range of lambda that was indistinguishable from the best, and the fact that the choice inside that range was made on simplicity. A reviewer who sees the band understands the model; one who sees only `lambda = 0.0137` has to take your word for it.