Why is a lasso fit computed down a grid of lambdas from lambda_max rather than at one lambda?

## The problem being solved You almost never know the right penalty strength before looking at the data, so a lasso fit in practice is not one optimisation but a hundred of them, one per lambda on a grid. Doing that naively — pick a hundred lambdas, solve each from a cold start — is wasteful. The standard recipe is: **start at the top of the penalty range, walk down a log-spaced grid, and warm-start every fit from the solution above it.** ## Where the grid starts: lambda_max The top of the grid is not arbitrary. There is a finite penalty `lambda_max` — the smallest penalty for which every coefficient is zero — and it is computable in closed form from the data: with the objective ``` (1 / 2n) * sum_i (y_i - b0 - sum_j x_ij * b_j)^2 + lambda * sum_j |b_j| ``` it is the largest absolute inner product between a (standardised) predictor column and the centred response, divided by n. Above that penalty, no predictor removes enough squared error to pay its own `lambda * |b_j|` cost, so the optimum is the all-zero vector. That gives you the first point of the path for free, exactly, with no iteration — and it also bounds the useful range, since anything above lambda_max is the same null model. ## Where it stops, and why log spacing The grid runs down to a small fraction of lambda_max — commonly a hundredth, or a thousandth when the signal is strong and there are many more rows than predictors. Going all the way to zero is usually pointless and sometimes harmful: as the penalty vanishes the fit approaches ordinary least squares, which is unstable when predictors are correlated and not even unique when predictors outnumber rows. Stopping early keeps the path in the region where the solution is well behaved. Spacing is logarithmic because that is the scale on which lambda acts. Halving lambda has roughly the same qualitative effect whether you are at 1.0 or at 0.001; an evenly spaced grid over the same range would spend nearly all its points near the top, where every coefficient is still zero, and skip the region where the active set is actually changing. ## Warm starts A warm start means initialising the optimiser for grid point k+1 at the converged coefficients of grid point k rather than at zero. This is effective because neighbouring lambdas produce nearly identical solutions — usually the same active set with slightly larger coefficients, occasionally one predictor entering. Coordinate descent, which cycles through predictors updating one at a time by soft-thresholding, then needs a few passes instead of many, and the zero coefficients it inherits let it skip most of the inactive predictors entirely. The key correctness point, and a favourite interview probe: **warm starting affects speed, not the answer.** The lasso objective is convex, so each grid point has a single optimal value of the objective and the optimiser converges to it regardless of where it began. Starting from a good guess only shortens the trip. (This is not true of every fitting problem — for a non-convex objective the starting point can decide which optimum you land in — but the lasso is convex.) ## Why the direction matters Running the grid *upward*, from small lambda to large, throws away the advantage. The small-lambda end has no known solution, so the first fit is a cold start on the hardest, densest problem in the whole range; and the dense solution you carry upward is a poor initialisation for sparse ones. Descending, you always begin at a point whose solution you know exactly and whose active set is empty. ## What you get out of it The by-product is the coefficient path itself. Having solved the whole grid, you can plot every coefficient against lambda, see the order in which predictors enter, and hand the same grid to whatever out-of-sample procedure picks the winning penalty. In other words, the sequence of fits you needed for speed is exactly the sequence of fits you needed for tuning — which is why path-wise fitting is the default rather than an optimisation trick.