What does an L2 (ridge) penalty do to a linear regression's coefficients?

Ridge regression is ordinary least squares with a price tag attached to the size of the coefficients. ## The objective Least squares picks the weight vector `w` minimising the residual sum of squares, `RSS = sum_i (y_i - x_i . w)^2`. Ridge minimises `RSS + lambda * sum_j w_j^2` - the same data-fit term plus `lambda` times the sum of squared weights. `lambda >= 0` is a hyperparameter you choose, not something the fit learns. Two conventions travel with it: the intercept is excluded from the penalty, and the predictors are put on a common scale first, because the penalty charges coefficient magnitude and magnitude depends on the units a feature happens to be measured in. ## The closed form Unlike most penalised fits, ridge has an exact solution: `w = (X'X + lambda I)^-1 X'y` where `X` is the n-by-p feature matrix, `X'` its transpose, and `I` the p-by-p identity. Compare it with the least-squares solution `(X'X)^-1 X'y`: the only change is `lambda` added along the diagonal of `X'X`. That single change is what makes ridge well behaved numerically. `X'X` is symmetric with non-negative eigenvalues; adding `lambda I` lifts every eigenvalue by `lambda`, so the matrix is strictly positive definite and invertible for any `lambda > 0` - even when two columns carry almost identical information, and even when there are more features than rows. ## Why the shrinkage is proportional Take the singular value decomposition `X = U D V'`, with singular values `d_1, ..., d_p`. Along the direction `v_j`, ridge multiplies the least-squares coordinate by `d_j^2 / (d_j^2 + lambda)` Every such factor lies strictly between 0 and 1: nothing is left untouched and nothing is driven to zero. Directions in which the data varies a lot - large `d_j` - have a factor near 1 and survive almost unchanged. Directions the data barely pins down - small `d_j` - are crushed. That is the whole idea: ridge spends its shrinkage where the data has least to say. In the special case of an orthonormal design, where `X'X = I` and every singular value is 1, the formula collapses to `w_ridge = w_ols / (1 + lambda)`. Every weight is divided by the same constant. This is the crispest statement of what proportional means, and it sits next to the obvious arithmetic fact that dividing a nonzero number by `1 + lambda` never yields zero. ## No exact zeros, therefore no selection Because the shrinkage is multiplicative, a coefficient that was nonzero under least squares stays nonzero under ridge for every finite `lambda`. Ridge keeps all p predictors in the model; it makes them all small rather than making some of them disappear. If you want a fit that discards features, the squared penalty is the wrong tool. If you want a fit that uses everything but trusts nothing too much, it is the right one. ## What you are buying Shrunken coefficients are biased, deliberately so: their expected values are pulled toward zero. In exchange, they move far less from sample to sample. For a wide range of problems there exists some `lambda > 0` whose out-of-sample squared error beats the least-squares fit's, and finding it is the job of held-out validation, not of theory. ## Choosing lambda Tune on held-out error - cross-validated error over a grid of `lambda` values spaced logarithmically, since the useful range spans orders of magnitude. Training error rises monotonically as `lambda` rises, so training fit can never be used to choose it. Plotting each coefficient against `lambda` gives a coefficient path: weights slide smoothly toward zero, approaching it asymptotically rather than snapping to it. At `lambda = 0` you recover least squares; at very large `lambda` the slopes vanish and the model predicts the intercept for everyone, which with centred predictors is the target's mean. ## The same penalty on other losses The squared penalty is not tied to squared error. On a classifier fitted by log loss, the objective becomes `sum_i -[y_i * log(p_i) + (1 - y_i) * log(1 - p_i)] + lambda * sum_j w_j^2` with `p_i` the predicted probability. There is no closed form, so it is minimised numerically, but the objective stays convex and the behaviour is identical in kind: on a 40-feature manufacturing quality model, every log-odds weight shrinks proportionally as `lambda` rises, none reaches exactly zero, and predicted probabilities are pulled toward the base rate carried by the unpenalised intercept. ## Common confusions The penalty is squared, not absolute - that difference is exactly why there are no zeros. Ridge does not remove collinearity or features from the data; it changes how the fit responds to them. `lambda` is not on any natural scale, so a value that is strong for one dataset is negligible for another. And a large `lambda` is not safer by default: it is simply a more heavily biased model, and only held-out error can say whether the trade paid off.