Why does ridge stabilise a gearbox model whose three vibration channels are near-collinear?

## The symptom A wind-turbine gearbox model takes three vibration channels from accelerometers bolted to the same housing. Physically they see nearly the same motion, so the three columns of the feature matrix are near-duplicates. Fit ordinary least squares and the coefficients come out enormous and contradictory: one channel gets a large positive weight, another a large negative one, and their contributions nearly cancel. Refit on a slightly different window of data and the signs swap. The predictions may be fine on training data and terrible off it. ## Why least squares behaves that way Least squares minimises squared error and nothing else. When three columns are nearly the same, the model can add a large amount along one channel and subtract almost the same amount along another with almost no effect on the fitted values. The objective is therefore nearly flat along that difference direction - a long shallow valley rather than a bowl. Anywhere in that valley is nearly optimal, so the estimate is decided by whatever noise happens to be in this sample. Formally, `X'X` is ill conditioned: it has at least one very small eigenvalue, and inverting it multiplies noise by the reciprocal of that small number. ## What the penalty changes Ridge minimises `RSS + lambda * sum(w_j^2)`, whose solution is `w = (X'X + lambda I)^-1 X'y`. Adding `lambda` along the diagonal raises every eigenvalue of `X'X` by `lambda`. The nearly flat valley becomes a genuine bowl with a unique, well-separated minimum, and the reciprocal of a tiny eigenvalue is replaced by the reciprocal of `small + lambda`, which is bounded. The singular-value picture says the same thing more precisely. Writing `X = U D V'`, ridge scales the least-squares coordinate along each right singular vector `v_j` by `d_j^2 / (d_j^2 + lambda)`. For a direction the data measures well, `d_j` is large, the factor is close to 1, and nothing much happens. For the difference-between-near-duplicate-channels direction, `d_j` is tiny, the factor is close to 0, and that component of the solution is effectively deleted. Ridge does not shrink uniformly; it shrinks the badly determined directions and leaves the well determined ones alone. ## Why the effect gets split evenly The cleanest way to see the consequence is the duplicated-column thought experiment. Copy one predictor exactly, so the model has twins `a` and `b` for the same column. Any pair with the same sum `a + b = s` gives identical predictions, so the data cannot choose between them - but the penalty can: subject to `a + b = s`, the quantity `a^2 + b^2` is smallest when `a = b = s/2`. Ridge therefore splits the effect evenly and zeroes neither twin. With three near-identical channels rather than exact twins the same logic applies approximately: the penalty prefers three moderate, similar coefficients over one huge positive and one huge negative one, because squaring punishes a large cancelling pair severely. A single large weight of size `2c` costs `4c^2`, while two weights of size `c` cost only `2c^2`. ## What ridge does not do It does not remove the collinearity - the sensors still measure the same thing, and no penalty changes the physics. It does not tell you which channel matters. Because the split among near-duplicates is chosen by the penalty rather than by evidence, the individual stabilised coefficients are an artifact as much as a finding: stable, but biased toward zero and shared out by an arbitrary rule. And ridge does not drop a channel; every one stays in the model with a nonzero weight. ## Verifying the fix Do not take stability on faith. Refit on bootstrap resamples, or across cross-validation folds, and look at the spread of each coefficient. Under least squares the near-collinear channels swing in magnitude and sign; under a well-chosen `lambda` that spread collapses to something narrow. Track held-out prediction error over the same grid of `lambda` values so you can see what the stability cost you: a good choice typically improves out-of-sample error as well as steadying the weights, and if error worsens sharply while coefficients keep shrinking you have gone too far. ## The alternatives worth naming Shrinkage is one answer to redundant predictors; it is not the only one. You can combine the channels into one physically meaningful feature - an average, a magnitude, a principal direction - or instrument the machine so the channels are not redundant, or simply drop two and accept the loss. Ridge is the right choice when all three carry a little independent signal and you want a stable predictor without deciding which is real. It is the wrong choice when the deliverable is an answer to which sensor drives the fault, because a penalised coefficient cannot support that claim.