Why is adding Gaussian noise to a least-squares fit's inputs equivalent to a ridge penalty?

## The setup Take a linear model that predicts `x·w` and is fitted by minimising the sum of squared residuals. Now corrupt the training inputs: before each row is used, add to every feature an independent draw from a zero-mean distribution with variance `sigma^2`. The targets are left alone. The claim is that fitting on the corrupted inputs is, in expectation, the same as fitting on the clean inputs with a squared-weight (ridge) penalty attached. ## The derivation Write the clean residual for row i as `r_i = y_i - x_i·w`, and the noise vector added to that row as `e_i`, with `E[e_i] = 0` and `E[e_i e_i'] = sigma^2 * I`. The jittered residual is `y_i - (x_i + e_i)·w = r_i - e_i·w`. Square it: `(r_i - e_i·w)^2 = r_i^2 - 2*r_i*(e_i·w) + (e_i·w)^2` Take the expectation over the noise, term by term. - `r_i^2` does not involve the noise, so it survives unchanged. - `-2*r_i*(e_i·w)` has expectation zero: the noise is zero-mean and independent of the data, so the cross term averages out. - `(e_i·w)^2` is a quadratic form. Its expectation is `w' E[e_i e_i'] w = sigma^2 * ||w||^2`, which does **not** average out — a square of a zero-mean quantity is positive on both sides of zero. Sum over the n training rows: `E[jittered loss] = SSE + n*sigma^2*||w||^2` That is precisely the ridge objective, with `lambda = n*sigma^2`. If you write the loss as a *mean* squared error rather than a sum, the n cancels and `lambda = sigma^2`. Either way, more noise means a stronger penalty, quadratically in the noise scale. ## What the identity is telling you The weight on a feature is a multiplier on that feature's value. If the feature arrives with measurement error, the weight multiplies the error too, so a model with large weights is a model whose predictions swing wildly when the inputs wobble. Asking for robustness to input wobble and asking for small weights turn out to be the same request. That is a useful frame beyond this one identity: many regularizers are best read as a statement about which perturbations the model must be insensitive to. ## Details that decide whether it works **The intercept escapes.** The constant column is not a measured feature and is not jittered, so no penalty falls on the intercept — which matches the usual convention that ridge leaves the intercept alone. **Column scale matters.** One `sigma` applied to a column measured in millimetres and a column measured in kilometres perturbs the first enormously and the second not at all, so the induced penalty per feature is set by arbitrary units. Put the columns on a common scale first, or set a per-column noise scale deliberately. **Correlated noise gives a generalised penalty.** If the noise has covariance C rather than `sigma^2 * I`, the induced term is `w' C w` — a Tikhonov penalty with a shaped matrix rather than a plain squared norm. That is a feature, not a bug: it lets you say that some directions in feature space are noisier than others. **It is exact only here.** Squared loss, linear model. For a smooth nonlinear model and small noise, the same expansion produces a penalty on the model's *sensitivity* — the squared gradient of the output with respect to the inputs — which reduces to `||w||^2` in the linear case but is not the same object in general. For a non-squared loss the clean cancellation does not happen at all. **In expectation is not the same as in practice.** The identity averages over the noise. If you actually generate three jittered copies of each row, you have a stochastic estimate of the penalty, which adds variance to the fit and multiplies your training cost — while a closed-form penalty term is deterministic and free. ## So why ever jitter instead of writing the penalty? Because sometimes the noise encodes something true. Consider fitting an activity classifier on wearable accelerometer readings. The sensor genuinely has measurement error of a known order of magnitude, and the label — walking, sitting, cycling — does not change when a reading moves by that much. Jittering the readings at the sensor's own error scale states an invariance you actually believe, and it can be shaped to match the physics: correlated across the three axes, larger at high amplitudes. A generic squared-weight penalty says only 'keep the coefficients small'; honest jitter says 'this specific perturbation must not change the prediction'. The equivalence is what tells you what jitter costs you when the invariance is *not* real: it is just ridge with extra variance and extra compute. ## Choosing the noise scale Either anchor it to a known measurement error, or treat it as a hyperparameter and tune it on validation exactly as you would tune a penalty strength. Both routes are defensible; inventing a number because it looks small is not. And remember the identity runs both ways — if a jitter scale you tuned corresponds to an implausibly huge lambda, that is a signal you are shrinking harder than you intended.