Why does a zero-mean Gaussian prior on regression coefficients turn MAP into ridge regression?

## The correspondence in one line MAP estimation maximises `log-likelihood + log-prior`. Penalised estimation minimises `loss + penalty`. Because minimising a negated quantity is the same as maximising it, every prior corresponds to a penalty and every penalty corresponds to a prior: ``` penalty(w) = -(constant) * log p(w) ``` The two most familiar penalties in linear regression fall straight out of the two most familiar symmetric priors centred at zero. ## Deriving the ridge case Take a linear model where each response is the dot product of the predictors with a coefficient vector `w`, plus independent Gaussian noise of variance `sigma^2`. The log-likelihood of the data is ``` log p(data | w) = -RSS(w) / (2 * sigma^2) + constant ``` where `RSS(w)` is the residual sum of squares, `sum over i of (y_i - prediction_i)^2`. Maximising this alone is ordinary least squares — the Gaussian-noise MLE. Now place an independent zero-mean Gaussian prior with variance `tau^2` on every coefficient. A Gaussian density is proportional to `exp(-w_j^2 / (2 * tau^2))`, so summing over coefficients gives ``` log p(w) = -sum(w_j^2) / (2 * tau^2) + constant ``` Add the two and drop the constants: ``` maximise -RSS(w) / (2 * sigma^2) - sum(w_j^2) / (2 * tau^2) ``` Multiplying by `-2 * sigma^2` flips it into a minimisation without moving the optimum: ``` minimise RSS(w) + (sigma^2 / tau^2) * sum(w_j^2) ``` That is the ridge objective, and the correspondence is `lambda = sigma^2 / tau^2`. The penalty weight is the noise-to-prior variance ratio, which is where the usual shorthand *prior variance behaves like one over lambda* comes from. The two limits are worth being able to state instantly. As `tau^2` grows without bound the prior flattens, `lambda` goes to zero, and the MAP estimate becomes ordinary least squares — the flat-prior collapse of MAP onto the MLE, seen through a regression lens. As `tau^2` shrinks toward zero the prior insists the coefficients are near zero, `lambda` explodes, and the fitted coefficients are crushed toward the origin. One practical detail that interviewers like to hear: the intercept is normally left unpenalised, which in the Bayesian reading means giving it a flat prior rather than a zero-mean Gaussian one. Shrinking an intercept toward zero would be a statement that the response is centred near zero, which is rarely something you believe. ## Deriving the lasso case Keep the same Gaussian likelihood and swap the prior for an independent zero-mean Laplace (double-exponential) prior with scale `b`, whose density is proportional to `exp(-|w_j| / b)`. Then ``` log p(w) = -sum(|w_j|) / b + constant ``` and the same rearrangement gives ``` minimise RSS(w) + (2 * sigma^2 / b) * sum(|w_j|) ``` which is the lasso objective with an L1 penalty. The Laplace density has a sharp kink at zero — its derivative jumps rather than passing smoothly through — while the Gaussian density is flat-topped there, with zero derivative. That kink is exactly why the L1 posterior mode can sit at precisely zero for a coefficient whose data support is weak: the penalty keeps a constant pull toward zero no matter how close you get, whereas the quadratic pull of the Gaussian prior fades away as the coefficient approaches the origin. The Gaussian prior shrinks; the Laplace prior shrinks and can zero out. ## Where the analogy stops The correspondence is between penalised estimation and the **mode** of the posterior, not the posterior itself, and this is the point candidates most often miss. With a Gaussian likelihood and a Gaussian prior the posterior over the coefficients is itself Gaussian, hence symmetric, so the mode and the mean agree and the ridge solution really is the whole posterior's centre. Under a Laplace prior the posterior is not symmetric in the relevant way and, crucially, its **mean is not sparse**: the posterior assigns zero probability to a coefficient being exactly zero, so averaging over the posterior gives small non-zero values. Only the mode lands on exact zeros. A candidate who says *the Bayesian version of the lasso gives you sparsity for free* has confused the mode with the posterior. Similarly, reading `lambda` as `sigma^2 / tau^2` explains what the penalty *means* but does not by itself hand you a number, because you rarely know `tau^2`. What the Bayesian reading buys you is interpretation: a penalty is an assumption about coefficient magnitudes, stated as a distribution, and the strength of that assumption is measured relative to how noisy the data are.