Which prior makes the ridge estimate the MAP solution of a Bayesian linear regression?

## The setup Bayesian linear regression makes two modelling statements. First a likelihood: the target is a linear function of the features plus Gaussian noise, `y_i = x_i . w + e_i` with `e_i ~ N(0, sigma^2)`, independent across observations. Second a prior on the unknown weights, expressing what you believe before seeing data. Choose an independent zero-mean Gaussian, `w_j ~ N(0, tau^2)` for every j: you believe coefficients are centred on zero and unlikely to be enormous, with `tau` setting how unlikely. MAP - maximum a posteriori - estimation returns the single weight vector that maximises the posterior density, which by Bayes' rule is proportional to likelihood times prior. ## The derivation Maximising a product is awkward, so take the negative logarithm and minimise instead; the logarithm is monotone, so the location of the optimum is unchanged. The Gaussian likelihood of the data contributes, up to constants that do not involve `w`, `sum_i (y_i - x_i . w)^2 / (2 * sigma^2)` because the exponent of a Gaussian density is a negative squared deviation over twice its variance. The Gaussian prior contributes, again up to constants, `sum_j w_j^2 / (2 * tau^2)` for exactly the same reason - it is a Gaussian in `w_j` centred on zero. Adding them and multiplying through by `2 * sigma^2`, which does not move the minimum, gives `RSS + (sigma^2 / tau^2) * sum_j w_j^2` That is the ridge objective, with `lambda = sigma^2 / tau^2`. ## Reading lambda The ratio is the whole payoff of the derivation. Lambda is noise variance over prior variance: how much the data is expected to lie, divided by how much freedom you grant the weights. - Noisy data, `sigma^2` large: lambda large, shrink hard, because individual observations should not be trusted to move a coefficient far. - A confident prior that effects are small, `tau^2` small: lambda large again, for a different reason. - A vague prior, `tau^2` very large: lambda tends to zero and MAP tends to plain least squares. A flat prior adds no penalty at all, which is why maximum likelihood and unpenalised least squares coincide here. This also explains a practical observation. The prior term is a fixed cost while the likelihood term accumulates over observations, so as the sample grows the data outvotes the prior and the fitted weights drift toward the least-squares answer. The tuned lambda typically falls as data accumulates. ## The prior assumes a common scale Using one `tau` for every weight is a statement that all coefficients are expected to be about the same size. That is only defensible once the predictors are on comparable scales - a single shared prior variance across a column in metres and a column in millions of currency units is claiming something nobody intended. This is the Bayesian restatement of the practical rule that predictors are standardised before a penalised fit. The intercept, by the same logic, is given a flat prior rather than the shared Gaussian, which is why it is left unpenalised. ## What MAP throws away Under a Gaussian likelihood and a Gaussian prior the posterior over the weights is itself Gaussian, with mean `(X'X + lambda I)^-1 X'y` and covariance proportional to `(X'X + lambda I)^-1`. Two things follow. First, because a Gaussian's mode and mean coincide, the ridge estimate in this conjugate case is both the posterior mode and the posterior mean. Second, the point estimate discards the covariance - and that covariance is the interesting part of a Bayesian treatment, since it says how uncertain each weight is and supports predictive intervals rather than bare point predictions. If uncertainty is what the decision needs, MAP is not enough; if a good point predictor is what the decision needs, the extra machinery may not be worth it. ## Why the view is worth having Three reasons. It makes lambda interpretable instead of a number pulled off a validation grid - you can reason about a plausible prior scale for coefficients in your domain and get an order of magnitude for lambda before tuning. It explains why shrinkage is principled rather than a hack: you are not distorting the fit, you are combining data with a prior belief. And it lets you place other penalties in the same frame - a different prior shape gives a different penalty, which is how the absolute-value penalty arises from a heavier-tailed prior. ## Where the analogy stops MAP is not fully Bayesian. It depends on the parameterisation - reparameterise the weights and the mode moves while the posterior does not - and it gives no uncertainty. And in practice you rarely know `sigma^2` or `tau^2`, so lambda is still chosen by held-out error or by estimating it from the data through the marginal likelihood, rather than read off from stated beliefs.