Why does a mean-field variational approximation understate posterior variance?

## What "mean-field" means Variational inference approximates a posterior `p(z|x)` by the closest member `q(z)` of some family you choose. The **mean-field** family is the simplest useful choice: partition the unknowns into blocks and force the approximation to factorise across them, `q(z) = q1(z1) q2(z2) ... qK(zK)`. Each factor gets its own parameters and is optimised independently of the others' shapes (they still interact through the objective). With Gaussian factors and scalar blocks, this is a Gaussian with a **diagonal covariance matrix** — axis-aligned, no off-diagonal terms. The attraction is computational: each factor is low-dimensional, updates are often closed-form for conditionally conjugate models, and the cost grows linearly rather than quadratically in the number of parameters. The cost is representational: **whatever correlation the true posterior has is unrepresentable**. ## The geometry of the failure Picture a posterior over two parameters that is strongly positively correlated — a long, thin, diagonally tilted ellipse, a ridge. Its marginal spread along each coordinate is wide, because the ridge extends far in both coordinates. But conditionally, once you fix one coordinate, the other is pinned down to a narrow slice across the ridge. Now fit an axis-aligned two-dimensional Gaussian. It has no ability to tilt. If it stretched to match the wide marginals, it would become a large axis-aligned blob covering the two off-ridge corners where the true posterior density is essentially zero. If it shrinks to roughly the width of the ridge's narrow cross-section, it sits entirely inside the region of high posterior density but reports far too little uncertainty. The optimiser picks the second option, so **the fitted variances approximate conditional spread rather than marginal spread**. The general statement: for a jointly Gaussian target, the mean-field solution recovers the means exactly and produces variances equal to the reciprocals of the diagonal entries of the precision matrix — the conditional variances — which are less than or equal to the true marginal variances, with equality only when the parameters are already uncorrelated. Correlation strength is exactly what drives the shortfall. ## Why the objective pushes that way The divergence being minimised is `KL(q || p(z|x)) = E_q[log q(z) - log p(z|x)]`, with the expectation under `q`. Look at where each region contributes: - Where `q` has mass and the posterior density is near zero, `log p(z|x)` is very negative and the integrand blows up. This is a **large penalty** — the objective is intolerant of overreach. - Where the posterior has mass but `q` has essentially none, the region is weighted by `q` itself, so it contributes almost nothing. Missing posterior mass is **nearly free**. The asymmetry is one-sided in the shrinking direction: `q` is pushed to stay strictly inside the posterior's support and high-density region. Combined with a family that cannot tilt, the result is systematic under-dispersion. This behaviour is often described as zero-forcing: wherever the target is near zero, `q` must be near zero too. ## What is and is not damaged - **Point summaries survive reasonably well.** Posterior means from a mean-field fit are often close enough for prediction and ranking, which is why the method is used at all. - **Uncertainty is optimistic.** Posterior standard deviations, tail probabilities, and any decision rule sensitive to spread inherit the shrinkage. Probabilities like "chance the effect exceeds zero" are pushed towards 0 or 1. - **Joint statements are worst.** Any question about two parameters simultaneously — the probability both exceed a threshold, the spread of a sum or difference — depends on the correlation the family threw away, and can be badly wrong in either direction. ## What to do about it 1. **Enrich the family.** Allow a full covariance matrix, or group tightly coupled parameters into a single block so their dependence lives inside one factor rather than across the factorisation. This costs more per iteration but removes the structural constraint. 2. **Reparameterise to decorrelate.** Centring predictors, standardising scales, or a non-centred parameterisation of a hierarchical model can turn a tilted ridge into something close to axis-aligned, at which point the mean-field assumption becomes almost harmless. 3. **Check against a reference.** Fit an exact or sampling-based posterior on a subsample or a simplified version of the model and compare spreads; a consistent factor-of-two shrinkage tells you how far to distrust the reported uncertainty. 4. **Match the method to the use.** If the deliverable is a point prediction, the shrinkage may not matter at all. If the deliverable is a risk statement, it matters a great deal. The answer an interviewer wants is the causal chain: factorisation removes correlation, the divergence penalises overreach far more than under-reach, and the two together produce a fit that hugs the high-density core and reports too little uncertainty.