Why does a VAE with a pixel-wise squared-error decoder produce blurry image samples?

## The mechanism The blur is a direct consequence of the loss, not a symptom of undertraining. Scoring a reconstruction with summed squared error per pixel is equivalent to assuming the decoder outputs a Gaussian distribution over each pixel with a fixed variance, and taking its negative log-likelihood. Under that assumption the network is not asked to produce *a* plausible image; it is asked to produce the value that minimises expected squared error. For any random quantity, that value is its **mean**. So ask what the decoder is uncertain about. The latent code is a compressed, low-dimensional summary. It may say "a face, turned slightly left, dark hair, smiling". It does not say where each individual hair strand falls, or the exact pixel at which the jawline edge sits. Many images are consistent with that description. The squared-error-optimal output is their pixel-wise average. Averaging low-frequency content is harmless: the average of many dark-haired faces still has dark hair. Averaging *high-frequency* content is destructive: edges at slightly different positions cancel into a soft ramp, and textures at different phases cancel into flat grey. That cancellation is what you see as blur. The model is not failing at its objective — it is succeeding at an objective that rewards hedging. ## Why generated samples are blurrier than reconstructions Reconstruction conditions on a code the encoder produced from a real image, so it sits in a region the decoder has seen and the residual uncertainty is relatively small. A sample drawn from the standard-normal prior can land in a region that few or no training posteriors occupied. The decoder has less to go on there, more images are consistent with the code, and the average it hedges toward is broader. Sharpness degrades from reconstructions, to interpolations between two real codes, to fresh prior draws — and that ordering is itself a useful diagnostic. ## The contributing factors **Rate.** The blur worsens as the code carries less information. A heavily regularised VAE — a large KL weight, or a small latent — leaves more of the image undetermined, so the conditional distribution the decoder is averaging over is wider. Turning the KL weight down sharpens samples, at the cost of a latent whose posteriors no longer tile the prior. **Decoder capacity and shape.** A decoder that upsamples from a low-resolution feature map has a limited ability to place high-frequency detail even when the code contains it. Capacity is a real but secondary factor: adding capacity without changing the likelihood mostly gets you a sharper *mean*, which is still a mean. **The likelihood itself.** This is the dominant factor. Fixed-variance Gaussian output is a poor model of natural images. It says every pixel deviates independently and symmetrically, which is why the loss cannot distinguish "the edge is one pixel to the left" from "the edge is smeared over two pixels" — the smeared version often scores better. ## What actually helps - **Change the output distribution.** A discretised logistic or a categorical distribution over pixel intensities lets the decoder express multi-modal uncertainty per pixel instead of collapsing to a mean. An autoregressive decoder conditioned on the latent can produce genuinely sharp detail because each pixel is drawn conditioned on the previous ones rather than averaged — though a decoder that strong risks learning to ignore the code. - **Score on features, not pixels.** A perceptual loss compares deep features of the reconstruction and the target rather than raw pixels, so a texture that is correct but misaligned by a pixel is not punished. This directly removes the incentive to hedge on high-frequency content. - **Add an adversarial term.** Training the decoder against a discriminator adds pressure to be *realistic* rather than merely close on average, and a hybrid of a VAE encoder with an adversarial reconstruction term is a standard way to keep a usable latent while sharpening outputs. - **Learn the output variance** instead of fixing it. This lets the model report uncertainty rather than only hedge on the mean, and it also implicitly rescales the reconstruction term against the KL — a learned global variance behaves like a tunable weight on the trade-off, which is worth knowing before you conclude the change "fixed" the blur. - **Hierarchical latents.** Multiple stochastic layers, with fine-scale detail carried by higher-resolution latents, reduce the amount of detail the decoder must invent from a single global code. ## The thing not to say "It needs more training." Blur is the minimiser of this loss, not a point on the way to it; more epochs converge to a cleaner average, not to a sharp sample. Likewise "the latent is too small" is at best a partial answer — increasing the code size helps a little, but a squared-error decoder with an enormous latent still averages over whatever remains undetermined. If the deliverable is sharp imagery, the fix is at the likelihood or the objective.