When a flow beats a GAN on exact likelihood but its samples look worse, what do you conclude?

## Two numbers, two objectives The scenario is real and well documented, and it is a favourite senior question because it separates candidates who understand *what an objective optimises* from those who treat every number as "model quality". **Bits-per-dimension** is the standard exact-likelihood report for image models: the model's negative log-likelihood expressed in base two and divided by the number of dimensions. Read literally, it is the average number of bits an optimal coder would need per dimension using the model's density. Lower is better. Autoregressive models and flows can report it exactly. Implicit generators such as GANs cannot report it at all — they define a sampler, not a density — which is itself part of why the comparison is awkward. **Sample quality** is a different quantity altogether: whether an individual draw looks like a plausible member of the data distribution. ## Why maximum likelihood does not chase sharpness Maximising `E_data[log p_model(x)]` is equivalent to minimising the forward Kullback-Leibler divergence `KL(p_data || p_model)`. Look at where the penalty lives. The expectation is over the *data*, so wherever real data exists and the model puts near-zero density, `log p_model` goes to minus infinity and the loss explodes. This is a **mode-covering**, zero-avoiding objective: the model is compelled to place density on every real example. Now look at the other error. If the model puts substantial density on a region containing no data at all, the data expectation never visits that region, so the term contributes essentially nothing. Wasted mass is nearly free. A finite-capacity model under this objective therefore hedges — it smears density broadly to make sure it covers everything — and smeared density produces samples that are averages of plausible things rather than any one plausible thing. In image terms, blur. The adversarial family sits at the opposite end. A GAN never evaluates a density; it is trained so a discriminator cannot separate its samples from real ones. Every generated sample is judged, so implausible samples are punished directly, which rewards sharpness. Nothing in the game punishes *failing to generate* a whole region of the data — hence mode dropping. Sharper samples, worse coverage, no reportable likelihood. ## Why likelihood is dominated by local structure The second half of the answer is dimensional. A bits-per-dimension score sums a contribution from every dimension of the data. In a high-resolution image, the overwhelming majority of those dimensions are locally predictable from their neighbours — smooth gradients, texture, sensor noise statistics. A model that nails those low-level correlations captures most of the available bits, whether or not it has any grasp of global composition. Conversely, getting global structure right moves comparatively few bits. The well-known consequence, argued formally in the generative-model evaluation literature: log-likelihood and sample quality can be made almost arbitrarily independent in high dimensions. A model can be constructed with excellent likelihood and terrible samples, and another with beautiful samples and appalling likelihood. Neither number is a proxy for the other, and a small likelihood improvement should never be reported as a visual improvement. ## What a principal-level answer does with this 1. **Refuse the single-number framing.** Ask what the model is for. If downstream consumers read `log p(x)` — lossless compression, likelihood-ratio anomaly scoring, model comparison, importance weighting — then likelihood *is* the deliverable and its exactness is the reason to pick this family. If the deliverable is images a person looks at, likelihood is a weak proxy and should not drive the decision. 2. **Compare like with like.** Bits-per-dimension is only comparable across models trained on identically preprocessed data at the same bit depth and with the same treatment of discretisation. Different preprocessing silently shifts the number, and cross-paper comparisons are frequently invalid for exactly this reason. 3. **Report both, and say which is the objective.** The failure mode to avoid organisationally is a team optimising the number that is easy to compute while shipping the property nobody measured. 4. **Recognise it is not a bug to fix.** The candidate who says "the flow must be undertrained" has missed the point. The gap is the expected behaviour of a mode-covering objective under finite capacity, not a training defect. ## Where the exactness genuinely pays It is worth being concrete about the upside so the answer is not one-sided. An exact likelihood gives you: a principled model-selection criterion that does not require a learned evaluator; density values usable as anomaly scores; a training signal with no adversarial instability, no minimax equilibrium and no bound gap; and a number you can hold constant while you change the architecture. Those are real, and they are why the family survives despite losing sample-quality comparisons. The conclusion to state out loud: a better likelihood is genuine evidence about density fit and nothing more. Judging generative families requires deciding first which property you are actually buying.