How do you decide whether to minimise KL(P||Q) or KL(Q||P) when approximating a distribution?

## Two directions, two different objectives Let `P` be the target distribution — complicated, possibly multi-modal — and `Q` the simpler family you are fitting, say a single unimodal distribution. There are two ways to point KL at the problem: ``` forward: minimise KL(P||Q) = sum_x P(x) log( P(x)/Q(x) ) expectation under P reverse: minimise KL(Q||P) = sum_x Q(x) log( Q(x)/P(x) ) expectation under Q ``` The weights differ, and that is the whole story. The first averages over where the *target* has mass; the second averages over where the *approximation* has mass. ## Forward KL is zero-avoiding In `KL(P||Q)`, every region where `P(x) > 0` contributes, weighted by `P(x)`. If `Q(x)` is near zero there, `log(P/Q)` is large and the penalty is severe — infinite in the limit `Q(x) -> 0`. There is no corresponding penalty for `Q` putting mass where `P` has none: those terms carry weight `P(x) = 0` and vanish. The optimiser therefore does whatever it takes to keep `Q` positive everywhere `P` is positive. Fitting a single unimodal `Q` to a two-mode `P`, forward KL produces a wide fit centred between the modes, assigning real probability to the low-density valley that separates them — a region the target almost never visits. The fit is safe in the sense that it never says "impossible" about something that happens, and wrong in the sense that its typical draws are atypical of `P`. When `Q` is fit to samples, forward KL is also the direction that only needs draws from `P`, since the expectation is under `P`. ## Reverse KL is zero-forcing In `KL(Q||P)`, terms are weighted by `Q(x)`. Where `Q` puts no mass, nothing is charged, no matter what `P` does there. But where `Q` does put mass, `P` had better agree: `log(Q/P)` blows up if `Q` places mass where `P` is near zero. So the optimiser shrinks `Q` onto a region where `P` is comfortably large, and abandons the rest. A unimodal `Q` fitted to a two-mode `P` by reverse KL picks one mode, fits it snugly, and pretends the other does not exist. Draws from that `Q` look like plausible draws from `P` — they just do not represent the full picture, and the fit is typically over-confident, understating spread. ## Choosing, as a judgment call There is no universally right direction; there is a right one for the cost structure you face. **Prefer mass-covering when omission is the expensive error.** Anything where declaring a real outcome impossible is catastrophic — tail risk, safety-relevant rare events, a segment you cannot afford to be blind to, a proposal distribution that must not have holes — argues for forward KL. The price you accept is a diffuse, under-confident approximation that will assign probability to things that never happen. **Prefer mode-seeking when a sharp, self-consistent answer is what gets used.** If the downstream consumer takes a single representative value or a small set of draws and acts on them, a fit straddling a valley between two modes produces answers that are typical of *neither* mode and therefore useless in practice. The price you accept is under-stated uncertainty and total blindness to whatever `Q` walked away from. **Watch what is computable.** Forward KL needs expectations under `P`, which is natural when you have samples from the target and awkward when you only have an unnormalised description of it. Reverse KL needs expectations under `Q`, which you control and can sample from freely. That asymmetry, not aesthetics, often decides the matter — and it is worth being honest about when it does, because a direction chosen for tractability should not be defended afterwards as a modelling preference. **Know that neither is symmetric, and neither is a distance.** If you genuinely need a symmetric, bounded comparison rather than a fitting objective, the Jensen-Shannon divergence, `0.5*KL(P||M) + 0.5*KL(Q||M)` with `M` the mixture, is the standard answer. ## Organisational discipline The practical failure is not choosing wrong; it is not writing the choice down. "The KL between the model and the data" is ambiguous, and two teams reporting "KL" from opposite directions will produce numbers that disagree for reasons nobody can reconstruct months later. The convention to enforce is simple: name the arguments in order every time, state which is the target, and record the reason the direction was picked. If the fitted approximation is under-dispersed, that should be a documented consequence of a deliberate choice, not a surprise discovered when a downstream interval turns out to be too narrow. ## Diagnostics worth running Whichever direction you pick, check the failure mode it invites. For a mass-covering fit, ask what fraction of the approximation's mass sits in regions the target essentially never produces. For a mode-seeking fit, ask what the approximation assigns to regions the target does produce, and whether an entire sub-population has been dropped. A one-number objective will not tell you either of these; a comparison of the two distributions over the regions you care about will.