Why does a sentence VAE's KL term fall to zero while its decoder ignores the latent?

## What collapse looks like You are training a VAE over sentences with a recurrent or otherwise autoregressive decoder. The loss curve looks healthy. Then you log the terms separately and find the KL term has fallen to essentially zero and stayed there, while the reconstruction term flattened out at whatever a decent unconditional language model achieves. Decoding a sentence, swapping in a completely different input's latent, and decoding again produces the *same* output. The latent is inert. That is posterior collapse: `q(z|x)` has become approximately the prior `N(0, I)` for every `x`. The encoder has stopped being a function of its input, and the decoder has learned to generate without a conditioning signal. ## Why the optimiser chooses it Look at the two terms as a transaction. The KL term is the *price*, in nats, of every piece of input-specific information the encoder pushes into the code. The reconstruction term is what that information *buys*. For a weak decoder — a small feed-forward network mapping a code to an output in one shot — the purchase is obviously worth it. Without the code the decoder cannot do better than the dataset average, so reconstruction improves steeply with every nat bought. For a strong autoregressive decoder the calculus flips. Such a decoder is trained with the real prefix available at every step, so it can already predict each token from the tokens before it. The marginal reconstruction gain from the latent is small — and, crucially, it is *not available yet* at initialisation. Early in training the encoder emits noise, so conditioning on the code makes the decoder's job harder, not easier. The KL charge, however, is immediate and exact. Gradient descent takes the locally cheap route: drive the posterior to the prior, bank the KL saving, and lean on the decoder. Once the code is inert the decoder stops attending to it, and the gradient that would teach the encoder to be useful dries up. The state is self-reinforcing. It is worth being precise here: for a sufficiently expressive decoder, collapse is not only a local trap — a collapsed solution can be a genuine optimum of the objective, because the decoder alone can match the data distribution and the KL adds a strictly positive cost. Nothing in the loss says the latent must be used. ## The fixes **KL annealing (warm-up).** Start the KL term's weight at zero and raise it to its target over the first stretch of training. The model gets to load information into the code while the channel is free, and by the time the price kicks in the decoder has learned to depend on it. Cheap, standard, and only a partial fix: once the weight reaches its target, the KL can still be driven back down, so anneal and then keep watching. **Free bits.** Instead of penalising the full KL, penalise only the amount by which each latent dimension's KL exceeds a floor. Below the floor the dimension is free, so the optimiser has no incentive to squeeze it to zero and the code keeps a guaranteed minimum rate. This is a genuine constraint rather than a schedule, and it survives past warm-up — the usual reason to prefer it over annealing alone. **Weaken the decoder.** If the decoder cannot do the job unaided, the reconstruction term forces it to use the code. Drop input tokens at random during training so the prefix is unreliable, shorten the decoder's context window, or cut its capacity. You are deliberately spending reconstruction quality to buy an informative latent. **Make the code hard to ignore.** Feed the latent to every decoding step rather than only the first, or add a skip path from the code to the output. A latent injected once at the start of a long sequence is easy for a recurrent decoder to forget; one that is present at every step is not. ## Diagnosing it Do not trust the total loss. The checks that actually settle it: - **Log KL per latent dimension.** A healthy model shows several dimensions with clearly nonzero KL. Collapse shows all of them at zero; partial collapse shows a handful alive and the rest dead, which is often fine — the model has chosen an effective code size. - **Latent swap.** Encode input A, decode with input B's latent. If the output is unchanged, the decoder is ignoring the code, and no amount of loss-curve staring will tell you otherwise. - **Prior-sample diversity conditioned on nothing.** Sample several codes from the prior and decode each. Identical or near-identical outputs across very different codes is the same evidence from the generative side. ## The trade you are actually making Every fix above buys latent usage by giving something up: annealing complicates the schedule, free bits forces you to hold rate you might not need, and weakening the decoder directly costs reconstruction quality. The right question is not "how do I get the KL off zero" but "does anything downstream need this latent". If you only want a good density model of sentences, a strong decoder that ignores the code is not a bug — it is a language model, and you should have trained one. Collapse only matters when the code is the product: when you need interpolation, controllable generation, or a representation another system consumes.