How do you tune the KL coefficient in an RLHF run, and what breaks at each extreme?

## The objective Every preference-RL run optimises something of the shape *reward minus beta times KL from a reference policy*. The reference is a frozen copy of where you started, usually the supervised-fine-tuned checkpoint. Implementations differ on mechanics - the penalty may be folded into the per-token reward or added as a separate loss term - but the intent is identical: reward pulls the policy away, KL pulls it back. ## Why the penalty is not optional Three reasons, and a strong answer gives more than one. **The reward is only valid locally.** A reward model was trained on responses drawn from a particular distribution. Far from it, its outputs are extrapolation, and an optimiser will find the extrapolated peaks. KL keeps the policy in the region where the reward model's opinion means something. **Capability preservation.** Pretraining and SFT bought fluency, breadth and knowledge that preference data covers only thinly. Unconstrained optimisation of a narrow objective erodes them - the alignment tax in its acute form. **Diversity.** Reward maximisation is inherently mode-seeking. Without a brake, entropy falls and the model converges on a single formulaic response shape for every prompt, which destroys sampling-based techniques downstream. ## What too low looks like The signature is a diverging pair of curves: reward up, quality down. Concretely you see measured KL growing without bound, output entropy collapsing, and text that acquires strange tics - repeated stock phrases, runaway length, unnatural formatting, sometimes near-gibberish that nonetheless scores at the top of the reward model's range. If you only watch reward, this looks like the best run you have ever had. ## What too high looks like The opposite and quieter failure. Measured KL sits near zero, the policy is effectively pinned to the reference, reward creeps, and evaluation results are statistically indistinguishable from the SFT checkpoint. Teams often misread this as *preference training does not work for our task* when it is simply over-regularised. ## How to actually tune it The coefficient itself is not interpretable - its meaning depends on reward scale, batch composition and how KL is estimated. So invert the problem: - **Instrument measured KL** per step, alongside reward, entropy and mean response length. These four curves diagnose almost everything. - **Choose a KL budget** empirically: the divergence at which held-out evaluation still improves. Beyond that point, added divergence buys reward and loses quality. - **Use an adaptive controller** where available - raise the coefficient when measured KL overshoots the target, lower it when it undershoots. This is standard in mature implementations. - **Normalise the reward** so the two terms stay comparable, otherwise a change in reward scale silently retunes your penalty. - **Evaluate checkpoints independently.** Only an evaluation the policy was never optimised against tells you which point on the reward-versus-KL curve you actually want. ## Where the same idea shows up elsewhere Offline direct preference methods carry the identical control under a different name: their temperature parameter scales an implicit KL to the reference, so a low value permits large drift and a high value keeps the policy conservative. Reasoning runs against verifiable rewards sometimes shrink or drop the penalty, on the argument that a programmatic grader is much harder to hack than a learned reward - but they then have to watch language quality and diversity directly, because nothing else is holding them. ## The framing that lands Call it a trust region on behaviour. You are not trying to minimise KL; you are trying to spend a divergence budget where it buys real quality. That framing turns a hyperparameter question into a measurement question, which is what senior interviewers are listening for.