When does PPO's clip alone suffice, and when do you add explicit KL control?

## The gap between what the clip does and what it is assumed to do The clipped surrogate is a good heuristic that is routinely described as something it is not. Three precise statements matter for this decision. **It is per sample.** Each sampled action's likelihood ratio is individually discouraged from leaving a band around 1. Nothing in the objective looks at the batch as a whole, so an update in which thousands of samples each move a permitted amount in a correlated direction produces a large shift in the policy distribution while no single ratio ever misbehaves. This is most acute with high-dimensional continuous action distributions, where small per-dimension changes compound into a large divergence. **It is one-sided.** Clipping flattens the objective in the direction the advantage favours. It never applies a restoring force. If a ratio is already outside the band when an epoch begins - which is exactly what happens after several passes - that sample simply contributes zero gradient. It is neither pushed further nor pulled back. The clip has gone silent for precisely the samples that have moved the most, which is the opposite of what an intuitive reading suggests. **It constrains the objective, not the optimiser.** The gradient is evaluated at the current parameters, so a large learning rate can carry the parameters well past the point where ratios leave the band before the flat region is ever felt. The consequence is that a PPO run can produce a large policy change while reporting a perfectly ordinary clip range. That is not a bug - it is what a heuristic proxy for a trust region gets you. ## Measurement is not optional; control is a decision I treat these as separate calls. Instrumentation is free and belongs in every run: the fraction of samples clipped per epoch, and the realised divergence between the collecting policy and the post-update policy, computed on the batch you already have. Without these two numbers you cannot tell a healthy run from one that is drifting, and you certainly cannot tune the epoch count or the clip range for a new environment. Whether to add a *control loop* on top is a cost question with three inputs. **What does a bad update cost?** If the environment is a cheap, resettable, massively parallel simulator, the answer is a restart from the last checkpoint and some wasted compute. Clip plus checkpointing is then the right engineering trade, and adding a divergence controller means one more knob to tune, one more failure mode, and one more thing to explain to whoever inherits the run. If the environment is a physical system, an expensive licensed simulator, or a live market-making agent whose policy is trading, a destroyed policy costs money and time that cannot be recovered by re-running. **Can the environment be re-run?** A non-stationary environment is the sharp case. A market-making agent cannot replay yesterday's order flow to recover from a bad update; the data that would have retrained it is gone. That asymmetry, more than the raw compute cost, is what justifies paying for control. **How far from the assumptions is the setup?** Many epochs per batch, wide clip ranges, high-dimensional continuous actions and aggressive learning rates all widen the gap between per-sample clipping and actual policy movement. The further out you are, the less the clip alone is worth. ## The controls, in ascending order of intrusiveness **Early-stop the epoch loop.** After each epoch, compute the divergence between the collecting policy and the current one on the batch. If it exceeds a threshold, stop the update and go collect fresh data. This is my default addition: it costs almost nothing, changes no objective, and it converts an unbounded heuristic into something with a measured ceiling. In the market-making case it is what I would run. **Shrink the obvious knobs.** Fewer epochs, a smaller clip range, a smaller learning rate. Cheapest of all, but they trade away sample efficiency uniformly rather than acting only when drift actually occurs. **Add an explicit divergence penalty with an adaptive coefficient.** Keep first-order optimisation, add a penalty term, and raise or lower its coefficient depending on whether the measured divergence overshot or undershot a target. More direct than clipping because it acts on the quantity you actually care about, but the coefficient is now a control loop of its own that can oscillate and needs tuning. **Move to a method with a checked constraint.** The strongest option and the most expensive in complexity and per-update cost. Justifiable when a violated step is genuinely unacceptable, rarely justifiable otherwise. ## The organisational half of the answer Every knob you add is inherited by someone. A team that will retrain this agent weekly, possibly without deep reinforcement learning expertise, is better served by a simple configuration plus loud alarms than by an elegant adaptive controller nobody understands. So my ordering is: always instrument, add early stopping when the environment is non-stationary or an update is expensive, and reach for penalties or constrained methods only when the measurements show that the simpler options are not holding. Adding control before the measurements justify it is tuning without evidence, which is how run configurations accumulate settings no one can defend.