How does TRPO's hard KL constraint differ from PPO's clipped objective?

## The theory both descend from Both methods come from the same result: the expected return of a new policy can be written in terms of the old policy's advantages plus a term that depends on how much the state distribution shifts. That gives a lower bound of the form *surrogate improvement minus a penalty proportional to the divergence between the policies*. Maximising the bound guarantees the true return does not go down. The awkward part is that the theoretically correct penalty coefficient is very conservative, so faithfully optimising the bound produces steps too small to be practical. ## What TRPO does with it TRPO converts the penalty into a constraint with a knob you can set: maximise `E[ (pi_new(a|s)/pi_old(a|s)) * A ]` subject to `E_s[ KL(pi_old(.|s) || pi_new(.|s)) ] <= delta`. The constraint is on a *divergence between distributions*, averaged over the states in the batch - a statement about the policy as a whole, not about any individual sample. Solving it needs three pieces: 1. **A local quadratic model of the KL.** Near the old parameters, the average KL is well approximated by a quadratic form whose matrix is the Fisher information matrix of the policy distribution. This makes the trust region an ellipse in parameter space whose shape reflects how sensitively the policy's *output distribution* responds to each parameter direction - not a sphere of fixed parameter radius. 2. **A natural-gradient direction.** The step direction is the gradient preconditioned by the inverse Fisher matrix. That matrix is far too large to build for a network, so it is handled with conjugate gradient using Fisher-vector products, which require only extra backward-style passes rather than an explicit matrix. 3. **A backtracking line search.** The quadratic model is only an approximation, so the proposed step is shrunk geometrically until an actual evaluation shows the KL constraint holds and the surrogate genuinely improved. This is what makes the constraint real rather than nominal. The result is a method that will refuse a bad step. The cost is substantial: second-order machinery, several extra passes per update, fragility with parameter sharing between policy and value networks, and an implementation most teams find hard to get right. ## What PPO does instead PPO keeps the goal - do not let one update move the policy far - and abandons every part of the apparatus. It has no constraint, no Fisher matrix, no line search. The clipped surrogate is an ordinary differentiable objective that first-order minibatch gradient ascent optimises directly, which is why several epochs over one batch are affordable and why it composes with shared trunks and any architecture you like. The approximation is doubled, and it is worth being precise about where. First, PPO measures closeness in *ratio space per sample* rather than *KL over the state distribution*: a bound on individual likelihood ratios is related to but not the same as a bound on average divergence, and many samples each moving a permitted amount can still add up to a large distributional shift. Second, it does not enforce anything even in ratio space - flattening the objective removes the incentive to move further, but a ratio already outside the band contributes zero gradient rather than being pulled back, and a large learning rate can step past the boundary before the flat region is felt. ## How to talk about the tradeoff TRPO offers a checked guarantee at the price of complexity and per-update cost. PPO offers a heuristic that is much cheaper, much simpler, tolerant of the tricks practitioners want to use, and empirically strong across a wide range of tasks - which is why it became the default and TRPO largely a reference point. The honest framing is that PPO's clip is a first-order stand-in for a trust region, and its reliability comes from a body of practice rather than from a theorem. The practical consequence follows directly: because nothing is enforced, a PPO run needs measurement where TRPO has proof. Watching the fraction of clipped samples and the realised divergence between the pre-update and post-update policy is how you recover, empirically, the assurance the constraint would have given you. ## A middle option There is also a penalty formulation that sits between the two: keep first-order optimisation but add an explicit divergence penalty to the objective, with a coefficient adapted up or down depending on whether the measured divergence overshot or undershot a target. It is simpler than a hard constraint and more directly aimed at distribution distance than clipping is, and it is the natural thing to reach for when the clip alone proves too loose.