What does clipping rewards to their sign cost a deep RL agent's learned policy?

## Why anyone clips at all A deep value function is trained by regression onto bootstrap targets built from rewards. The scale of those targets propagates directly into the loss and hence the gradient: a reward a hundred times larger produces a TD error a hundred times larger and an update a hundred times larger through the value head and any trunk it shares with the policy. In practice that shows up as a training curve that is calm for a long time and then spikes the moment the agent first hits a jackpot event, sometimes destroying the representation. Clipping rewards to their sign removes the problem entirely: with rewards in `{-1, 0, +1}` the discounted return is bounded by `1 / (1 - gamma)`, targets live in a known range, and one learning rate serves every task in a suite. So the technique is not foolish. What is foolish is treating it as a free stabilisation trick. ## What it actually changes Clipping is a transformation of the reward function, and any non-affine transformation of the reward can change which policy is optimal. Sign clipping is aggressively non-affine: it maps every positive reward to the same value, destroying all the information that distinguished them. The ad-bidding case makes it concrete. Suppose a conversion is usually worth about $2 but a small fraction are worth $500, and the value head is trained on clipped reward. Under the true objective the policy should spend aggressively to win the impressions that lead to $500 conversions and be indifferent to a handful of $2 ones. Under the clipped objective every conversion is worth exactly one, so the policy maximises conversion **count**. It will bid down on expensive, high-value inventory and chase volume, and it will do so *correctly*, because that is what its reward says. The training curve will look excellent. Revenue will not move, or will fall. The same distortion applies to costs. A $5,000 spend mistake clipped to -1 is as cheap as losing a cent, so the agent has no reason to prefer small mistakes to catastrophic ones. Clipping also removes the ability to express risk preferences: a rare large payoff and a routine small one are indistinguishable, so no policy can trade variance against expectation. ## How to get stability without the distortion The requirement is bounded, well-scaled targets. That does not require destroying order. - **Reward or return scaling.** Divide rewards by a running estimate of the standard deviation of the discounted return. This is an affine transformation with a positive coefficient, so it leaves the optimal policy unchanged in the stationary limit. The caveat is that a running estimate makes the effective reward non-stationary, and early in training, when returns are near constant, the estimated scale is unreliable, so it needs a floor and a warm-up. - **Adaptive target normalisation.** Methods that normalise value targets while simultaneously adjusting the output layer so the network's predictions are unchanged, in the spirit of PopArt, decouple the learning scale from the reward scale without ever touching the reward the agent is optimising. - **An invertible squashing transform.** Applying a monotone, invertible compression such as a signed square-root-style transform to the value targets and inverting it when acting keeps the ordering intact while pulling large magnitudes into a workable range. Unlike clipping, $500 stays larger than $2 after the transform. - **Distributional value learning.** Learning a distribution over returns on a fixed support handles heavy-tailed rewards more gracefully than regressing a single mean, because the rare jackpot occupies its own part of the support instead of dragging the mean estimate around. - **Reshape the units, not the order.** A log transform of a monetary reward compresses scale enormously and is monotone, so ordering survives, though it does change the relative trade-offs and so is a genuine change to the objective that should be a deliberate product decision, not a numerical one. ## The judgment an interviewer is testing The question to ask about any reward transformation is whether it changes the *ranking* of the policies you care about. If your reward is already a count of equally valuable events, clipping is harmless. If reward magnitude carries the business preference, clipping is a change of objective wearing the clothes of an optimisation trick. Two operational habits follow. First, keep the raw reward stream alongside the transformed one and always evaluate the trained policy against the raw objective, so the training curve can never be mistaken for the result. Second, when you must clip, treat the choice of threshold as part of the specification and check what fraction of total value falls above it: if the top one percent of events carries half the revenue, a clip at the ninety-ninth percentile throws away half of what you were trying to maximise.