Your RL variant beats a baseline that lacked observation normalisation — is the gain real?

## The situation Someone reports that a new algorithmic idea beats a baseline. The baseline ran without observation normalisation. The question is what fraction of the reported gap belongs to the idea. ## Why the details are not details A family of small, usually undocumented choices sits between an algorithm's equations and a working implementation, and empirically they carry a large share of measured performance. **Observation normalisation.** Inputs are scaled by a running mean and standard deviation maintained over the observations seen so far. Without it, an observation vector whose components differ by orders of magnitude — joint angles near 1, velocities in the tens — makes the first layer's response dominated by the large components, and a single learning rate cannot suit all of them. On continuous-control benchmarks this alone flips a run between learning and flat. **Reward and return scaling.** Value targets inherit the reward scale. A critic trained on targets in the thousands needs a different effective step size from one trained on targets near one, so scaling rewards by a running estimate of return magnitude changes how the critic and policy losses trade off. **Per-batch advantage standardisation.** Subtracting the batch mean from advantages is, in expectation, a baseline shift that leaves the policy gradient's expectation essentially unchanged; dividing by the batch standard deviation is not neutral, because it rescales the gradient magnitude and therefore acts as an adaptive step size. On clipped policy-gradient methods it also changes how often the clipping threshold binds. Others in the same family: gradient-norm clipping, learning-rate annealing, the initialisation scale of the final policy layer, and how the value loss is weighted against the policy loss. ## Two failure modes in the comparison **Unmatched details.** If the new arm has normalisation and the baseline does not, the experiment measures the union of the two changes. The literature on reproducing policy-gradient results has repeatedly found that re-adding these details to a baseline erases much of a published algorithmic gap. **Unmatched tuning budget.** If the variant was tuned over hundreds of configurations and the baseline runs the original paper's defaults on a new environment, part of the gap is tuning, not method. This one is easy to miss because it looks like diligence. Both are made worse by the seed spread. If a single run per arm decides the comparison, and within-arm seed spread is large, the reported difference may be neither algorithm nor detail but chance. ## How to actually attribute the gain 1. **Fix the protocol first.** Same environment, same interaction budget, same evaluation procedure, same set of seeds, decided before results are seen. 2. **Give both arms every detail.** Turn normalisation, reward scaling and advantage standardisation on for both, or off for both. The comparison is only meaningful with the detail axis held constant. 3. **Match the tuning budget.** Same search space, same number of trials per arm, and re-tune the baseline on this environment rather than importing defaults. 4. **Ablate one axis at a time.** With the algorithmic change fixed, toggle each detail and measure the swing. This produces the sentence that makes the result credible: how much of the gap each cause owns. 5. **Size the effect against the noise.** If the algorithmic delta is smaller than the seed spread inside one arm, you have not demonstrated it, regardless of how clean the ablation looks. ## Operational consequences beyond the paper Normalisation statistics are state. The running mean and standard deviation are part of the model: they must be checkpointed with the weights, frozen at evaluation and deployment, and applied identically to serving inputs. Evaluating with statistics that keep updating, or with statistics reset, silently changes the policy's inputs and is a common source of an offline score that will not reproduce online. Reward scaling has the same property when a running return estimate feeds it. ## The lead's call As the person who sets the standard, the useful policy is not 'ban implementation details' — the details are often genuinely necessary. It is to require that comparisons hold them constant, that tuning budgets are matched and stated, and that the write-up separates 'this idea helps' from 'this configuration is strong'. The failure this prevents is the expensive one: a team building on a claimed algorithmic contribution that was really a normalisation switch, and finding a year later that their stack already had it on.