Why must you re-run a learning-rate range test after the batch size or initialization changes?

## What the curve is a measurement of A loss-versus-rate curve is not a property of the architecture. It is a property of the exact pairing of model, data, per-device batch size, starting weights, loss composition and optimizer that produced it. Change any of those and the sweep you ran earlier measured a different system. Two changes move the curve in opposite directions and come up constantly in real work. ## Batch size moves the knee up Each step's gradient is an average over the examples in the batch. For B roughly independent examples, the variance of that average falls about as `1/B`, so the noise standard deviation falls about as `1/sqrt(B)`. Going from 32 to 512 is a sixteen-fold variance reduction — roughly a factor of four in noise scale. Two consequences follow. The update direction is closer to the full-data gradient, so a given rate produces a step that is more reliably downhill. And the tail events — the occasional batch whose gradient is unusually large and points somewhere unhelpful — become rarer and smaller. Those tail events are precisely what turn a marginal rate into a divergent one, so the sharp rise on the curve moves toward larger rates. The keyword-spotting sweep that turned upward at some rate with a batch of 32 keeps improving past it at 512. This has a ceiling. Once the batch is large enough that its gradient is already close to the full-data gradient, adding more examples buys almost no further stability and the knee stops moving. There is also an accounting change worth stating out loud: at a fixed epoch budget, a batch of 512 takes one sixteenth as many steps as a batch of 32, so carrying over the old, now-too-conservative rate gives you both fewer steps and smaller ones. ## Initialization moves the knee down A randomly initialized encoder holds no information. Its loss starts near the chance level, its gradients are large, and the job of early training is to move a long way through weight space — large rates are both tolerated and needed. A pretrained encoder is the opposite. Its weights already encode useful features, the loss starts far lower, and the run's job is a comparatively small adjustment. On the curve, two things change: the whole trace sits lower, and the rise begins much earlier, because a large update does not improve the features, it erases them and drags the loss back toward the from-scratch level. It is common for the usable rate on a pretrained trunk to be one to two orders of magnitude smaller than on the same architecture trained from scratch. Run both sweeps on the same model and you get two curves of similar shape displaced along the rate axis; that displacement is the entire point of re-running the test. ## The mixed case A pretrained trunk with a freshly initialized head is the case that catches people. The head needs a large rate to move at all; the trunk is damaged by it. One curve reports one compromise number, usually dictated by whichever part destabilizes first, and it hides the conflict entirely. If you intend to use separate rates per parameter group, sweep per group — or sweep once with the trunk frozen and once with it unfrozen — and read two numbers instead of one. ## What else invalidates a previous curve Anything that changes the scale or noisiness of the gradients: a changed loss weighting or a new auxiliary term, adding or removing normalization layers, a different gradient-clipping threshold, a longer input sequence, a different data mix, a change of width or depth, or a change of optimizer. The safe rule is that the curve belongs to the exact recipe that produced it. ## The practical habit Re-measure. Two hundred steps is negligible next to the run it protects, and it turns an assumption into an observation. Record the rate together with the fingerprint that makes it valid — batch size per device, initialization source, loss composition — so that anyone inheriting the number can tell at a glance whether their setup is the one it was measured on. Then confirm on the first hundred steps of the real run, where a rate that is too hot or too cold shows up immediately.