When is square-root learning-rate scaling preferred over linear scaling as the batch grows?

### Two rules, two invariants When the batch grows by a factor `k`, the two candidate recipes are: - **Linear:** `lr -> k * lr` - **Square-root:** `lr -> sqrt(k) * lr` Neither is a theorem. Each falls out of asking *what should stay constant* as the batch grows, and they answer that question differently. Take as given the basic fact that averaging over more examples makes the mini-batch gradient a tighter estimate of the full-batch gradient, with the fluctuation around it shrinking like `1/sqrt(B)`. **Square-root scaling holds the per-update noise constant.** One update moves the weights by roughly `lr * (true gradient) + lr * (noise of size proportional to 1/sqrt(B))`. The random part of a single step therefore has magnitude proportional to `lr / sqrt(B)`. If you want each individual update to be as noisy as it was before — no more jittery, no more deterministic — you set `lr` proportional to `sqrt(B)`. **Linear scaling holds the noise per unit of data constant.** Modelling SGD as a continuous-time process, the amount of random exploration it performs is governed by a ratio that behaves like `lr / B`, sometimes called the temperature of the process. Fixing that ratio gives `lr` proportional to `B`. This matches the step-matching derivation of the linear rule: over a fixed number of examples the large-batch run takes `k` times fewer steps, and each is `k` times bigger, so both the signal and the accumulated randomness per epoch stay put. So linear scaling preserves the character of a *pass over the data*; square-root scaling preserves the character of a *step*. They cannot both be right, and which is closer to right depends on the update rule. ### Why the optimizer decides it Plain momentum SGD applies the gradient as-is: if the batch changes the gradient's typical magnitude or its noise level, the update inherits that change directly, and the linear rule's step-matching argument applies cleanly. This is why linear scaling is the convention in the vision setting where it was popularized — a convolutional classifier trained with momentum SGD on a fixed epoch budget. Adaptive per-coordinate optimizers behave differently. An Adam-family update is roughly `g / (sqrt(v) + eps)` where `v` is a running average of the squared gradient per coordinate. That division is a normalization: it already removes much of the change in the raw magnitude of the gradient, including part of the change caused by averaging over more examples. Multiplying the rate by the full factor `k` on top of an update that has already been rescaled therefore tends to overshoot, and `sqrt(k)` is the better-behaved default. Practitioners training large language models with adaptive optimizers and decoupled weight decay commonly report square-root-like scaling as the setting that transfers. ### How to use this in practice The useful posture is: *the rule sets the search range, the sweep sets the rate.* 1. Pick the rule that matches your update rule — full factor for momentum SGD, square root for adaptive methods. 2. Compute the scaled rate and treat it as the centre of a small grid, typically a factor of two either side. 3. Bring the rate up gradually at the start of the run rather than applying the scaled value at step zero; this is more important the bigger the jump. 4. Compare loss against **examples processed** with the baseline. If the scaled run tracks the baseline, the rule transferred; if it sits above it everywhere, either the rule is the wrong one for your optimizer or you have pushed the batch past the range where any rate scaling recovers the baseline. ### The honest interview answer If someone asks 'linear or square-root?', the answer that reads as experienced is not a single word. It is: linear is the default with momentum SGD, square-root is the default with adaptive methods, both are empirical heuristics that hold over a range and then stop, and in either case the scaled value is a starting point you verify rather than a number you trust. A candidate who insists one rule is universally correct is quoting a paper rather than describing a run they have done.