What is the linear scaling rule for the learning rate when the mini-batch size grows?

### What the rule says The linear scaling rule is a one-line recipe for transferring a tuned configuration to a bigger batch: **when you multiply the mini-batch size by `k`, multiply the learning rate by `k`.** A run that was tuned at batch 256 with learning rate 0.1 becomes, at batch 8,192 (`k = 32`), a run at learning rate 3.2. The momentum coefficient, the weight-decay coefficient, the shape of the decay schedule and the number of epochs are all normally left alone; only the rate moves. A rate of 3.2 sounds absurd if you are used to seeing rates near 0.1. It is not: the update it produces per *example seen* is the same as before. The step is bigger because it replaces 32 of the old steps. ### Why it is the right first guess Take `k` consecutive steps at batch `n` with rate `lr`. Write `g_i(w)` for the average gradient over the `i`-th small batch evaluated at weights `w`. After `k` steps: `w_k = w_0 - lr * (g_1(w_0) + g_2(w_1) + ... + g_k(w_(k-1)))` Now take a single step over the union of those batches, which has size `k*n`, at rate `k*lr`. The large batch's gradient is the average of the `k` small-batch gradients, all evaluated at the starting point: `w_1 = w_0 - k*lr * (1/k) * (g_1(w_0) + ... + g_k(w_0)) = w_0 - lr * (g_1(w_0) + ... + g_k(w_0))` The two right-hand sides are the same sum. The **only** discrepancy is *where* each gradient is evaluated: the small-batch trajectory evaluates `g_i` at the weights it had actually reached, the large-batch step evaluates everything at `w_0`. So the linear scaling rule is precisely the statement *one big step should travel as far as the `k` small steps it replaces*, and it is valid exactly when the gradient is approximately unchanged over that short stretch of the trajectory. That framing tells you immediately where it breaks. ### Where the assumption fails 1. **The opening of training.** Weights are far from any solution, the loss surface is changing fast, and the gradient at `w_0` is a poor stand-in for the gradient at `w_(k-1)`. Applying the full scaled rate from step zero is the standard way to blow a large-batch run up in the first few hundred iterations. The fix is to bring the rate up gradually rather than starting at its scaled value; the mechanics and length of that ramp are a topic of their own, but *that a large-batch run needs one* is part of the scaling rule's own fine print. 2. **While the rate itself is moving quickly.** When the learning rate changes sharply from step to step, a momentum buffer still holds gradients that were accumulated under the old rate, and that stale contribution is applied at the new one. Correcting the buffer for the change in rate is the known remedy. 3. **At the far end of the batch range.** The rule holds over a wide span and then stops paying. In the large-scale image-classification setting where it was popularized — a fixed 90-epoch budget, scaled from batch 256 outward — the scaled runs tracked the baseline closely across a broad range of batch sizes and then began to fall behind as the batch grew into the thousands and tens of thousands, *even with a correct ramp and a correctly scaled rate*. Beyond that point, pushing the rate further does not recover the gap; it makes things worse. ### What to do when it stops holding The reflex — 'scale the rate harder' — is wrong. The realistic options are: train for more epochs at the large batch, accept a smaller batch, or change the *form* of the update rather than its magnitude. Layer-wise adaptive schemes such as LARS and LAMB take the second route: they set a per-layer step size from the ratio of the layer's weight norm to its gradient norm, so that no single layer takes a step that is enormous relative to its own weights, which is one of the failure modes that shows up at very large batch. ### What the rule is not It is a statement about **matching optimization progress per example**, and nothing more. It does not promise that the large-batch run reaches the same loss in the same number of epochs once you are far outside the range where it holds, and it is not a claim about how the resulting model behaves on held-out data. Keep it in its lane: it moves your search range so that a single well-chosen run replaces a re-tune, and you should still sweep a small grid around the scaled value rather than trusting `k * lr` blindly. ### How to verify it worked Plot training loss against **examples processed**, not against steps, for the baseline and the scaled run. If the rule is holding, the two curves nearly overlap. If the scaled curve is above the baseline everywhere, you are outside the rule's range or your ramp is wrong; if it diverges early, the ramp is the first suspect.