Why can a batch-32 run walk off a loss plateau that a full-batch run sits on for thousands of steps?

## What a plateau is, mechanically A plateau is a region where the loss changes very little in most directions, so the averaged gradient `g_full` has small norm. Deterministic gradient descent takes a step proportional to that norm, so on a plateau it takes small steps: the run is not stuck in the sense of having converged, it is creeping. A character-level language model parked on the same loss for two thousand steps is the canonical sighting. The worst case is a saddle: a stationary point where the gradient is exactly zero, with directions of positive curvature (the loss rises) and directions of negative curvature (the loss falls). Exact gradient descent initialised at a saddle stays there forever, and started near one it takes a very long time to leave, because the escape direction's contribution to the gradient starts microscopic and grows only geometrically. High-dimensional loss surfaces of deep networks have many more saddles than local minima, so this is not a corner case. ## Why noise changes the picture The key observation is that a small averaged gradient does not imply small per-example gradients. On a plateau the individual examples typically still push hard, they just push in directions that cancel. The per-example variance `s^2` can be large exactly where the mean is near zero. A mini-batch of size `B` keeps a sample of that disagreement. The batch gradient is the mean plus a fluctuation of typical size `s / sqrt(B)`, so a step is roughly `lr * (g_full + fluctuation)`. When `g_full` is near zero, the step is dominated by the fluctuation — the run keeps moving even though the averaged gradient says stay. And the fluctuation is not confined to the directions the mean points in: it has components along every direction in which examples disagree, which includes the negative-curvature escape directions of a saddle. Once the iterate is displaced along such a direction, the loss surface itself amplifies the displacement — the mean gradient there now points further downhill — and the run leaves. This is a published result rather than folklore: gradient methods with injected or sampling noise escape strict saddle points in a number of steps that scales polynomially in the problem size, while exact descent has no such guarantee. A run at batch 32 has this property; a full-batch run does not. ## The same mechanism ejects a run from a narrow basin Think of the iterate as having a steady-state spread rather than a position. Around a minimum, the mean gradient pulls the iterate back toward the bottom while the noise pushes it out, and the two balance at a characteristic radius that grows with the learning rate and with the noise magnitude `s / sqrt(B)` and shrinks with how strongly the basin curves back. A basin narrower than that radius simply cannot hold the iterate: the noise ball is bigger than the container. This is what people are seeing when a six-layer network on a sensor stream drops into a promising basin early in training, sits in it while the loss falls, and then a few hundred steps later is out of it and back to wandering. Nothing went wrong; the run's noise level was too high for a feature that narrow, and the escape was a matter of time. The corollary is that noise is a dial with two directions. Retention improves when you lower the learning rate or raise the batch size; exploration improves when you do the opposite. A run that has never left a region since step 1000 has too little noise for the stage it is at; a run whose loss never settles has too much. ## Diagnosing a flat stretch in practice Before reaching for the noise dial, distinguish the cases. Is the run stationary or creeping? Track the gradient norm and the distance moved in parameter space per hundred steps — a converged run and a plateaued run look identical on the loss curve and quite different here. Is there noise available to exploit? Measure the spread of gradients across batches: if per-batch gradients all agree and are all near zero, there is nothing for a smaller batch to find, and the cause is elsewhere in the model rather than in the sampler. Does perturbing the run break the stretch? Halving the batch or nudging the learning rate up for a few hundred steps is a cheap test; if the loss immediately resumes falling, you had a plateau that the run lacked the noise to leave. And resist the reflex to declare a long flat stretch a converged model. On deep networks these stretches are frequently transient, and a run that leaves one often drops sharply afterwards. ## The tradeoff to state out loud Noise is what lets a run leave places it should leave, and it is also what stops it from settling in places it should stay. Both effects come from the same quantity, and the batch size and learning rate set that quantity jointly. Any answer that treats gradient noise purely as an approximation error to be minimised misses half the mechanism.