Growing the training batch from 256 to 8192 keeps the final training loss but costs two points of test accuracy — why?

## The situation Two runs of the same architecture on the same data. One uses a batch of 256, the other 8192. Both converge; the final training losses match to within noise. Test accuracy differs by two points in favour of the small batch. Nothing in the training curve explains it, which is precisely why this scenario is an interview favourite: the candidate has to reason about *where* the optimizer landed rather than *how low* it got. ## The sharp-minima account The best-known explanation is that large-batch training converges to sharper minimizers. Under this account both runs reach the same training loss, but the large-batch solution sits in a narrow basin: perturb its weights slightly and the training loss climbs steeply, whereas the small-batch solution's loss barely moves. Since the test surface behaves like a slightly displaced training surface, the narrow solution loses more when that displacement is applied, and the loss becomes a test-accuracy gap. The mechanism usually offered is about exploration. A small-batch trajectory is erratic: successive updates disagree, and the parameters wander rather than descending cleanly. A large-batch trajectory is much closer to deterministic descent, so it tends to enter the first basin it meets and stay there — including basins narrow enough that a wandering trajectory would have been knocked out of them. Nothing about a narrow basin makes it worse *on the training data*, which is why the training loss gives no warning. ## The confounds you must rule out before believing it This account is an empirical observation, not a law, and the literature has repeatedly shown that a naive comparison overstates it. **Update count.** At a fixed epoch budget, moving from batch 256 to 8192 cuts the number of parameter updates by a factor of thirty-two. Half the reported large-batch gaps in casual experiments are really under-training. The honest comparison holds the number of updates fixed, or extends the large-batch run until it stops improving, not until it has seen the same number of epochs. **Hyperparameter transfer.** Every hyperparameter that was tuned for batch 256 is now being used out of distribution. A configuration that is merely mistuned for the new batch looks exactly like a generalization gap. Retuning for the new batch size is a topic in its own right, but its existence means an untuned comparison proves nothing. **Regularization budget.** Fewer updates also means fewer draws of whatever stochastic regularization you use, and a shorter effective exposure to augmentation. Match those before attributing the difference to landscape geometry. ## How to test the sharpness claim directly Don't argue about it — measure it. Take both final checkpoints and apply the same probe to each: add random weight perturbations drawn at a fixed radius to all weights, evaluate the training loss on a fixed batch, average the rise over many draws, and sweep the radius. If the large-batch checkpoint's loss rises substantially faster at every radius, the sharpness story has support. If the two curves overlap, the gap is coming from somewhere else and you should look at the update count, the input pipeline, or the evaluation itself. One caution when reading such numbers: a fixed-radius, coordinate-uniform probe is sensitive to how large the weights happen to be, so compare checkpoints whose weight norms are comparable, and be suspicious of a sharpness ratio that tracks a weight-norm ratio. ## What to do about it If you must keep the large batch — usually because you are filling a fixed hardware budget and want throughput — the levers that stay inside the flatness story are: train for more updates rather than more epochs; strengthen data augmentation and other explicit regularization to compensate for the smoother trajectory; and adopt an objective that explicitly penalizes sharpness, which optimizes the worst loss in a small ball around the weights and reliably reduces measured sharpness at roughly double the compute per step. If you do not need the large batch, the cheapest fix is often not to use one. Batch size is a throughput decision; past a certain point the extra examples per step buy very little additional progress per update, and you are paying for hardware utilization rather than learning. ## The honest summary Say the observation, say the mechanism, then say the caveat. Large-batch runs *tend* to land in sharper basins and that tendency is a real, repeatedly observed phenomenon, but much of the raw gap in a casual experiment is under-training and mistuning, and a good chunk of the remainder can be closed with enough updates and the right regularization. A senior answer names the measurement that would settle it.