Why do you try to overfit a single small batch before launching a full training run?

## The test Take one small batch — eight to thirty-two examples is typical — and train on that *same* batch repeatedly for a few hundred to a few thousand steps, with every source of randomness in the input path switched off. The expected outcome is a training loss that falls to near zero and a model that predicts every label in that batch correctly. On a small model this takes seconds; on a large one, a minute or two. Nothing else about the run has to be right yet: no validation split, no schedule, no tuned regularization. ## Why it is informative Memorizing a handful of fixed points requires no generalization at all. A network with more free parameters than the batch has examples can, in principle, drive the loss on those points to zero by rote — it does not have to discover any structure, only to bend its decision boundaries around eight specific inputs. That makes the test a single, sharp question: **can this pipeline move parameters in the direction the loss says they should move?** If the answer is no, more data will not help, a longer schedule will not help, and a bigger model will not help. The value is in the cost asymmetry: the same verdict arrives in one minute rather than after six hours of a full run whose flat loss curve you would then have to interpret from a distance. ## What a failure implicates A one-batch failure narrows the suspect list to the training machinery itself: - **The update path is broken.** Some parameters are frozen, or were never handed to the optimizer, or an operation between the loss and the weights discards gradient, so the tensors that should change never do. - **The step size is wildly wrong.** Far too small gives a loss that drifts imperceptibly; far too large gives oscillation, or a loss that becomes non-finite within a few steps. - **The loss and the targets are wired to something the head cannot reach.** Targets outside the range the loss expects, class indices offset, a loss whose minimum is not where you assume, or positions counted in the average that carry no supervision. - **Numerics kill the signal.** Activations saturated at initialization so that the gradient through the whole batch is effectively zero. A partial failure is just as informative. Suppose a named-entity tagger is given one batch of eight sentences and, after thousands of steps, the loss will not go below about 0.9. Perfect memorization of eight sentences should be trivially reachable, so a stable positive floor means something is holding the loss up: padding positions still included in the averaged loss, two identical inputs in the batch carrying conflicting labels, or a regularizer that was never switched off. A floor is a different diagnosis from a flat line, and the two point at different halves of the code. ## What a pass does *not* prove This is where candidates overclaim. A near-zero loss on one batch says nothing about generalization, nothing about whether the learning rate suits the full dataset, nothing about the shuffling and sampling path, and nothing about whether the labels are correct — a batch whose labels were assigned at random memorizes just as happily as a correct one, which is precisely why the test cannot detect that class of problem. It certifies plumbing, not modelling. ## Where it sits in the ladder The test is one rung of a cheap-to-expensive sequence, and each rung is run only after the one below it passes: 1. **Loss at initialization** matches what an uninformed model should report for that loss and label distribution. 2. **One batch** driven to near-zero loss, all regularization and augmentation off. 3. **A small subset — roughly 200 examples** — driven to near-zero training loss, still with regularization off. This one exercises the sampler, shuffling and a wider spread of labels, so it catches things a single hard-coded batch cannot. 4. **The full set** with a real validation curve, where generalization becomes the question at last. Stop at the first rung that fails; the failed rung is the smallest surface that can contain the bug. ## Practical details Use a batch of at least eight rather than one. A fully-connected batch-normalization layer normalizes each unit across the batch dimension, so with a single example every unit's normalized value is zero and the layer emits its learned shift regardless of the input — the test then fails for a reason that has nothing to do with your code. Log the loss every step rather than every epoch; on a memorization run the whole story happens in the first few hundred updates.