What breaks in the mini-batch gradient when a label-sorted file is streamed unshuffled?

## The assumption that quietly does all the work Every statement about mini-batch gradients — that the expected step equals the full-batch step, that the noise falls with batch size, that the run performs a noisy walk around the exact descent path — assumes the batch is a uniform random sample of the training set. That assumption is not a technicality. It is the only thing standing between a mini-batch optimizer and an optimizer that is confidently descending a different function than the one you wanted. A file sorted by label and read front to back is the cleanest case of the assumption failing. Suppose the rows are grouped: all of class A, then all of class B, and so on. Read sequentially with a batch of 32, and for thousands of consecutive steps every example in the batch belongs to one class. ## What exactly breaks The expectation. Where the uniform sample gives `E[g_B] = g_full`, here the batch is a uniform sample of one class's rows, so `E[g_B] = g_classA` — the gradient of the average loss over class A alone. The estimator is still an estimator; it is just estimating a different quantity. Formally it is biased with respect to the training objective, and the bias is the difference between the class-conditional gradient and the dataset gradient, which for an imbalanced or well-separated problem is large. Call this bias rather than noise, because the two behave in opposite ways under the one knob people reach for. Noise shrinks as the batch grows — quadruple the batch and the standard deviation halves. Bias does not move at all: a batch of 512 rows drawn from one class block gives a very precise estimate of the wrong gradient. You cannot buy your way out with memory. ## What it looks like from the outside The symptoms are recognisable once you have seen them. The training loss sawtooths: it falls smoothly within a class block as the model learns to predict that class, then jumps at the boundary when the next class arrives. The model's outputs drift toward whichever class is currently streaming, because for thousands of consecutive steps every gradient says the same thing — predict this class more often — and nothing pushes back. Then the next block overwrites it. Validation accuracy, measured on properly mixed data, oscillates or plateaus far below where the same model reaches with shuffling. Any layer that computes statistics over the batch also sees only one class, so those statistics track the block rather than the data. At the end of a full epoch the run has seen every example exactly once, which tempts people into thinking the ordering washes out. It does not, and the reason is that the parameters move in between. The epoch's gradients are each evaluated at a different point in parameter space, so their sum is not the full-batch gradient at any single point. Optimization is path-dependent; feeding it a path made of long single-class stretches is not the same as feeding it a mixed one, even when the multiset of examples is identical. ## Fixing it Restore uniformity of the sample. Shuffle the file once on disk so that sequential reads are already mixed, and reshuffle the order each epoch so the same batches do not recur. When the dataset is too large to hold in memory, combine two levels: shuffle the order of the shards or files, and draw each batch from a large in-memory shuffle buffer that is continuously refilled — the buffer needs to be big relative to the length of a same-label run, or single-class batches survive it. If the pipeline reads from several sources, interleave them rather than concatenating. Where labels are severely imbalanced, uniform sampling is still unbiased for the training objective, but you may deliberately choose a non-uniform sampler — class-balanced sampling, for instance. That is a bias you are introducing knowingly, and it changes the objective being optimised in a way you can compensate for. The unshuffled sorted file is the same mathematical situation arrived at by accident, with nobody compensating for anything. ## Why the question gets asked It is a fast probe for whether a candidate knows why shuffling matters rather than merely that it does. The candidate who says the ordering adds noise has the wrong model of what went wrong and will reach for a bigger batch. The candidate who says the estimator became biased knows immediately that batch size is irrelevant here and that the sampler is the thing to fix.