Why does the loss fall smoothly with full-batch gradient steps but rattle with stochastic ones?

## Two different objects called the gradient The training loss is an average over n examples, `F(theta) = (1/n) * sum_i f_i(theta)`. Full-batch gradient descent computes `grad F` exactly, using all n examples per step. Stochastic gradient descent computes the average gradient over a random subset of b examples and treats it as a stand-in for `grad F`. If the subset is drawn uniformly at random, that mini-batch gradient is an **unbiased estimator**: its expected value, over the draw of the batch, equals the full gradient. Unbiased does not mean accurate on any single draw. If per-example gradients have variance sigma squared around the full gradient, the average over b of them has variance about `sigma^2 / b`, so its standard deviation falls like `1 / sqrt(b)`. Quadrupling the batch halves the noise. ## Why full-batch is monotone and stochastic is not For a smooth objective with curvature bounded by L, a full-batch step with `eta < 2 / L` is guaranteed to decrease the objective at every iteration — the direction is exactly downhill and the step is short enough not to overshoot the descent. The loss curve is therefore smooth and monotone by construction; any bump in it is a bug or a step size past the threshold. A stochastic step points downhill only *in expectation*. On a given iteration the sampled direction can be tilted enough that the true objective goes up. Averaged over many iterations the drift is downward, which is why the curve trends down while individual points scatter around the trend. There is a second, purely cosmetic source of wiggle that candidates often miss: the number usually plotted is the loss of the current mini-batch, computed on different examples every step. Some batches are intrinsically harder than others, so the plotted series jitters even when the parameters barely move. Plotting a running average, or evaluating a fixed held-out set periodically, separates the two effects. ## The noise ball The important consequence is what happens at the end. Approaching the optimum, the full gradient shrinks to zero, so full-batch steps shrink to nothing and the iterates settle. The stochastic gradient does **not** shrink to zero: even at the exact minimiser of the average loss, individual examples still pull in different directions, so the sampled gradient has magnitude around `sigma / sqrt(b)`. With a constant step size the iterates keep taking steps of that size forever and end up wandering inside a region around the optimum rather than converging to it. The size of that region grows with the step size and with the gradient noise, and shrinks as the batch size grows. That gives a clean operational reading: if the smoothed loss curve has flattened, halve the step size. If the plateau drops to a new lower level, you were sitting at the noise floor. If it does not move, you are genuinely near an optimum of the objective you are optimising. ## Why anyone accepts the noise Cost accounting. One full-batch step costs n per-example gradient evaluations. For the same compute you can take `n / b` stochastic steps. Early in training the parameters are far from any optimum, and a rough estimate of the direction is more than good enough to make progress; many cheap approximate steps beat one exact step by a wide margin. Only near the end, where the true gradient is small compared with the noise, does the accuracy of the estimate start to matter, and by then most of the useful progress has been made. This is why stochastic steps dominate at scale even though full-batch steps have the stronger per-iteration guarantee. There is a second practical reason: a full-batch step requires touching the whole dataset before any parameter changes, which is impossible for data that does not fit in memory or arrives as a stream. A mini-batch step needs only b examples at a time. ## Diagnosing a run - Noisy but downward-trending curve, with a smoothed version falling: normal and healthy. - Noisy curve whose smoothed version is flat: at the noise floor or at an optimum — distinguish by cutting the step size. - Noise amplitude growing over time: the step size is too large for the local curvature; this is instability, not sampling noise. - Curve that looks smooth but stalls high: check that the batches are actually being resampled, and that the estimate is unbiased — sorting the data and reading it in order makes consecutive batches systematically unrepresentative and breaks the unbiasedness that the whole argument rests on. ## What to say in an interview Name the three pieces in order: the mini-batch gradient is unbiased but has variance about sigma squared over b; steps therefore descend only in expectation, so individual steps can go uphill; and near the optimum the noise does not vanish, so with a constant step the iterates hover in a region whose radius grows with the step size and shrinks with the batch size. Then close with the reason it is worth it — n over b times as many steps for the same compute.