If you inject Gaussian noise into a full-batch gradient to imitate a small-batch run, what fails to transfer?

## Why the experiment is worth running The claim under test is that mini-batch gradient noise acts as an implicit regularizer — that a noisy run ends up somewhere different, and better, than the run that follows the exact gradient. Stated bare, it is a slogan. The obvious falsification is to keep everything about the deterministic run and add noise by hand: compute the exact full-batch gradient, add a Gaussian perturbation of comparable magnitude to each parameter's gradient, and see whether the small-batch behaviour reappears. Being able to say what such an experiment would and would not establish is the point of this question; there is no single right answer, but there are wrong ones. ## What matches Magnitude. A useful way to read a stochastic run is that the learning rate and the batch size together set a noise level, roughly proportional to the learning rate divided by the batch size — the standard continuous-time reading of stochastic gradient methods treats this ratio as a temperature. Injected noise has a variance you control, so you can dial it to sit at a comparable level. And with the magnitude matched, some of the qualitative behaviour does return. A deterministic run creeping on a plateau will start moving. An iterate sitting in a very narrow basin will be knocked out of it. If your only claim was that noise of a certain size prevents the optimizer from resting at the first near-stationary point it meets, injected noise supports it. ## What does not match **Direction.** The covariance of the mini-batch gradient is the covariance of the per-example gradients divided by the batch size. That matrix is nothing like a scaled identity. It is large along directions in which training examples disagree and near zero along directions where they agree, and near a well-fit solution it is approximately aligned with the local curvature of the loss, so the noise is strongest precisely in the directions in which the loss rises fastest. Isotropic Gaussian noise puts the same variance in every coordinate. In a model with millions of parameters, the overwhelming majority of directions are ones the loss is nearly indifferent to, so most of the injected energy goes into a random walk that neither explores anything useful nor is resisted by the loss. **Annealing.** Mini-batch noise is state-dependent. Its size is set by the spread of per-example gradients at the current parameters, and as the model comes to fit the training data those per-example gradients shrink toward zero — at a point that fits every training example, there is no disagreement left and the noise vanishes with it. A stochastic run therefore anneals its own noise as it succeeds. A fixed injected variance does not: it is the same at step one and step one million. The visible consequence is a training loss that stops falling at a floor set by the injected variance and the learning rate, with the iterate performing a random walk of fixed radius forever, which is not what a small-batch run does. **Shape of the distribution.** A mini-batch gradient is an average of `B` per-example gradients, so for moderate `B` it is approximately Gaussian by the central limit theorem — but only approximately, and the approximation is poor when per-example gradients are heavy-tailed, which happens with outliers, rare classes or hard examples. In that regime the run's behaviour is dominated by occasional very large steps, and a Gaussian injection with the same variance never produces them. ## Better controls If the goal is to establish whether the *structure* of the noise matters, design the control to isolate it. Estimate the per-example gradient covariance on a batch and draw the injected noise from that estimate rather than from an isotropic Gaussian; if the small-batch behaviour returns only under the covariance-matched injection, structure is doing the work and magnitude is not sufficient. Run a small-batch configuration but reuse a single fixed batch every step. Step sizes and cost stay at small-batch values while the sampling noise disappears entirely, which separates noise from every other consequence of a small batch. And hold the deterministic run's step size fixed while varying only the injected variance across a sweep, so you can see whether the effect is monotone in noise level or only appears in a narrow band — a slogan-level claim rarely survives that plot. ## What to conclude, and how to say it The defensible position is that mini-batch noise is not merely randomness added to a gradient: it is randomness whose covariance is determined by the data and the current model, which is why it concentrates where examples disagree and fades as they stop disagreeing. Injected isotropic noise reproduces the crudest consequence — that the run does not settle at the first stationary point — and reproduces neither the direction nor the schedule of the real thing. A candidate who says the injection works is not paying attention to the loss floor; a candidate who says it does nothing has not run it. The valuable answer names the three axes on which it can differ — magnitude, covariance structure and state dependence — and says which one their experiment actually controlled.