Why must global-norm gradient clipping run after the last micro-batch of an accumulation group?

## What the ordering actually is One optimizer update spanning `N` micro-batches has exactly one correct skeleton: 1. **Zero the gradient buffer** — once, at the top of the group (or immediately after the previous step). 2. **For each of the `N` micro-batches**: forward pass, compute the loss with the accumulation scaling applied, backward pass. Gradients add into the buffer. Nothing else happens here. 3. **Clip once**, on the fully accumulated buffer. 4. **Optimizer step.** 5. **Zero the buffer**, and **advance the learning-rate schedule** — once per step, not once per micro-batch. Everything that follows explains why steps 3 to 5 sit outside the loop. ## Why clipping cannot move inside the loop Global-norm clipping is a single operation on a single vector: take the gradient `g` the optimizer is about to consume, compute its norm, and if that norm exceeds the threshold `c`, replace `g` with `g * c / ||g||`. The defining property is that it **preserves direction** and only limits magnitude. Under accumulation, the vector the optimizer consumes is `g = g_1 + g_2 + ... + g_N`. Clipping inside the loop computes `clip(g_1) + clip(g_2) + ... + clip(g_N)` instead, and those are not the same object: - **Direction is no longer preserved.** Each `g_j` that exceeds the threshold is multiplied by its own factor `c / ||g_j||`. Micro-batches with large gradients are shrunk, others are not, so the sum is a *reweighted* combination of micro-batches, not a rescaled version of the true group gradient. Samples are effectively down-weighted according to which micro-batch they happened to land in — a data-order artefact. - **It fires far more often.** A micro-batch gradient is an average over fewer samples, so its norm is noisier and, in expectation, larger than the group gradient's norm. A threshold calibrated against full-batch norms will clip most micro-batches, meaning the safeguard has quietly become a permanent constraint rather than an occasional one. - **The resulting magnitude is unpredictable.** The clipped-and-summed vector can have a norm anywhere up to `N*c`, so you have not even achieved the bound clipping exists to enforce. The practical symptom is a run that trains but underperforms, with a loss curve that plateaus early. Nothing errors, and the gradient-norm value you log looks small and healthy — because you are logging post-clip micro-batch norms, which is not the quantity you think it is. Log the accumulated norm *before* clipping if you want a number that means something. (The one place per-micro clipping is deliberate is a privacy-preserving regime, where per-example gradients are clipped precisely so that no single example can move the weights much. That is a different objective, imposed on purpose, not the general-purpose stability safeguard discussed here.) ## Why the step and the zeroing sit outside too **Stepping inside the loop** turns accumulation back into `N` small-batch updates. The memory saving remains, the training dynamics you were trying to buy are gone, and because you also applied the accumulation scaling you are now taking `N` steps each `1/N` too small — which is not the same as one step of the right size, since the weights move between passes and momentum sees a different sequence. **Zeroing inside the loop** is the mirror bug: each backward pass wipes the previous one, so the update is computed from the last micro-batch alone, scaled down by `1/N`. Your effective batch is `m`, not `B`, and your learning rate is `N` times smaller than intended. The tell is a run that behaves like a tiny-batch run despite the configuration saying otherwise. **The learning-rate schedule** counts optimizer steps. Advancing it once per micro-batch runs the schedule `N` times too fast: a warmup that was supposed to span some number of updates completes in a fraction of them, and a decay reaches its floor long before training ends. The same applies to any step-counted logic — checkpointing intervals, evaluation cadence, early-stopping patience, and the total step budget the recipe specifies. When you turn on accumulation, every one of those counters needs to be reinterpreted in units of updates. ## Answering it well in an interview State the skeleton first, then justify the one line that is easiest to get wrong: clipping is defined on the vector you step with, and that vector does not exist until the group is complete. Add that the wrong ordering fails silently — no exception, just a slightly wrong direction and an over-eager safeguard — which is exactly why the ordering is worth memorising rather than rederiving under pressure.