Training Dynamics and Debugging

Floating-point addition is not associative, and parallel kernels sum partial results in an order that varies between launches. Last-bit gradient differences feed back through thousands of updates until the two runs are visibly apart.

Cross-entropy scores confidence, accuracy scores only the decision. As training continues the model becomes more confident everywhere, so the examples it still gets wrong contribute a large and growing penalty while newly-correct ones add almost nothing.

It makes every residual branch output exactly zero at step zero, so the stack starts as the identity map and the signal keeps the scale it entered with instead of growing block by block. The branches then switch on gradually.

Judge it on trend and consequence, not the level. A fraction still growing means an active pathology; a stable one is a capacity tax to weigh against the validation metric. Attribute cause by changing one knob and re-measuring.

A double descent plot does not show that scale replaces regularization: those sweeps are run under-regularized on noisy labels, and tuning the penalty at each capacity can flatten the peak away.

It breaks sharpness as a standalone explanation of generalization. Scaling one layer up and the next down computes the same function with the same test error, yet raw curvature measures change freely.

A range test measures one thing: the largest rate that keeps training loss falling over a couple of hundred steps. It is silent on generalization, on stability thousands of steps later, and on recipes whose good rate is already known.

Check that the loss and the gradient norm are finite before the update is applied, not after, because the optimizer's running buffers keep a NaN forever. Skip and log the step, persist the batch, and alarm on the skip rate.