Two training runs with identical seeds and identical data order diverge by step 4000 - why?

## Seeds do not control arithmetic A seed fixes which random *values* are drawn. It says nothing about the order in which a parallel machine adds numbers together. Two runs can request exactly the same initialization, exactly the same batch order and exactly the same dropout masks, and still compute slightly different gradients - because the reduction that produced those gradients was executed in a different order. ## Why order matters at all Floating-point addition is not associative: `(a + b) + c` and `a + (b + c)` can differ, because each intermediate result is rounded to a finite number of digits. For a handful of similar-magnitude numbers the difference is in the last bit. For a sum over thousands of terms with mixed magnitudes it can be larger. Nothing here is a bug; it is the arithmetic behaving as specified. ## Where the order varies between launches - **Atomic accumulation.** A scatter-add of gradients into an embedding table is the canonical case. A batch of click-through examples may reference one popular ad id hundreds of times, so hundreds of threads atomically add into the same embedding row. Whichever thread wins the race first is a scheduling accident, and it changes from launch to launch. - **Multi-stage reductions.** A large sum is split across blocks, each producing a partial result that is combined later. If the split or the combination order depends on how work was distributed at runtime, the total's last bits move. - **Algorithm autotuning.** Some libraries benchmark several implementations of the same operation on first use and keep the fastest. A machine under different load can pick a different implementation, with a different internal summation order, on a different day. - **Asynchronous combination across replicas.** When partial gradients arrive in a different sequence and are summed on arrival, the sum differs in the last bits even though every contribution is identical. ## Why a last-bit difference becomes visible Training is a long feedback loop, and small perturbations do not stay small. A gradient that differs in its last bits produces weights that differ in their last bits, which produce activations that differ slightly more, and so on for thousands of steps. Three things accelerate the separation: 1. **Discrete decisions.** An argmax, a top-k selection, a hard routing or gating choice, or a threshold comparison converts a microscopic numeric difference into a completely different downstream computation the moment two candidates are nearly tied. 2. **Operating near instability.** At a learning rate close to the edge of divergence, or during a loss spike, the dynamics amplify perturbations quickly rather than damping them. 3. **Path-dependent bookkeeping.** Anything that records a best-so-far value - early stopping, best-checkpoint selection - can flip to a different decision on a tie that is decided in the last bits. The typical signature is exactly what the question describes: identical for hundreds of steps, then identical to fewer and fewer decimals, then plainly different curves. ## Diagnosing it correctly Compare the two runs step by step and find the *first* step where they differ. - **Differ at step 1** - this is not reduction order. Something random is unpinned (initialization, batch order, an augmentation or dropout stream) or the two runs are not actually running the same code or data version. - **Identical for a while, then diverge** - consistent with nondeterministic accumulation. Confirm by rerunning the same step in isolation on the same inputs and checking whether one operation's output is bitwise stable across repeats. Skipping this step is how teams spend a week hunting a data bug that was arithmetic all along - and, worse, how a real bug gets waved away as "just nondeterminism". ## What to do about it If you genuinely need bitwise reruns, most stacks offer deterministic algorithm selection, deterministic (ordered or sort-based segmented) reductions in place of atomics, and a way to pin autotuning. Expect to pay in throughput, sometimes substantially, and expect some operations to have no deterministic implementation at all. When is it worth paying? Bisecting a regression to a specific commit; unit-testing a training step; an audit that must reproduce a number exactly; debugging a rare crash that appears at one step. When is it not? Routine training, where the honest response is different: if a conclusion cannot survive arithmetic reordering, it was never going to survive a change of seed either, and the answer is to measure the effect across several runs rather than to chase bit-identical ones.