Two totals of the same floating-point column, one accumulated strictly in order and one block-by-block, disagree — why?

## Why addition rounds, and what that breaks A floating-point value stores a fixed number of significant digits and a scale. Adding two of them lines the scales up, adds, and then **rounds the result back** to the nearest representable value, because the exact answer usually needs more digits than the representation has. That single rounding per operation is enough to break associativity. Concretely: once a running total has grown large, an addend smaller than one unit in the last place of that total changes nothing at all — the exact sum rounds straight back to the total you already had. Add a million such small values one at a time to a large running total and you can lose most of them. Add the small values to each other first, and their combined weight is large enough to survive being added to the big one. Addition is still **commutative**: swapping the two operands of a single addition gives the same answer. It is the regrouping that changes things, and candidates who reach for commutativity here are answering a different question. ## The two orders, and which is more accurate | accumulation | what the partial sums look like | typical error growth | |---|---|---| | strictly sequential, left to right | one total that keeps growing, added to small values | grows roughly with the number of values | | blocked or pairwise | many partial sums of similar magnitude, combined at the end | grows far more slowly | | compensated — a second value tracks what each rounding discarded | one total plus a correction term | very small, at some extra cost per value | The point worth carrying out of that table is that **the loop's answer is not the truth**. The reference is the exactly rounded total of the true values, and a blocked pass is usually nearer to it than a long sequential one. Treating the original loop as ground truth and "fixing" the rewrite to reproduce it locks in the worse answer. ## What changes the grouping in practice The grouping is not something you chose once in the source text. It moves when: - the block size of a deliberate loop over blocks changes; - a pass is spread across a different number of threads, so the partial sums are cut differently; - an expression is fused into a single pass rather than materialised step by step; - a tool switches internally between a straight accumulation and a pairwise one according to length. So the answer is deterministic for a fixed configuration and can move when any of that configuration moves. That is worth knowing before someone reports it as flakiness. ## What is affected and what is not - **Floating-point values:** affected. Two groupings differ in the low digits, and the difference grows with the number of values and with the spread of their magnitudes. - **Integers within the representable range:** not affected. The additions are exact, nothing rounds, and any grouping gives the same result. - **Fixed-point decimal amounts held in their smallest whole unit:** not affected, for the same reason. This is why monetary amounts are so often held as whole minor units — it makes the accumulation order irrelevant. - **Integers where the accumulator can overflow and the arithmetic traps rather than wrapping:** the order can decide whether the overflow is reached at all, which is a different hazard with the same smell. ## What this means for a rewrite When a per-record loop is replaced by a whole-column or blocked pass, expect the totals to differ in the last digits, and plan for it: 1. Compare with a **relative tolerance** — the difference scaled against the magnitude of the total — rather than an equality. An absolute threshold tuned on thousands will fail on billions. 2. Let the tolerance grow with the number of values accumulated, because the error bound does. 3. Where the number has to be exact and reproducible across configurations, change the representation rather than the tolerance: hold the amounts in an exact type and the whole problem disappears. 4. Where accuracy matters but the values must stay floating point, a compensated accumulation — keeping a second running value for the part each rounding discarded — buys most of the accuracy back for a modest per-value cost. ## The sentence to avoid "The total of a column is the same number however the pass is organised" is true of exact representations and false of floating point. It is a comfortable thing to assume and it is the reason the disagreement gets escalated as a bug. Say which representation you are talking about, and the claim becomes both true and useful.