A large file is read in pieces of unequal row counts and each piece's mean is recorded. Why is the average of those piece means not the file's mean?

## A mean is two numbers wearing one name The value a report shows as *the mean* is derived: a **total** divided by a **count**. The instant that division happens the count is gone from the result. Nothing about the number `47.2` says whether twelve rows or twelve million stood behind it. While the whole input sits in one place the loss costs nothing, because the division happens exactly once, at the very end. It becomes a defect as soon as the input is handled a piece at a time in one process: each piece divides early, discards its own count, and hands on a bare number that has forgotten its weight. Averaging those bare numbers gives **one vote per piece**. The mean of the input gives **one vote per row**. The two answers coincide only in the special case where every piece contributed the same number of values - and a real pass almost never produces that, because the final piece is short, because the input was cut on a natural boundary such as one file per day, or because rows were filtered before the aggregate ran. ## The arithmetic, with numbers on it | piece | values counted | piece mean | contribution to the total | |---|---|---|---| | A | 100,000 | 10 | 1,000,000 | | B | 200 | 500 | 100,000 | | whole | 100,200 | **10.98** | 1,100,000 | The average of the two piece means is `(10 + 500) / 2 = 255`. The mean of the input is `1,100,000 / 100,200`, which is about `10.98`. The gap is not a rounding artefact and it does not shrink as pieces are added: it is a different quantity. Cutting the same input into a thousand pieces instead of two does not converge on the right answer, it only redistributes the wrongness. The direction of the error is set by whatever happened to land in the small pieces, which is why the mistake survives review - the number looks plausible and moves in a plausible direction. ## What the fold has to carry between pieces 1. **A running total** of the values that were added. 2. **A running count** of how many values were added into that total. 3. **One division, after the last piece** - never inside a piece. Combining two partial results is then plain addition of the pair: `(total_a + total_b, count_a + count_b)`. If you would rather keep a mean than a total, the equivalent statement is to weight each piece's mean by its own count: `mean = sum(mean_i * n_i) / sum(n_i)`. Both are the same arithmetic. What neither of them does is treat the piece as the unit of weight. Both carried quantities are **fixed in size** - one number and one integer, whatever the input's length. That is the property that makes this fold safe to run over an input of any size: the state travelling from one piece to the next does not grow with the data. ## The count has to count the right thing The count you carry must be the number of values that actually entered the total, not the number of rows in the piece. If the addition skipped cells holding nothing - however the design marks absence, whether as a reserved bit pattern inside the number itself or as a separate flag kept alongside the column - then the piece's row count and its contributing count are two different integers, and pairing the wrong one with the total gives a denominator describing a larger set of rows than the numerator does. Report the pair that belongs together and the merge stays exact. A piece that contributed nothing needs the same care: it merges as `(0, 0)`, not as a mean. No mean exists for it, and inventing a zero mean for an empty piece adds a fictitious observation to the fold. ## Where this hides in real work - **The short tail.** Nine pieces of a million rows and a tenth of forty thousand: the bug is present, the error is small, and it is systematic rather than noisy, so it survives every eyeball check. - **Pieces cut on a boundary.** One file per day, per region or per upload - the counts differ by orders of magnitude, and the unweighted average reads as a per-file average, which is occasionally what somebody wanted and is never what *the mean* means. - **Per-key means.** The same mistake reappears one level down: folding a mean per key across pieces needs a total and a count per key, not a mean per key. - **Anything else built on a division.** A rate, a share, a percentage, a per-unit cost: each needs its numerator and denominator carried separately and divided once. ## Where designs differ Some tools hand back a bare number from a per-piece aggregate; others hand back an object that carries labels with it. Where the partial results carry labels, adding two of them pairs values **by label** and yields the union of both sides' labels, so a key one piece never saw turns up with nothing in it; where the partials are plain positional containers, values pair **by position** and the two sides must already be in the same order. Establish which you have before writing the addition, because both designs will cheerfully return something. Separately, some folds skip the running total and update the mean in place - `mean <- mean + (piece_mean - mean) * n_piece / n_total` - which still carries a count and earns its keep when a fixed-width total would overflow, or when the magnitudes are large enough that adding one more small value to a very large total stops changing it.