A rolling 60-sample mean slowly diverges from a freshly recomputed mean after days of uptime — why?

## The real lesson: incremental state has no self-correction Every fixed-window rolling aggregate trades work for a piece of retained state. The bargain is excellent asymptotically and carries a hazard that only shows up on the timescale of production: a recomputed window is *derived from the data every time*, so a transient fault produces one bad output and then the next output is fine. A rolling accumulator is derived from its own previous value, so a transient fault produces a *permanent* offset that every subsequent output inherits. "It was right yesterday and it is wrong today, and nothing in the input changed" is the signature. That framing is what an interviewer is listening for. The specific mechanisms below are the three ways the fault gets in. ## Mechanism 1: rounding residue in fractional arithmetic Floating-point addition is commutative but **not associative**, and each operation rounds its result to the nearest representable value. In a rolling sum you perform two rounded operations per slide: `sum + entering`, then `sum - leaving`. Neither residue is recoverable afterwards. Critically, these residues do not cancel on average in any useful way — the error in the accumulator performs something like a random walk, growing roughly with the square root of the number of updates in the benign case and linearly in the adversarial one (for example a stream whose values differ hugely in magnitude, where small readings vanish into a large running sum and then are subtracted out at full value, a catastrophic-cancellation pattern that leaves the accumulator visibly wrong within thousands of slides rather than millions). After a week at 1 kHz that is roughly 600 million updates. Even a residue in the last bits, compounded that many times, is enough to move a mean across an alarm threshold that was calibrated on the true value. ## Mechanism 2: overflow in whole-number arithmetic Whole-number accumulation is exact — there is no rounding at all — which is why fixed-point accumulation is the robust choice for sensor pipelines. Its failure mode is different and sharper: the *sum* of k values needs more range than the values do. Summing k values each up to `2^(b-1)` requires about `b + ceil(log2 k)` bits. For 60 readings that nearly fill a signed 32-bit range: `32 + ceil(log2 60) = 32 + 6 = 38` bits. An accumulator of the same width as the readings therefore wraps, and depending on the environment that is either a silent wrap to a large negative number or a fault. The alarm then compares a nonsensical mean against its threshold, usually in the direction of *not* firing, which is the worse direction for a safety monitor. The correct habit is to make the accumulator strictly wider than the value type — the same discipline that makes a midpoint calculation safe — and to reason about the bound as *value width plus log of the window size*, not by guessing. ## Mechanism 3: a broken add/remove pairing The invariant is that the accumulator equals the sum of exactly the k values currently inside the window. Anything that adds without subtracting, or subtracts a value that was never added, breaks it permanently: - a restart that reloads the accumulator from persisted state but not the circular buffer of live samples; - a runtime change of k that resizes the buffer without re-summing; - a dropped or duplicated reading in the ingest path; - an exception thrown between the add and the subtract. None of these are arithmetic problems, and none of them are detectable from the output alone without a reference. ## Diagnosis The cheap, decisive test is a periodic audit: every so often, sum the k live values directly and compare with the accumulator. If the gap grows monotonically with uptime, you have rounding residue. If it appears as one large jump and then stays constant, you have a wrap or a broken pairing at a specific event — correlate the timestamp with restarts and configuration changes. If the gap resets to zero whenever the service restarts, that is confirmation that the fault lives in retained state rather than in the input. ## Fixes, in order of preference 1. **Accumulate exactly.** Use a whole-number or fixed-point accumulator strictly wider than the reading type. This removes rounding entirely and turns the remaining risk into a bound you can compute. 2. **Re-derive periodically.** Every k slides, recompute the sum from the k live values. That costs O(k) once per k steps — amortized O(1) per slide, so the asymptotic win is untouched — and it caps drift and repairs any historical pairing bug. This is the single highest-value change and the one candidates most often miss. 3. **Assert the invariant in tests and in a low-rate production check.** Comparing incremental state against a fresh recomputation is exactly the property-based test this pattern deserves. 4. Compensated summation is worth knowing about for plain summation, but it does not cleanly rescue a *sliding* sum, because removal reintroduces error the compensation term was not tracking. Prefer exact accumulation plus periodic re-derivation, and say so rather than reaching for a clever summation trick that does not fit the access pattern. ## What not to say "Floats are fine, the errors cancel out" and "sixty numbers cannot overflow anything" are the two answers that end this line of questioning badly. The first mistakes zero-mean *inputs* for zero-mean *rounding residue*; the second forgets that the accumulator's range requirement is set by k times the value range, not by the value range.