A team sized its retained set by counting per-key accumulators alone and came out tenfold low — which held entries did that count miss?

## Why accumulators are the part everyone counts An accumulator is easy to picture: one running sum per key, a few tens or hundreds of bytes, obviously proportional to key count. It is also the smallest thing in the retained set, and the only category whose count is exactly one per key. Every other category is counted on a quantity the team did not put in the estimate, which is why estimates fail low rather than high, and usually by a large multiple rather than a few percent. ## The categories that were missed | Missed category | What is held | Entry count driven by | Why the estimate misses it | |---|---|---|---| | Records buffered awaiting a group | whole records, not a folded value | records per key inside the horizon | the team assumed a fold that the computation cannot do | | Join side | records from each input still matchable | records per key x the match horizon | the join was thought of as a lookup, not as retention | | Deduplication entry | one entry per distinct identifier | distinct identifiers inside the horizon | identifier cardinality is near the record count | | Pending per-key scheduled callback | key, moment, and any payload attached | keys with a wake-up outstanding | a wake-up does not feel like stored data | | Per-entry overhead | the holder's own bookkeeping per entry | total entries across all categories | it is invisible in the program's source | | Entries for keys that went silent | everything above, for keys that stopped arriving | horizon, not activity | the team counted *active* keys, not *live* ones | ## The buffered-records case is the biggest single miss Whether a computation can fold each record into one value or must keep the records decides the whole scale of the estimate. A sum, a count, a maximum or a small sketch can be folded: one entry per key, constant size. A median over the group, a list of the last ten items, a payload that must be emitted verbatim when the group is released, or any computation the engine cannot decompose, all force the records themselves to be kept. That turns "one entry per key" into "records per key", and records per key inside a long horizon can be four or five orders of magnitude larger. When an estimate is tenfold low, this is the first place to look. ## Pending wake-ups are storage A per-key scheduled callback is a wake-up the job registers against one key and one moment, which fires later. It is easy to think of it as a scheduling concept rather than data, but it is stored, it is carried in every durable snapshot, and it can grow without bound exactly like any other entry — one outstanding wake-up per live key, plus any payload attached to it. A job that registers a fresh wake-up per record without cancelling the previous one accumulates them per record rather than per key, which is a failure mode that never appears in an accumulator-only estimate. ## Live is not the same as active The team almost certainly counted keys seen in a recent interval. The retained set holds every key whose entries have not yet been removed, including keys that arrived once and never returned. Over a long horizon the silent keys usually outnumber the active ones. The rule that removes them, and the clock it is measured from, belong to a different node; the point here is only that the inventory includes them, and that an estimate built on active keys is an estimate of the wrong population. ## The direction of the correction differs by runtime How much each category costs depends on the processing model, and a candidate should say which they mean: - Where the runtime handles **each record as it arrives**, buffered records and join sides accumulate continuously and every access is charged individually. - Where the runtime cuts the endless input into **short finite chunks**, the same entries exist but are loaded and written once per chunk, so the cost shows up as bulk read and write per chunk rather than per record. - In the **two-phase disk-to-disk batch model**, none of this is retained between runs at all — the equivalent work is redone from the input every time, which trades retention cost for recomputation cost. ## How to redo the estimate 1. List every stateful step and, for each, name which of the six categories above it contributes to — not "it has state", but which entries and counted on what. 2. For each category, write the count as an explicit product of quantities you can source, and name where each number came from. 3. Add a per-entry overhead line across all categories and treat it as a range until it has been measured. 4. Re-run the estimate at the key cardinality expected twelve months out, because cardinality growth, not traffic growth, is what moves this number.