Measured disorder shows 99.9% of a source's records within three hours and a tail reaching nine days — how long should the job wait, and what covers the rest?

## Two populations, not one distribution A distribution with 99.9% inside three hours and a maximum at nine days is almost never one phenomenon with a long tail. It is two populations that a single curve has fused: | population | typical cause | share | can waiting cover it? | |---|---|---|---| | body — seconds to minutes | transport jitter, retries, producer batching, merged parallel inputs | the overwhelming majority | yes, cheaply | | shoulder — minutes to hours | degraded regions, a slow producer fleet, one lagging parallel input | small but real | sometimes, at rising cost | | tail — days | producers that were disconnected and reconnected | a handful per million | no, at any price you would pay | The first discipline is to confirm the split rather than assume it: break the measurement down per producer release, per geography, per network type and per parallel input. If the tail localises to one identifiable fleet, you have a population, and populations are treated separately. If it is uniformly smeared across every breakdown, you have one heavy-tailed phenomenon and the treatment is the same but the diagnosis is different. ## Why the tail cannot be bought with waiting Waiting is a uniform tax. Every group is held for the full duration before it can be treated as finished, so a nine-day hold would make **every** figure the job publishes nine days old — for the benefit of a few records per million. It also holds nine days of open groups' accumulators, which for a source of any volume is not a state problem but a storage system. Stated as a rule worth saying out loud in an interview: *you pay for the tail on every record, and you receive it on almost none*. That is an economic objection rather than a mechanical one — nothing stops a job holding for nine days except that no consumer would accept the result and no cluster would accept the state. ## Where to put the boundary 1. **Set the hold at the shoulder of the body.** Take the point where the curve flattens — the region past which each additional order of magnitude of waiting buys a diminishing fraction of a percent — and stop there. On the distribution described, that is minutes, not the three hours the 99.9th percentile might tempt you into. 2. **Check the hold against the output's purpose.** An alerting path may sit well below the shoulder and accept less; a figure nobody reads until the next morning may sit above it for free. 3. **Write down the residual.** "At this hold, roughly N records per million arrive after their group was eligible to close" is the sentence that makes the choice reviewable. Without it, the number is folklore within two quarters. ## What covers the remainder instead The tail is handled outside the waiting mechanism: a later pass re-derives the affected time ranges from the retained, re-readable input, once the reconnecting producers have caught up. The mechanics of running history back through the same logic are their own subject; what belongs to this decision is the *policy* around it: - **Two figures, two meanings.** The streaming figure is provisional and fast; the re-derived figure is slower and more complete. Publishing them as if they were one number guarantees that someone will notice a discrepancy and lose confidence in both. - **Say which is authoritative for which use.** Operational dashboards and alerts run on the provisional figure; anything that settles money, reports externally or feeds a model's training set runs on the re-derived one. - **Bound the reconciliation window.** A pass that may revise the last nine days needs the retained input and the downstream capacity to absorb a revision; if the destination can only ever append, the pass is a plan rather than a capability, and that constraint must be discovered before the policy is announced, not after. ## The part that is genuinely a lead's call The wait itself is a setting. The decisions that outlast it are organisational: - **Who may change the hold**, and what evidence a change requires — the distribution's current shape, not a complaint about a missing record. - **What the lateness promise says** to consumers: how long after a period ends a figure may still move, and how they learn it did. - **When the distribution is re-measured**, and what is re-justified when it shifts — a client release that changes background upload behaviour can move the body by an order of magnitude with no change on your side. - **When to refuse the requirement.** "Every record must be in the streaming number" and "the number must be available in one minute" cannot both hold on a source with a nine-day tail. The honest answer is to say so and offer the two-figure design instead of quietly setting a longer hold that satisfies neither. ## What varies between engines The shape of the second pass depends on the runtime you are on, so do not assert one design. Where the runtime executes continuous work as a succession of small finite jobs, the reconciliation is often the same code pointed at a stored range and is nearly free to build. Where it is record-at-a-time with long-lived per-key state, re-deriving a nine-day range may mean a separate run with its own retained state rather than a replay through the live job. And where the existing model is already a single pass over a finished bounded input, reconciliation is simply the next scheduled run of what you have — which is why a team that had a nightly computation and moved to continuous processing should keep the older pass rather than delete it.