A report sums per-group results into daily totals. Which grouping shapes may it sum safely, and which will double count?
answer
- ask whether the groups partition
- each record in exactly one
- overlap inflates by the fan-out
- per key only, for gap-closed
- identity is key plus span start
basics
~20 sOnly shapes whose groups partition the input may be summed: back-to-back equal spans globally, and gap-closed groups per key. Overlapping spans cover each record length-divided-by-step times, so summing them inflates the total by that factor.
solid answer
~50 sThe test is whether the groups **partition** the input. Back-to-back equal spans do: every record's moment lies in exactly one, so their totals add up to the true figure. Gap-closed groups do too, but only within one grouping key and only once any bridging of adjacent groups has settled. Overlapping spans do not: a record lies in length-divided-by-step of them, so a naive sum inflates the total by that factor, and no correction after the fact recovers it — the duplication is in the definition. The recoveries are to take only every nth overlapping group so the ones taken do not overlap, or to compute totals from a separate non-overlapping grouping. Two further traps: a result row is identified by the grouping key **together with the span's start**, never the key alone; and averaging per-group rates across gap-closed groups of unequal extent produces an unweighted mean of incomparable numbers.
go deeper
Remember one test: results can be added only when each record went into exactly one group. Overlapping spans break that, because one record is counted in several of them.
Explain why the inflation factor is only roughly the fan-out and why it cannot be corrected later: nothing in the emitted results records which records were shared between groups.
Design for it. Define a separate non-overlapping grouping when a correct total is needed, key result rows by the grouping key plus the span start, and carry each gap-closed group's extent so consumers can weight it.
Treat it as a contract with the consumers of the numbers. Publishing a rolling series without saying it must not be summed guarantees that somebody sums it, so the shape and its re-aggregation rule belong in what is published, not in the pipeline's code.
## The property that decides it Re-aggregation is safe exactly when the groups **partition** the input: every record belongs to one and only one group. That is a property of the shape, not of the aggregate and not of the engine. | shape | partitions the input | may totals be summed | |---|---|---| | back-to-back equal spans | yes, across all records | yes | | fixed length restarted every step | no; each record is covered several times | no | | gap-closed grouping | yes, within one grouping key, once bridging has settled | yes, per key and across keys | Ask the partition question before the arithmetic question, and most re-aggregation bugs never get written. ## Where the overlapping shape breaks a total With a length of ten minutes and a restart step of one, every record is inside ten groups. Summing group totals therefore reports about ten times the truth. Three things make this a production bug rather than a caught mistake: - The inflation factor is roughly, not exactly, the fan-out — records near the start and end of the reporting period are covered fewer times, so the number is wrong by an amount nobody can reconcile. - The result looks plausible. A tenfold count of page views is not obviously absurd to whoever reads the dashboard. - It cannot be fixed downstream. Nothing in the emitted results says which records were shared between groups, so no later correction recovers the true total. The two honest recoveries: 1. **Take a disjoint subset.** Keep only every nth group, where n is the fan-out, so the kept groups do not overlap each other. This works and throws away the smoothness the overlap was bought for. 2. **Compute the total separately.** Run a non-overlapping grouping alongside the rolling one. Two definitions, two outputs, each honest about what it is for. ## Where the gap-closed shape breaks a comparison Gap-closed groups do partition, so their **counts and sums** add correctly. What they break is anything per unit of time or per group: - Extents are unequal, so a rate computed inside each group is not comparable across groups; an unweighted mean of those rates weights a nine-second group the same as a nine-hour one. - Nothing lands on a calendar grid, so a daily figure needs the groups re-bucketed by, say, the moment each started — and a group straddling midnight has to be assigned or split by a stated rule. - If a later record bridges two groups for a key, a result already published is superseded rather than supplemented. Counts summed over a set that mixes superseded and current rows are wrong, and whether bridging happens at all varies between engines. A separate subtlety: if a maximum extent was layered on to stop one busy key running forever, then a single stretch of activity arrives as several rows, so counting groups is not counting visits unless consecutive rows for a key are stitched. ## The identity of a result row When more than one result can exist for a grouping key, the key alone does not identify a row. - For equal spans and for overlapping spans, a row is identified by the **grouping key together with the span's start moment**. A store keyed on the entity alone will have each interval overwrite the last, and the report will show only the most recent one while claiming to show a day. - For gap-closed groups, the natural identity is the key together with the group's start, with the extent carried alongside so consumers can weight or bucket correctly. ## The aggregate itself One caveat that is not about shape: summing group results assumes the aggregate is one that combines. Counts and sums do; an average has to be recombined from its total and its count rather than averaged; a distinct cardinality cannot be recovered from per-group cardinalities at all without a summary built for the purpose. Whether a given aggregate can be kept as one mergeable value is a neighbouring subject, and the right move in an interview is to name the constraint and say where it is settled, rather than to work it out here. ## Saying it cleanly *Sum only what partitions. Equal spans partition globally, gap-closed groups partition per key, overlapping spans do not partition at all and their totals are inflated by roughly the fan-out. Where I need both a rolling view and a correct total, I define two groupings rather than deriving one from the other.* That answer is about the definitions, so it holds whichever engine is under discussion — and it is the part a reviewer of the report would otherwise have to discover from a number that is wrong by an order of magnitude.
- A dashboard needs both a rolling view and a correct daily total. What do you build?Two groupings from the same input: an overlapping one for the rolling view and a non-overlapping one for the total. Deriving the total from the rolling results is either wrong, if summed naively, or a disjoint subsample that discards the smoothness the overlap paid for. Two definitions are cheaper to explain and impossible to misread.
- How should a daily figure be built from gap-closed groups that straddle midnight?State a rule and publish it. Either assign each group wholly to the day it started, which is simple and slightly wrong at the edges, or split the group's contribution across days, which requires the aggregate to be splittable and so is often impossible. The failure is not choosing; it is leaving consumers to discover which rule was used.
- Why is the inflation from summing overlapping results only approximately the fan-out?Because coverage is uneven at the edges. Records near the start of the reporting period fall inside fewer spans, since some of the spans that would have covered them started before the period, and the same happens at the end. The interior is inflated by the full fan-out and the boundaries by less, so the total is wrong by an amount that cannot be divided out.
saying these in an interview costs you the question
- Sums overlapping results into a daily total
- Averages per-group rates across groups of unequal extent
- Identifies a result row by the grouping key alone
- Assumes any per-group total can be re-aggregated
- Thinks an inflated total can be corrected after the fact
- Averages per-group averages instead of recombining totals and counts