Shapes of a Window

## The definition and the arithmetic An overlapping grouping is stated with two numbers: a **length** — how much of the timeline one group covers — and a **restart step** — how often a new group begins. When the step is shorter than the length, a new group starts before the previous one has ended, so several are open at once. With a ten-minute length and a two-minute step, a new group opens at every even minute and each stays open for ten, so at any instant five groups are live. A record's moment lies inside every group that started within one length before it. That is the whole derivation: - groups open at each multiple of the step; - a group starting at time `s` covers moments from `s` up to `s + length`; - so the groups containing moment `t` are those with `t - length < s <= t`, and there are **length divided by step** of them. Ten divided by two is five. Fifteen-minute spans restarted every five minutes give three. Sixty-minute spans restarted every minute give sixty. ## When the ratio is not a whole number If the step does not divide the length evenly, the count is that ratio rounded up or down, and **it is not the same for every record**. With a ten-minute length and a three-minute step, some moments sit inside four groups and some inside three. Nothing is broken, but any explanation that leans on a single fan-out number is only true when the step divides the length. Requirements are usually written with a step that divides the length precisely so the fan-out is uniform. Two degenerate cases are worth holding: | step against length | overlap | a record belongs to | |---|---|---| | step shorter than length | length minus step | length divided by step groups | | step equal to length | none | exactly one group | | step longer than length | negative, so gaps appear | one group, or none if its moment lands in a gap | ## What the fan-out costs downstream The fan-out is not an internal detail; it is visible to everyone who reads the output. 1. **Output volume.** For the same input and the same length, an overlapping grouping produces as many results per grouping key as the fan-out factor. A sixty-minute figure refreshed every minute is sixty rows per key per hour, not one. 2. **Results cannot be added.** Each record is inside several groups, so summing the group totals multiplies the true figure by roughly the fan-out. A daily total must be built from a non-overlapping shape, or by taking only every nth overlapping group so the ones taken do not overlap each other. 3. **A row needs a compound identity.** Because a grouping key now has several live results at once, a result row is identified by the grouping key **together with the span's start moment**, never by the key alone. 4. **Neighbouring results are highly correlated.** Consecutive overlapping results share all but one step's worth of records, so a threshold evaluated on each of them will alert repeatedly for one underlying event. What to do about that belongs to whoever owns alerting, but the shape is what causes it. ## What is an implementation choice and what is not The fan-out count is a property of the definition and is the same everywhere. How a runtime *achieves* it is not: - some assign each arriving record to every group that covers it, so the record is touched once per group; - some maintain one partial value per step-sized slice and combine the relevant slices when a group is handed downstream, so the record is touched once; - a runtime that advances by collecting arrivals for a short span and running one finite job over the set cannot restart a group more often than that span; - the oldest model in this class, a finite two-phase pass materialising its intermediate result to disk, does not keep overlapping groups open at all — the equivalent is re-running the pass once per span over a finite slice of history. What this shape costs in memory, and whether the aggregate can be kept as one mergeable value per group or needs every record, is a neighbouring subject and should be answered there rather than guessed at here. ## Saying it cleanly in an interview Give the number, the derivation and one consequence: *five, because the length divided by the step is five; the same record is counted in all five, so these results are five times as many rows and must never be summed into a total.* That answer survives being asked about any engine, because none of it depends on one.