In a job grouping an endless input into intervals, what rule, separate from the boundary, decides when a group's value goes downstream?

## Two independent decisions A computation over an endless input has no natural end, so the author imposes one. That imposed boundary — a **grouping interval**, a span of the chosen clock such that the records falling inside it produce one result — answers exactly one question: **which records belong together**. It says nothing about when anybody downstream sees the answer. The second, independent decision is the **firing condition**: the rule that decides at which moments the group's current value is handed downstream. One such handoff is **an emission**, and a single group may produce several of them over its life — an early one, a corrected one, a final one. | Decision | Question it answers | Change it and… | |---|---|---| | The boundary | Which records are in this group? | the arithmetic of the result changes | | The firing condition | When does this group's value leave the job? | the output latency and the consumer's obligations change | Conflating the two is the commonest error on this subject. A five-minute grouping does not mean "a number every five minutes", and it does not mean "a number five minutes late" either. ## What a firing condition can key on - **The group being declared finished.** The job carries a running assertion that no record older than a stated moment will still arrive — its **completeness claim**. When that claim passes the group's end, the group can be declared finished. How the claim advances, and what stalls it when a source goes quiet, is decided elsewhere in the job and is simply relied on here. - **An elapsed span on the clock the worker reads.** "Hand the current value downstream every ten seconds", regardless of how complete the group is. - **A count of records folded into the group.** "Hand it downstream every thousand records." - **Every input.** Restate the running value whenever it changes. - **A combination**, which is what production jobs usually run: early every few seconds while the group is open, once more when it is declared finished, and again for each record accepted during a grace period afterwards. ## Why one group emits more than once Two distinct mechanisms produce repeats: 1. **Early firing.** The group has not been declared finished, so the value handed downstream is provisional by construction: records that belong to the group have not arrived yet. 2. **A grace period after close.** A stated extra span during which a group already declared finished still accepts records and re-emits a corrected value. The price is that the group stays held that much longer. It follows that **emitting does not release what the group holds**. Output appearing downstream is no evidence that the job's memory has fallen — closing (about emission) and dropping the held records (about memory) are not the same moment. Where those held records physically sit, and what happens when they do not fit, is a separate subject. ## The granularity runtimes differ on How finely a firing condition can be honoured is a property of the runtime, not of the concept, and the designs in this class genuinely disagree: - Where a runtime **collects arrivals for a short span and runs one finite job over the collected set**, the smallest unit of progress is that span, so the earliest possible emission is the span's end; a count-based condition cannot fire between two spans. - Where a runtime advances **one record at a time through long-lived operators that carry values between records**, a group can be updated and handed downstream at the granularity of a single arrival. - Where the model is a **finite two-phase pass that materialises its intermediate result to disk between phases**, there is no open group at all: a grouped result over history means re-running the whole pass, and the "emission" is that run's output. An answer that states any one of these as the law of stream processing is wrong for most of the market. Say which design you are assuming, or say that the designs differ and how. ## What the consumer inherits from this choice Because the firing condition — not the boundary — decides how many times a group's value appears downstream, it also decides what the receiving system must be able to do. A sink that can only insert rows is correct only when exactly one emission per group ever reaches it. The moment a job fires early, or accepts a record during a grace period and re-emits, the receiving system is holding a value that will be superseded, and it must be able to apply that revision to a row it already has, keyed on an identity that distinguishes one interval from the next. That is why the firing condition is a contract rather than a tuning knob: changing it changes what somebody else's system has to be built for, and that change does not announce itself in the job's own output schema.