A job's claim that nothing older than some timestamp is still coming: at what points can it advance, and what does that decide?

## Two independent properties, routinely confused A completeness claim - the timestamp a job carries with its records asserting nothing older is still expected - has two separate settings, and candidates merge them constantly: - **How far behind** the newest observed moment the claim is held. This is the wait, and it decides how much of a disordered source is admitted before a period is treated as finished. It is a correctness dial. - **How often** the claim is recomputed and propagated. This is granularity, and it decides only the extra delay between the instant a period *could* be declared finished and the instant the job says so. It is a latency-and-overhead dial. Increasing the frequency of advance never makes a result more complete. Increasing the offset never makes it fresher. Saying that out loud is most of the answer. ## Where the advance points actually are | execution model | where the claim may advance | what follows | |---|---|---| | one record at a time through the operators | between any two records, in principle | finest possible granularity, at the cost of propagating a value constantly; damped in practice by a short emission interval | | continuous work as a rapid succession of small finite jobs over whatever arrived | at the boundary between one small job and the next | time-based results step at that interval; no time-bounded decision can move more finely | | a single pass over a finished, bounded input | nowhere - there is no running claim | the end of the input is the boundary, and completeness is observed rather than asserted | This is the part of the subject where an engineer's instinct is most likely to be one product's instinct. Someone whose experience is entirely record-at-a-time will describe a claim that moves smoothly and continuously, and will be describing something the repeated-small-jobs model cannot do. Someone whose experience is entirely the repeated-small-jobs model will assume time cannot move within a unit of work, and will be describing a limitation the record-at-a-time model does not have. Name the model you mean. ## Why a job damps its own claim emission Even where the runtime permits an advance between any two records, designs rarely take it, for a mechanical reason: 1. The new value has to reach every downstream operator instance, so a wide job multiplies each emission by its width. 2. Where records are redistributed across the network, each receiving instance must recombine the claims of all the instances that can send to it, so the recombination work also scales with the product of the two widths. 3. Nothing downstream consumes granularity finer than the decisions it takes. A grouping over one-minute periods gains nothing from a claim that moves every microsecond. So a short periodic emission is the usual compromise: the value is recomputed on a timer, and the interval is added to the delay before any period is treated as finished. Some designs instead attach the claim to distinguished records inserted into the flow, which ties advance to traffic rather than to the clock. ## Where the claim is first produced The claim usually originates at the point where the pipeline assigns each record its moment - the source-facing step that decides which payload field or piece of source metadata becomes the record's time. That step observes the moments, applies the chosen wait, and emits the claim into the flow. Downstream operators do not re-derive it from payloads; they combine what reached them from upstream, which is why a claim never becomes more advanced as it travels, only less. ## Consequences worth naming - **The latency floor of a time-based result is the advance point plus the wait.** A team that shortens the wait and still sees results arriving in visible steps is looking at the advance granularity, not the wait. - **A change in advance granularity is cheap to try and easy to over-credit.** It moves the delay by at most one interval; it moves completeness not at all. - **A bounded job needs none of this.** If the question is about a finished input, the honest answer is that the mechanism is unnecessary, and saying so is better than describing machinery that will not run. ## What to say Answer in the shape of the table: state that the advance point is a property of the execution model, name the three models and what each permits, then separate granularity from the wait and say which of the two the question is really about.