A job groups records by the moment stamped in the payload — where does one that arrives after its group closed go, and who would notice?

## The three things an arriving record can be Every record in a continuous job has a moment attached to it: the moment the thing being recorded actually happened, written into the payload by whatever produced it — often called **event time**, and here the **occurrence time**. A job that groups by that moment has to decide, *while it runs*, when a span of moments is finished. It does that with a **completeness claim** carried alongside the records — a *watermark*: a timestamp travelling with the data that asserts nothing older than it is still expected to arrive. The claim is a running guess the job makes, not a fact it reads off the data. Measured against that claim, an arriving record is one of three things: - **On time** — its moment is newer than the claim, so it lands in its group and there is nothing to discuss. - **Out of order** — it arrived after a record carrying a later moment, but the claim has not yet passed its own moment. It still lands in its own group. This is normal and costs nothing. - **Late** — the claim had already passed its moment, so the group it belongs to was already eligible to close before it showed up. Every late record arrived out of order; most out-of-order records are never late. Only the third case is a loss. ## Why the loss is silent A late record is not an invalid record. It parses, it satisfies the schema, its fields are sane — the one thing wrong with it is that it is *behind the job's own guess about time*. So none of the machinery that shouts at you fires: | Signal you already have | What it actually tells you | What it hides | |---|---|---| | Run status is green | No worker died and no step threw | Discarded records are not errors | | Output row count | How many groups were published | Nothing about records that missed one | | Input record count | How many records the job read | That some were read and then dropped | | Destination rejections | Which rows the destination refused | Records that never reached it | | Late-record counter | How many valid records missed a group | Which records, and what they were worth | The symptom, if anyone ever sees one, arrives weeks later from outside the pipeline: a finance or operations figure computed some other way is a fraction of a percent higher than yours, every day, and nobody can say since when. ## The counter, and getting one when you are not given one The **late-record counter** is an explicit count of records discarded for arriving after their group. Where the runtime publishes one, wire it to a dashboard and an alert on the *rate*, not the cumulative total — a total that only ever rises tells you nothing about today. Where it does not, you can produce the same number yourself: 1. Assign the record's moment explicitly, so you know which field the grouping is actually using rather than inheriting a default. 2. In a step *before* the grouping, compare that moment with the claim the job is currently carrying, where the runtime lets a per-record step read it; where it does not, compare against the newest moment seen so far minus the wait the pipeline has agreed to hold back. 3. Count every record that fails the comparison, and emit the count next to the group totals so a reader of the number can see both. The counter answers *how many*. It does not answer *which*, or *what they were worth* — a hundred dropped sensor readings and a hundred dropped payments are the same integer. ## What varies between engines This is one of the places where a sentence true of one engine is false of the next, so state the mode you mean: - **A grouping that republishes an updated running result** for as long as it retains the group will absorb the record rather than drop it. The counter then legitimately stays at zero while a previously published answer quietly changes — a different problem, not an absence of one. - **A grouping that emits once and releases the group** has nothing left to update, so the record has nowhere to go but away. - **A single pass over a finished bounded input** has no running claim at all, so during the run nothing is late by this definition; the same record simply turns up in the input some later run reads. - **Where the claim is readable from inside a per-record step** you can count lateness yourself; where it is not, your own comparison is an approximation of the runtime's. ## What the number is for A counter nobody watches is the same as no counter. The point of having it is that it converts an invisible, unbounded, unfalsifiable worry — *are our numbers short?* — into a quantity with a trend, which is the precondition for deciding what the pipeline should do about late records at all.