Yesterday's records are pushed through the same job again this morning, and a whole day of history lands in one hour's bucket. Which moment was it grouping by, and which would have been immune?

## Carried moments and generated moments The cleanest way to separate the three candidate moments is to ask a single question: **is this value inside the record, or is it produced at the instant the record is handled?** - The moment stamped in the payload — *occurrence time*, what the producer recorded as when the thing happened — is **carried**. It was written before transmission and is part of the bytes. - The moment the receiving system wrote the record down — *receipt time* — is **carried once the record is stored**, because the stamp was applied on acceptance and travels with the record from then on. - The wall clock of the worker that reached the record — the term of art is *processing time* — is **generated**. It exists only at handling. There is no such field anywhere; there is only a clock being read. A second pass over the same records is precisely the experiment that separates them. | grouping moment | second pass over the stored records | records fed through the receiving system again | |---|---|---| | payload stamp | identical buckets | identical buckets | | receipt stamp | identical buckets — the original stamp is still on the record | re-dated — a new stamp is applied | | worker's wall clock | re-dated into the hour of the second pass | re-dated into the hour of the second pass | So the absolute statement holds for exactly one row: the worker's clock never survives a second pass. The payload stamp always does. Receipt sits between them and depends on how the second pass is fed, which is why it is worth knowing how your own re-run is wired before relying on it. ## What the collapse actually looks like The damage is not subtle once you know to look for it, and it is completely silent until then: 1. A day that already had its counts now has approximately zero, because those records were re-dated out of it. 2. One hour of the current day holds a day's worth of volume, producing a spike that looks like a traffic event or an attack. 3. Any aggregate built on top — a rolling average, an anomaly threshold, a figure someone already reported — is now wrong in two places for one cause. 4. Nothing reports an error. Every record was processed successfully; they were simply filed under the wrong moment. Whether the old numbers are *replaced* or *duplicated* depends on how the destination handles a re-written result, which is a separate matter. The re-dating happens regardless. ## Why this is the decisive argument for the payload moment Every other argument for grouping by the payload moment is about accuracy at the margins: records near a day boundary, a source that buffers. This one is about whether the pipeline is **repeatable at all**. A pipeline that produces a different answer from the same input depending on when you ran it is not a computation over data; it is a computation over data and the calendar. Reprocessing is a completely ordinary operation — a code fix, a widened output, a destination rebuilt — and a pipeline that cannot survive it has an operational ceiling that will be discovered at the worst possible moment. ## What varies, and what does not The mechanism is the same everywhere, but the surroundings differ: - On an engine that runs continuous work as a rapid succession of small finite jobs, the worker's clock moves in steps at job boundaries rather than smoothly; the re-dating is identical, just chunkier. - On a record-at-a-time engine the clock is read per record, so a long second pass smears the history across however long the pass takes rather than into a single instant. - On an engine that only ever processes a finished bounded input, there is no running notion of now at all, and a grouping that depends on the run's clock re-dates the entire input to the run — the effect is at its most total precisely where people assume time is not involved. In all three the fix is the same and it is not a setting: nominate the moment stamped in the payload, and where no such stamp exists, nominate the receipt stamp and make sure the re-run re-reads stored records rather than pushing them back through the system that applies it. ## Using it as a test Because the behaviour is categorical, it doubles as a two-minute diagnostic on any pipeline you have inherited: run the same recorded input twice, some hours apart, into a scratch destination and compare the bucket boundaries. Identical means the grouping is on a carried moment. Shifted means it is on a generated one, and every historical number that pipeline produced was a statement about when the job ran.