A daily order count runs over a continuous stream. Which three moments could each order be counted under, and which should the count use?
answer
- three candidates, not one clock
- happened, received, handled
- one of them is about your pipeline
- which value survives a second pass
- count the world, use the payload
basics
~20 sThree moments compete: when the order was placed, stamped in the payload; when the receiving system wrote the record down; and the wall clock of the worker that handled it. A daily count belongs to the payload moment.
solid answer
~50 sEvery record carries three candidate moments. **Occurrence time** is the moment the thing actually happened, written into the payload by whatever produced it. **Receipt time** is the moment the system that first accepted the record stamped it, before this job saw anything. **Worker-arrival time** is the wall clock read by the worker that eventually handled the record. A daily order count is a statement about when customers bought, so it has to group by the payload moment; anything else counts when the plumbing moved. Worker-arrival time is tempting because it can never produce a record that missed its bucket — the bucket and the record are stamped by the same clock — but that is exactly why it absorbs every delay silently, and why the same record lands in a different day if it is ever processed again.
go deeper
Be able to name the three moments out loud — stamped in the payload, written down by the receiving system, read off the worker's clock — and say that a daily business count uses the first.
Explain why they diverge exactly at boundaries, delays and second passes, and why nothing is ever reported as late under the worker's clock even though records genuinely were.
Show that the choice is a promise to consumers: grouping by the payload moment means accepting that a counted day can still change, and saying so before anyone builds on the number.
Frame it as which clock a published figure is defined on across teams, so two dashboards over the same source cannot quietly disagree by a day.
## Three moments, and only one of them is about the world A record that describes something that happened carries more than one time, and they routinely disagree by seconds, hours or days. - **Occurrence time — the moment stamped in the payload.** The moment the thing being recorded actually happened, as written into the record by whatever produced it: the checkout completed at 23:58. This is the term of art usually called *event time*. It is set furthest from your infrastructure and closest to the fact you are trying to count. - **Receipt time — when the receiving system wrote it down.** The moment the system that first accepted the record stamped it on the way in, before any part of this job saw it. It is later than occurrence and earlier than worker-arrival, and it is set by a machine on your side of the boundary rather than by the producer. - **Worker-arrival time — the wall clock when a worker reached the record.** The clock read at the instant the record was handled; the term of art is *processing time*. It is not a property of the record at all. It is a property of the run. | moment | who sets it | distance from the event | stable if the record is handled twice | |---|---|---|---| | occurrence (payload) | the producer of the record | none, if the producer is honest | yes — it travels inside the record | | receipt | the system that first accepted it | transmission and buffering delay | yes when the stored record is re-read; no if it is fed through the receiving system again | | worker-arrival | the worker that handled it | all of the above, plus queueing and any backlog | no — it is a fresh clock reading every pass | ## Which moment the daily count needs Ask what sentence the number is supposed to support. "How many orders did customers place on the 11th" is a claim about the world, so the only moment that answers it is the one stamped in the payload. Group by receipt and you have answered "how many orders reached us on the 11th"; group by worker-arrival and you have answered "how many orders this job happened to handle on the 11th", which is a fact about your own infrastructure that no one asked for. The disagreement is invisible when everything is healthy, because all three moments fall inside the same bucket. It becomes visible exactly when something interesting happens: 1. **A boundary.** An order placed at 23:58 that reaches the job at 00:03 is counted on the wrong day by the last two moments and the right day by the first. 2. **A delay.** A source that buffers for an hour, or a job that falls behind, shifts records forward under receipt and worker-arrival while leaving them where they belong under occurrence. 3. **A second pass.** Handling the same records again changes worker-arrival for every one of them, because that value is generated at handling time rather than carried. ## Why the worker's clock is the tempting wrong answer Grouping by the worker's wall clock has one genuine attraction: it can never produce a record that arrived after its bucket was finished, because the bucket boundary and the record's moment are read from the same clock. There is nothing to wait for and nothing to reconcile. That apparent tidiness is the defect, not a feature: it is not that the delays vanished, it is that they have been quietly folded into the numbers and nothing will ever report them. Worker-arrival time is a legitimate choice in a narrow set of cases — a health metric about the pipeline itself, an alert on a rate that is meant to describe now, a cache expiry — where the question genuinely is about the job rather than about the world. ## What varies between engines The three moments are a property of the problem, but how a given runtime presents them differs sharply, so do not assume the one you learned is the model: - Some runtimes will not let a time-based grouping compile until the pipeline has nominated where the moment comes from; others accept the grouping and use arrival, which means a pipeline that never chose has still chosen. - Where receipt time is available at all depends on whether the receiving system stamps records and whether that stamp is exposed to the job; it is not universal. - An engine that only ever processes a finished bounded input has no running notion of *now*, so worker-arrival there is simply whenever the run happened to be scheduled — the same input yields a different answer each time it is run. ## The practical rule Decide what the number is a claim about, then pick the moment that matches, and write the choice down where the next reader will find it. If the answer is "about the world", it is the payload moment, and the cost of that choice — that some records turn up after their bucket has been counted — is a cost you accept on purpose rather than a problem you avoided by picking the convenient clock.
- When is the worker's wall clock the right moment to group by?When the question is about the pipeline rather than the world: records handled per minute, an alert on the current rate, an expiry on something the job is holding. Those are claims about the run, and the run's own clock is the correct authority for them. The moment the number is reported as a business figure, the payload moment is the only defensible choice.
- If the payload moment is the right one, why does receipt time exist at all?It is the compromise when the producer's stamp cannot be trusted or is missing. It is set on your side of the boundary, so it cannot be dated a week in the future by a misconfigured producer, and unlike the worker's clock it is fixed once and stored with the record rather than regenerated on every pass.
- Two dashboards over the same stream report different order counts for the same day. What do you check first?Which moment each one groups by. It is the most common cause and costs one question to rule out: one is almost always counting when orders were placed and the other when records reached the system or when a job got to them. Until both name the same moment, reconciling the numbers is meaningless.
saying these in an interview costs you the question
- Says a record simply has a timestamp, as though one is obvious
- Groups a business count by the worker's wall clock because nothing is ever late
- Treats receipt time as close enough to occurrence to be interchangeable
- Cannot say who sets each of the three moments
- Thinks the worker's clock is wrong because machine clocks drift, not because it describes the run