Before choosing how long a job waits for out-of-order records, what do you measure on the live source, and in what shape?

## The quantity you actually want You are not measuring how slow the pipeline is. You are measuring how far out of order the **source** delivers, which is a property of producers and transport and is already fixed before your job reads anything. The per-record quantity is straightforward. Track the newest **occurrence time** — the moment the thing happened, stamped into the payload by whatever produced the record — seen so far on that input. For each arriving record, subtract its occurrence time from that running maximum. A record that arrives ahead of everything contributes zero. A record arriving when the front has already moved ten minutes on contributes ten minutes. Call that the record's out-of-orderness. That is the only quantity a wait can honestly be chosen from, because a wait is precisely a duration held behind the newest stamped moment the job has seen. Measuring anything else and then choosing a wait from it is choosing a round number and calling it analysis. ## Measure against receipt, not against your own worker's clock There are two tempting reference points and one of them is wrong. | reference point | what it actually measures | trustworthy while catching up? | |---|---|---| | occurrence time against **receipt time** (when the accepting system wrote it down) | the source and its transport | yes — both values are fixed before this job reads | | occurrence time against the **worker's wall clock** at handling | the source, plus this job's own backlog | no — a job eight hours behind reports eight hours of "disorder" | The test is simple: a job that is fully caught up and the same job an hour behind must produce the same disorder measurement, or the measurement means nothing. The second reference point fails that test, and it fails it exactly when you most want the number — during an incident. ## It is a distribution, not a constant Real sources do not have *a* disorder. They have a shape, and the shape is what you read. | statistic | what it is for | |---|---| | median | describes the healthy path and is nearly useless for choosing a wait | | 95th / 99th percentile | the shoulder — the region where a wait first looks affordable | | 99.9th percentile | where each further nine starts costing an order of magnitude of waiting | | maximum | the disconnected-producer population; tells you a wait will never reach it | Break the distribution down, because an aggregate curve hides the thing you need: - **per parallel input**, since one input may be served by a slower region or a different producer fleet; - **per producer version**, since one mobile release that changes buffering behaviour moves the whole curve; - **per hour of day and per day of week**, since overnight and weekend traffic are different populations; - **per geography or network type**, where connectivity differs sharply. A source whose aggregate 99th percentile is forty seconds may be two clean populations: connected producers at under a second and a small reconnecting fleet at hours. Those two need different treatment, and the aggregate percentile hides that they exist. ## Where and when to take the measurement 1. **Offline, over a stored span of raw records** read from the durable, re-readable input. You can compute the running front in one pass without touching the running job, and you can span a weekend and a release. 2. **Live, as a histogram the job itself emits**, so drift is visible rather than discovered during an incident. 3. **Repeatedly.** The curve is a property of the client population. A new producer release, a new market, a new device class or a change in a mobile platform's background-execution rules all move it, and none of them touches your code. ## What the measurement does not tell you Being clear about the boundary is part of a good answer: - It does not tell you what the wait **should** be. That is a cost decision — how much end-to-end latency the consumer of the output can afford — and it is decided against the distribution, not read off it. - It does not tell you what to do with whatever still misses. That is a separate mechanism and a separate promise to downstream consumers. - It does not tell you whether the job is **behind**. Backlog is unprocessed records piling up; a job can be entirely caught up and still deliberately waiting, and a job with a perfectly ordered source can still carry an hour of backlog. ## What varies between engines What you can do with the measurement depends on the execution mode, so say which you mean. On a runtime that executes continuous work as a rapid succession of small finite jobs, the granularity at which any wait can act is at least one of those intervals, so a measured spread far below that interval cannot be acted on at all. On a record-at-a-time runtime, the measurement translates almost directly into a hold behind the newest stamped moment. And for a single pass over a finished bounded input, the measurement is informational: the whole input is present when the read begins, so disorder costs a sort rather than a wait.