A crafted test of a job's time logic passes every run, yet production groups are wrong — which time-specific things did the test never exercise?

## What the crafted test actually proved A driven feed — records handed over one at a time, with the completeness claim advanced by the test itself — proves a precise and narrow thing: *these* records, in *this* arrival order, with the claim moved to *these* points, produce *that* output. Everything outside those three variables was supplied by the author, and that is where production differs. The honest list is short enough to hold in your head. ## Where the event moment comes from In the fixture, every record carried a clean stamp because the author typed one. In production the pipeline **assigns** the event moment, and every assignment can go wrong: - The field is missing on some records, and the fallback — arrival, or a null treated as the beginning of time — is rarely what anyone intended. - The stamp is in seconds where the code expects milliseconds, so the moments land decades away and every group is either empty or settles instantly. - The stamp is local time without a zone, and the groups shift by hours twice a year. - Nobody assigned one at all. Some runtimes then group by arrival without complaint, and the job produces plausible numbers that answer a different question. None of these reach the grouping logic the test exercised, because the fixture replaced the assignment step with a literal. ## How the claim behaves with more than one input The test had one driven input, so the claim was whatever the test said it was. Production usually has several pieces of the input read in parallel, and where a runtime propagates a claim across them it generally advances the downstream claim only as far as the **least advanced** of its inputs — otherwise a group could settle while an input still holds older records. Consequences the fixture cannot show: - One input that goes quiet holds the claim still, so groups never settle and results simply stop appearing while the job looks healthy. The behaviour of a quiet input is a subject of its own, and this is where a single-input test hands it over. - Adding or removing input pieces changes which one is the laggard, so the job's apparent latency changes without any code change. - A restart resumes the claim from wherever the job's saved state put it, which is not where the test left it. ## How wide the real disorder is The fixture's disorder is a guess, and the tolerance the job honours was sized against that guess. Production disorder is a distribution with a long tail: a mobile client buffering offline, a region re-sending after an outage, a batch of backdated corrections. If the measured gap between event and arrival moments has a tail of minutes and the job tolerates seconds, the test is green and a steady fraction of records is being discarded or diverted every hour. ## Also unexercised, briefly - **Scale.** A handful of records never fills what a long-running job retains between records, so nothing about growth or eviction of partial groups was touched. - **Restart.** What the claim is after resuming, and whether a re-emitted group is absorbed or double-counted by the destination, involves machinery the in-process test did not run. - **The destination.** The test read emissions from memory; production writes them somewhere that may append where it should update. ## What to do instead of a bigger fixture 1. **Measure the gap on real input** — event moment against arrival moment, the typical value and the tail — and rebuild the crafted records from those percentiles, so the tolerance is a decision rather than an accident. 2. **Prove the assignment once, end to end.** One slow test against a real deployment, asserting that a real record ends up with a sensible event moment, catches the whole unit-and-zone family that no fixture can. 3. **Instrument what a test cannot observe.** In production, watch the count of records arriving after their group settled, and the distance between the claim and the present. Both are cheap and both move before anyone notices a wrong number. Rules about the data declared outside the job, and checked against what it published, are a neighbouring discipline worth using here rather than reinventing. 4. **Re-derive the tolerance when the source changes.** A new client, a new region or a new upstream buffer changes the distribution the number was chosen from, and nothing in the suite will go red when it does. The test is still worth having. It is a statement about logic, made cheaply and deterministically; the mistake is reading it as a statement about the job's behaviour on real input, which it never was.