Three teams build reports on your hourly figures — what must your lateness policy state so that they know when a number is final?
answer
- publish the promise, not the setting
- provisional, then final
- signal a change, do not hide it
- name what you will not cover
- per figure, not per pipeline
basics
~20 sWhich of drop, divert or restate applies; the point at which a figure stops being provisional; whether it can change afterwards and how a change is signalled; and that the long tail beyond the policy is reconciled separately rather than waited for.
solid answer
~40 sA lateness policy is a published statement about the mutability of numbers, not a setting in a job. It has to say which promise applies to records arriving after their group closed — discarded, preserved in a separate repair channel, or folded in and the figure restated; when a figure stops being provisional and becomes one consumers may copy; how a later change is signalled, so a reader can tell a correction from new data; and what the pipeline explicitly will not cover, because some records arrive days late and no wait covers them honestly. Different figures from the same pipeline may deserve different promises: an operational dashboard can tolerate movement that a billing extract cannot.
go deeper
Understand that consumers need to be told whether a published number can still change; it is not something they can work out by looking at it.
Be able to state the three promises and say which one a given figure is on, along with the count of records that fell outside it.
Design the promise around what the destinations and consumers can absorb, and make provisional-versus-final observable in the published data rather than in documentation alone.
Own it across teams: name the finalisation point, signal corrections, publish what was dropped, and refuse the uncoverable tail explicitly rather than implying it is handled.
## Why this is a policy rather than a setting Every pipeline grouping by the moment stamped in the payload carries a **completeness claim** — a timestamp travelling with the data asserting that nothing older is still expected. When it passes a group's end, the group can close and its answer is published. Records that arrive afterwards are **late**, and what happens to them is visible to everyone who reads the number: they are the difference between a figure that never moves and one that might. So the decision does not live in a job. It lives in what the consuming teams are entitled to assume, and once three teams build on a figure, undocumented behaviour becomes a commitment by accident — usually the accidental one, which is dropping in silence. ## What the policy must state 1. **Which promise applies, per published figure.** Discarded; preserved in a separate repair channel for reconciliation; or folded in with the figure restated. Not one answer for the pipeline — one answer for each figure it publishes, because an operational dashboard and a billing extract can genuinely differ. 2. **When a figure stops being provisional.** A named point after which the number is the one to copy. Without it, every consumer invents their own, and they disagree. 3. **Whether it can change after that point, and by how much.** A figure that may move by a fraction of a percent is a different object from one that may be recomputed wholesale. 4. **How a change is signalled.** A version or as-of stamp on the figure, so a reader can distinguish a corrected old value from a new one. A number that silently changes is indistinguishable from a reader misremembering. 5. **What is explicitly not covered.** The long tail — a device offline for a week — is not coverable by waiting, and saying so is more useful than implying it is handled. ## What each consumer is actually asking | The question they ask | What the policy answers | |---|---| | Can I cache this? | Whether the figure is provisional, and until when | | Why did yesterday change? | Whether restatement is in force, and how it is signalled | | Is this number the same as finance's? | What was dropped, and whether the gap is recorded anywhere | | Do I need to re-read old periods? | Whether corrections are published for closed periods | | How far back can it move? | The horizon beyond which nothing is restated | ## Finality is a word with teeth 'Final' should mean one thing: the pipeline will publish no further value for this group. It is worth stating the consequence out loud in both directions. Where a figure is final, late records are gone and should be counted, because a final number with no dropped-record count is an unquantified claim. Where a figure is not final, nobody may treat a copy of it as authoritative, which constrains what downstream systems may cache, export or bill from. The honest middle is common and should be named: **provisional immediately, final after a stated point, corrections published in between.** That is three states, not two, and consumers can build on three states once they know there are three. ## The tail you should refuse to wait for Waiting longer buys completeness with latency, and the curve flattens hard: past some point, each additional hour of delay recovers a handful of records while delaying every figure for every consumer. A mature policy names that point and hands what falls outside it to a separate reconciliation, done deliberately against the preserved records rather than by holding the whole pipeline back. Declining to cover the tail *in the streaming path* is not the same as ignoring it, and the policy should say which it is. ## What varies, and why the policy must be written in mechanism terms Consuming teams may run on different systems, and the same word will mean different things to them: - A grouping that **republishes an updated running result** makes every figure perpetually provisional until its group is released, whatever the policy says. - A grouping that **emits once at close** makes figures final by construction unless a grace period is deliberately configured. - A **single pass over a finished bounded input** has no running claim at all, so 'provisional' there means only 'the next run may read more records', which is a different guarantee wearing the same word. Write the policy in terms of what a consumer may observe — may this number change, when, and how will I know — rather than in terms of any one runtime's behaviour, and it survives the pipeline being rebuilt on a different engine.
- How do you make provisional-versus-final visible rather than documented?Put it in the data: a state or as-of field on each published figure, flipped at the stated finalisation point, plus the count of records that missed the group. A consumer then queries the guarantee instead of remembering it, and a change to the policy is visible in the output rather than only in a document.
- Should every figure from one pipeline carry the same promise?Rarely. An operational view that moves during the hour is useful and cheap; a figure someone bills from should be final and carry what was excluded. Same job, same claim, different contracts at the publication boundary — and stating them separately is clearer than one promise weak enough to cover both.
- What do you say when a consumer asks for a number that is both immediate and final?That the two are the same trade in different words: finality means no further records will be folded in, and immediacy means the decision is taken earlier, when fewer have arrived. Offer both objects — an immediate provisional figure and a later final one — rather than one number pretending to be both.
saying these in an interview costs you the question
- Answers with a wait duration instead of a statement to consumers.
- Assumes one promise fits every figure the pipeline publishes.
- Never defines what final means, so each consumer invents it.
- Publishes corrections with no signal that a value changed.
- Claims the long tail can be covered by waiting long enough.
- Treats an undocumented default as though it were a decision.