A record carries when the event happened, when it reached you and when it became valid — which belongs in the row labels?
answer
- three moments, one slot
- the world's ordering against your system's
- a re-run should not relabel rows
- lateness needs both moments kept
- late records rewrite, not append
basics
~20 sThe moment your questions are asked against, which for analysis is normally when the event happened, because that is the ordering the world had. Label by arrival instead and the same source re-run produces different numbers. Keep all three as columns.
solid answer
~50 sOnly one moment can be the row labels, so the choice is which ordering everything downstream inherits. When the event happened is the ordering the world had, and it is what almost every analytical question is really about. When the record reached you is the ordering your pipeline had — stable only within one run, and different on the next. When it became valid is the span over which a fact was true, which is a third thing again. Labelling by arrival is the common mistake: a re-run over the same source relabels every row, a late record lands at the end instead of in its place, and a value computed at a position can reflect facts that did not exist at that event's own moment. Keep whichever two you did not promote as ordinary columns — the difference between them is the only way to measure lateness at all.
go deeper
Know that a record can carry several different moments and that only one of them can be the row labels, so which one is chosen is a real decision rather than a formality.
Explain why a re-run over unchanged input produces different numbers under arrival labelling, and why the other moments have to survive as ordinary columns.
Show the failure that does not raise: a value stamped at one position that reflects a fact which did not exist at that position yet, and what the ingest step must do with a record that arrives three days late.
Own the choice as a standing commitment: it decides what a backfill means, what an audit can reconstruct, and what lateness can be measured at all — and no consumer can recover it once the other moments are gone.
## Three different moments, one slot A record about the world usually carries more than one time, and they answer different questions: - **When the event happened** — the moment the thing being recorded occurred. This is a property of the world and it does not change when you reprocess. - **When the record reached you** — the moment it arrived at your collecting process. This is a property of *this run of your pipeline*, and reprocessing produces a different value. - **When it became valid** — the span over which the fact the record asserts was true. A price effective from Monday to Thursday is true across that span regardless of when it was recorded or received. Promoting one of them to the row labels commits everything downstream to that ordering. The other two do not disappear — they become ordinary columns — but only one gets to be the thing ranges, windows and grids are expressed against. ## What each ordering can answer | Question | Needs labels by | |---|---| | How many failures happened on Tuesday? | When the event happened | | What did the dashboard show at 09:00 on Tuesday? | When the record reached you | | How stale is our view of this feed? | Both, as a difference | | Which price was in force on Tuesday? | When it became valid | That table is the whole argument for keeping all three. The moment you discard the ones you did not promote, an entire class of question becomes unanswerable, and no amount of care downstream brings it back. ## What labelling by arrival costs you Arrival time is the tempting default because it is always present, always non-decreasing, and free. It also has three properties that make it a poor ordering for analysis: - **It is not reproducible.** Re-run the same source through the same code and every label changes, because the labels record when *this run* saw each row. A backfill therefore produces different numbers from the original run over identical input, and the difference looks like a bug in the query. - **Late records land in the wrong place.** A record about Monday that arrives on Thursday sits at Thursday. A range over Monday misses it entirely, and a span walking the ordering pulls it into Thursday's neighbourhood, where it contaminates a computation about a completely different part of the timeline. - **It can put the future into a value.** A quantity computed at a position is supposed to reflect what was true at that position. Under arrival labelling, a record that arrived early but describes a later event is sitting before things that happened before it, so a value can incorporate a fact that did not exist yet at the moment it is stamped with. That is the failure that survives every review, because the number looks reasonable and the code looks careful. ## When arrival time IS the right label The choice is not a moral one, and there are real cases for each: 1. **Auditing the pipeline itself.** "What had we received by nine o'clock" is a question about your system, and arrival is precisely the ordering it is asked against. 2. **Measuring lateness.** The distribution of arrival minus event is the operational signal that tells you whether a feed is degrading, and it needs both moments present. 3. **Reproducing what a consumer saw.** If a report has to be explainable as of the day it ran, the ordering that explains it is the one your system had. The rule of thumb is simple: **label by the world's ordering for questions about the world, and by your system's ordering for questions about your system.** Almost all analysis is the first kind. ## Keeping all three honest 1. **Promote one, keep the others as columns.** The columns cost a fixed width per row and buy back every question in the table above. 2. **Write down which one is in the labels**, because a table where the labels are stamps looks identical whichever moment they came from, and the next reader cannot tell. 3. **Expect an event-labelled table to receive out-of-order appends.** A late record belongs at its own moment, which means the region around that moment is rewritten rather than appended to — and anything derived from that region has to be recomputed. 4. **Do not read a missing arrival moment as a missing event.** Where a record carries no stamp at all, the value a time column holds for it is a distinct absent-time marker, and it is neither the start of the epoch nor zero.
- Your labels are event moments and a record arrives three days late. What must the ingest step do?Place it at its own moment, which means the table is no longer appended to at the end — the region around that moment is rewritten, and anything derived from that region is stale and has to be recomputed. It also means the non-decreasing order you established earlier no longer holds for the raw append, so re-establish it before anything relies on it.
- How would you tell, looking at a table you did not build, which moment its labels carry?Compare the labels against the record's other time columns. Labels that are non-decreasing by construction, never revised, and always at or after every other stamp on the row are arrival moments. Labels that arrive out of order, cluster on the hour, or sit before another column's value are event moments. If the table carries only one time column, you cannot tell, which is itself the finding.
- Is "when it became valid" just a slower version of "when it happened"?No — it is a span rather than a moment. A fact can be recorded once and be true over an interval, and a record can be superseded by a later one that changes the end of that interval without any event occurring. Squeezing it into a single label loses the interval, so it usually stays as a pair of ordinary columns.
A letter carries the date written at the top, the postmark, and the date the office stamped it received. Filing the box by any one of them is legitimate, and it decides which questions are quick to answer — when it was written, when it was sent, when we saw it. What you would never do is throw the other two dates away, because the gap between them is the only record of how slow the post was.
saying these in an interview costs you the question
- Uses whichever timestamp the source happened to put first
- Says event and arrival moments agree closely enough to ignore
- Drops the other two moments once one becomes the labels
- Explains a changed backfill result as a bug in the query
- Thinks ordering by arrival cannot let future information in
- Treats "when it became valid" as a synonym for when it happened