Readings from 300 devices sit in one table and holes are filled by carrying the last value forward; what silently goes wrong?
answer
- the carry follows row order, not the sequence
- two boundaries: the entity and elapsed time
- interleaved devices share one walk
- a flat run looks like a steady sensor
- backward carry writes the future into the past
basics
~20 sThe carry walks down the rows in whatever order they currently sit, not down each device's own sequence, so one device's last reading fills another device's hole. Within a single device it also invents readings for the whole time that device was off.
solid answer
~50 s**Carrying the last value forward** — standing a hole up with the nearest earlier observed value in the order the rows are currently in — crosses two boundaries the data does not want crossed. The first is the entity boundary: rows from 300 devices are interleaved, so unless something confines the carry, the value that lands in a hole may belong to a different device. The second is the time boundary: the carry never asks how old the value is, so a device offline for nine days is given nine days of its last reading, and that flat run is indistinguishable from a sensor that genuinely did not move. Two aggravating details. The result depends entirely on the row order in force when the carry ran, which is an input nobody writes down. And carrying the *next* value backward is worse still — it writes an observation into rows that came before it.
go deeper
Know that this fill copies the nearest earlier value down the rows and that it has no knowledge of which device or which day a row belongs to. It reuses a real number to make a claim that was never measured.
Explain that the operation follows the current row order, so interleaved entities share one walk, and that it is indifferent to elapsed time, so a long outage becomes a long flat run rather than a gap.
Show the diagnosis and the guard: bound the reach in elapsed time or consecutive holes, establish the boundary the carry may not cross before running it, expect the leading rows to stay absent, and record the carried cells so the output can be audited.
The standing question is what the pipeline owes its consumers. A carried gap silences the one signal monitoring most needs, so a team rule on when a carry is permitted at all is worth more than any per-job review.
## What the carry actually does **Carrying the last value forward** means: walk the rows in the order they are currently in; whenever a cell is absent, write into it the nearest earlier observed value in that same walk. Its sibling, carrying the next value back, walks the other way. The operation is mechanical and it has no idea what your rows mean. Everything that goes wrong follows from that. It is the most seductive of the three dispositions because the value it writes is a **real measurement** — it was observed, by the same instrument, in the same units. That makes it feel unlike inventing a number. It is not: the claim being made is that the measurement *still held* at the later row, and that claim is manufactured. ## Boundary one: the entity With 300 devices in one table, consecutive rows usually belong to different devices. The carry does not know the device column exists. A hole in device 41's reading is filled from whichever row happens to sit above it — device 7, perhaps, measuring something else in a different place. The symptoms are characteristic: - Values appear for a device during a period it reported nothing at all. - Two devices show identical readings at the same timestamp far more often than physics allows. - Re-sorting the table and re-running the fill produces a **different** filled column, because a different row is now above each hole. That last point is the one that makes this hard to catch: the carry's output is a function of the row order at the moment it ran, and row order is rarely pinned anywhere in the pipeline. An upstream change that alters ordering changes the filled values with no code change and no error. ## Boundary two: elapsed time Even if only one device were involved, the carry is indifferent to how much time it is bridging. A hole two seconds after the last reading and a hole nine days after it are filled identically. So a device that goes offline gets a flat run at its last value for the whole outage, and downstream: - An average over the outage window is computed over 216 fabricated hours of readings. - A rule that alerts on *no change for N hours* sees a healthy-looking constant instead of a gap. - A chart shows a horizontal line, which reads as a stable sensor rather than a dead one. The distinctive damage here is that the fill converts a **loud** failure — nothing recorded — into a **quiet** one: a plausible, well-formed, completely false number. ## Direction matters more than people expect | | carrying the last value forward | carrying the next value back | |---|---|---| | where the value comes from | a row that already happened | a row that had not happened yet | | plausible for a slow-moving state | yes — a reading holds until it changes | rarely | | usable in a sequence read in time order | yes, with a limit on reach | no — the earlier row now holds later information | | typical use | a status or level that persists | filling the leading rows a forward carry cannot reach | Carrying backward writes into a row a value that was observed after it. Any analysis that walks the data in time order is then using information the row could not have had. It is not a subtler version of the forward carry; it is a different and usually worse claim. ## The gaps the carry does not close A forward carry cannot fill a hole that has no earlier observed value, so the leading rows of the table — or of each device's sequence, if the carry is confined — stay absent. A column that has been carried forward is therefore usually *partly* filled, and code written on the assumption that the fill completed the column will meet absence anyway, in the rows least likely to be tested. ## What to do instead, and what varies 1. **Establish the boundary the carry must not cross before you run it.** The carry is only meaningful within one entity's own sequence, ordered by its own timestamps. 2. **Bound the reach.** Some surfaces take a limit on how many consecutive holes one observation is allowed to stand up, or on how much elapsed time it may bridge; where a surface offers one, set it, and where it does not, the carry runs as far as the data lets it. Do not assume a default here — this is one of the settings that genuinely differs, and an unset limit is the difference between bridging a dropped packet and bridging an outage. 3. **Record what you carried.** A companion column marking the carried cells is the only thing that survives into the output and lets a reader tell a fabricated flat run from a real one. 4. **Consider leaving the holes.** A gap in a sequence is information: it says the device was not reporting. Filling it destroys the one signal that a monitoring pipeline most wants. The question to ask before any carry is simple and it is not about the tool: *for how long does this quantity remain true after it was last observed?* A contract status remains true for months; a temperature remains true for minutes; a network counter was never true for any duration at all. The answer to that question is the limit, and if it cannot be answered, the carry should not run.
- Why is carrying the next value backward worse than carrying the last one forward?An earlier row ends up holding a value that was observed after it. Any analysis that reads the data in time order is then using information that did not exist at that point, which makes results look better than the sequence could support. Backward carry is defensible mainly for the leading rows a forward carry cannot reach, and even there it should be recorded.
- After a carry, how would you tell a fabricated flat run from a genuinely steady reading?From the column alone you cannot, which is the point. Only a companion column marking the carried cells, the original unfilled data, or the pipeline's own history can separate them. This is the strongest argument for recording the fill at the moment it happens rather than reconstructing it later.
- Why can the same carry give different answers on two runs of the same pipeline?Because the value that lands in a hole is whatever the walk saw most recently, and the walk follows the row order in force at that moment. If an upstream step changes how rows arrive or are ordered, the filled values change with no code change and nothing raising.
It is like filling gaps in a hospital chart by copying down the line above. If the charts of several patients were shuffled together, you copy a stranger's blood pressure; and even on one patient's own chart, a reading copied across a week says the patient was stable when in fact nobody looked.
saying these in an interview costs you the question
- Carrying forward is safe because it only reuses real measurements
- The carry follows each device's own sequence automatically
- Sorting the table by timestamp is enough to make the carry correct
- Carrying backward is just as safe as carrying forward
- Every hole gets filled, so the column is complete afterwards
- A flat run in the output proves the reading was stable