A deferred pipeline fails at the trigger with an error about a step written twenty lines earlier — why there, and what would have caught it sooner?

This is the first price of recording steps instead of running them, and the answer an interviewer is listening for separates two things that get bundled together. ## Why the report points at the trigger When the bad step was written, it appended a node to a description. Appending succeeded. The step itself — the thing that could not be done — only executed when the whole chain was handed to the engine, at the trigger. A failure is reported where execution was, which is the trigger, and a stack trace from inside the engine describes the engine's own machinery rather than your twenty-lines-earlier line. Good designs attach the originating step to the message; weaker ones leave you to work it out. ## Two different things are being deferred | error class | example | knowable before the data is read? | where it can surface | |---|---|---|---| | schema-resolvable | a column name the source does not have | yes | plan build, if the design resolves schemas | | schema-resolvable | arithmetic on a column whose type forbids it | yes | plan build, if the design resolves schemas | | schema-resolvable | a step consuming a column an earlier step dropped | yes | plan build, if the design resolves schemas | | data-dependent | a value outside the range a later step assumes | no | the trigger, unavoidably | | data-dependent | a group key whose cardinality exhausts memory | no | the trigger, unavoidably | | environmental | the source moved or was truncated | no | the trigger, unavoidably | The blanket claim — *with deferral nothing is evaluated until you ask, so a mistake only surfaces at the end* — is too strong. A planner can resolve the schema as the plan is built: it knows the source's column names and types, it applies each recorded step to that schema, and it can reject an unknown column or an invalid operation at the moment that step is appended, while still deferring every byte of real work. Designs differ on whether they do this. Some resolve eagerly and fail fast; some resolve lazily and everything lands at the trigger; some resolve partially, checking names but not every type interaction. Find out which one you are on before you build a debugging strategy around it. ## The chain does not always reach the trigger intact There is a second reason a failure can appear before the final trigger, and it surprises people. Some operations cannot be described without being partly run: a step that needs the row count, one that needs the rows in sorted order, one that needs the set of distinct labels to decide the shape of its own output. When such a step is appended, a planner may have to execute the chain up to that point right there. The practical effects are that part of the work happens on a line you did not think of as the trigger, and a failure can be reported at that line rather than at the end. ## Narrowing it down when the design will not help 1. **Ask the planner to describe its output.** Most deferred designs can report the columns and types the chain would produce without running it. If that call fails, the failure is schema-resolvable and it tells you which step broke the schema — often naming the step directly. 2. **Trigger on a tiny prefix of the input.** Running the identical chain over a few thousand rows exercises every step for a fraction of the cost and reproduces every schema-resolvable failure and most type failures. It will not reproduce a value that only appears at row 4 million. 3. **Bisect the chain.** Trigger the first half, then three quarters, and so on. Each trigger re-reads the source, so do this against the small prefix rather than the full input. 4. **Split the pipeline at deliberate boundaries.** A long chain triggered once has one place for errors to land. Three segments, each triggered and checked, have three. The cost is the retained intermediates, which is a real footprint cost and the reason not to do it everywhere. ## What to build in Assert the shape at segment boundaries — the columns you expect to exist and the types you expect them to carry — rather than relying on the engine to complain in a way you can read. An assertion written where the step was written reports at that line, which is precisely the property deferral took away, and it costs nothing at plan-build time if the design can answer the schema question without running. The summary worth saying out loud: the work is deferred by construction; whether the *mistake* is deferred too is a property of the planner, not of deferral itself. Name the class of error, say whether the design resolves schemas at build time, and then reach for the small-prefix trigger — it answers most of these in under a minute.