A job declares three per-record steps in a row with no redistribution between them - what does an engine that fuses them actually run?

## What fusion is **Step fusion** is the rewrite that collapses several adjacent per-record steps into a single pass over each record, so that no intermediate collection exists between them. Three declared steps - say, drop a field, compute a derived value, reject records failing a test - remain three steps in the graph the author wrote. At run time they are one pass: a record is taken, the three operations are applied to it in order, and the result is handed to whatever comes next. The next record then follows. The contrast is the unfused shape, where each step consumes a whole collection of records and produces another whole collection for the next step to consume. That shape costs a materialisation between every pair of steps and touches each record once per step. ## What it removes and what it does not The honest list is short on both sides: - **Removed:** the intermediate collection between each pair of fused steps, and the per-record overhead of entering and leaving each step separately. - **Removed:** in engines that go further and compile the fused chain into one generated loop, a layer of indirection per record per step as well. Others simply call the operations in sequence and keep that indirection; both are real designs. - **Not removed:** the work each operation performs. Three operations still run on every surviving record. - **Not removed:** any movement of records between workers that a later step requires. - **Not removed:** the cost of an author-supplied body inside the chain. It is still called once per record, and it is also the thing most likely to end the chain. ## Where a fused chain has to end Fusion is only valid where a record can be carried forward on its own. It therefore ends at: 1. **A step that needs records other workers are holding.** Grouping by a key, joining on a key, or producing a global order all require records to be sent between workers first - a **network crossing**. Nothing downstream of that point can be fused with anything upstream of it, because the records have not arrived yet. The run of steps between two such crossings is a **phase**: a stretch the engine can execute without moving any records. 2. **A step that must see all of its input before it emits anything.** A sort, or an aggregate that produces one row for many, cannot hand a result forward per record. 3. **A step whose body the rewriter cannot look inside**, in engines where fusion is done by generating one loop from the operations - there is no source to fold into the loop, so the body is called as an ordinary function and the chain typically breaks around it. In engines that simply call operations in sequence, the body is just another call in the chain. ## The direction of the claim: who fuses, and when This is where candidates over-generalise from the engine they know. | model | what happens to three adjacent per-record steps | |---|---| | declared-operator surface with a plan rewriter | the rewriter recognises the chain and fuses it before the run, sometimes into one generated loop | | per-record function surface | there is nothing to rewrite, but the runtime still commonly pulls one record through the chained functions rather than building a collection per step | | continuous record-at-a-time model, where one fixed graph stays running and each record passes through as it arrives | chaining per-record steps within a worker is the normal execution shape, independent of any rewriter | | two-phase disk-handoff model, which runs one grouping step at a time and writes every intermediate to disk | no fusion across grouping steps; the written intermediate is the hand-off | So 'the engine fuses my steps' and 'records flow one at a time through adjacent steps' are two different claims with two different causes, and only the first is a plan rewrite. ## Why an interviewer asks it Because it separates two costs that beginners merge. A job that is slow because each record is expensive is not helped by fusion; a job that was slow because it built a collection between every step is. And the practical consequence is the one to say out loud: the number of declared steps in your program is almost free within a phase, so splitting a long expression into readable named steps costs you nothing, while inserting a step that forces records to move costs you a phase boundary.