A filter written above an author-supplied per-record body never reaches the read - which edges stop a rewriter, and how do you move them?
answer
- the rewriter can call it, not read it
- three edges, not one
- a read with no way to take a condition
- split the body, declare the readable part
basics
~20 sRewriting stops at three edges: a body the rewriter can only call and never read, a read with no way to accept a condition, and a step whose output something else depends on. Authors move the first by declaring the readable part of the work in named operators instead.
solid answer
~60 sA plan rewriter edits the declared graph only where it can prove the edit preserves the answer, so it halts wherever that proof is unavailable. Three edges account for nearly all of it. First, an opaque step - a step whose body is ordinary code the rewriter cannot look inside, so it can only be called, never reasoned about; a condition above it cannot be shown to commute with it, so it stays where it was written, and the read keeps producing every column because field use above the body is unknowable. Second, a read that cannot accept a condition at all, in which case the condition lands immediately above the read instead. Third, a step whose output another branch consumes, or that the author asked to be kept for reuse, since moving work around it changes what the other consumer sees. The repair for the first is structural: express in named operators the part of the work the engine could have understood, and write the narrowing and the condition above the body yourself.
go deeper
The idea to hold on to is that the engine can only change what it understands. A step handed over as ordinary code is a wall: the engine calls it and cannot reason about anything across it.
Explain each edge by mechanism - no proof of equivalence across an unreadable body, no field references to narrow a read with, an input that cannot take a condition, a step another consumer depends on.
Show the restructuring: split the body so the declarable part is declared, place the narrowing and the condition above the body by hand, and stop pinning results nothing reuses. Then verify the edits actually landed rather than assuming.
The call you own is where bespoke code is allowed to sit in a platform's jobs. Every such body is an edge the engine will not cross, so a convention about what must be expressed in named operators is a performance policy, not a style preference.
## Why a rewriter stops at all A **plan rewriter** is the engine component that edits the declared graph - the **step graph**, the ordered set of steps derived from the program, each naming the steps whose output it reads - into an equivalent, cheaper graph before running it. Every edit it makes has to be justified: the rewritten graph must produce the same answer. Where the justification is unavailable, the correct behaviour is to make no edit. So the interesting question about a rewriter is never 'what can it do', it is 'where does it stop', because that edge is where an author's decisions start to matter. ## The three edges **1. A body the rewriter can only call.** An **opaque step** is a step whose body is ordinary code the rewriter cannot look inside. The engine knows that the body runs once per record and nothing else - not which fields it reads, not whether it produces the same output for the same input, not whether it has an effect outside the job. Two consequences follow, and they are the two the question is about: - A condition written above the body cannot be moved below it, because the rewriter cannot establish that the two commute. If the body changes the field the condition tests, or drops records, or adds them, the reordered graph would answer differently. - Read-time column narrowing loses its information. Field use is collected by walking the graph; a body exposes no field references, so the rewriter must assume the whole record is needed and the read produces everything. **2. A read that cannot take the condition.** Even a perfectly analysable condition cannot be evaluated by every input. Where the input has no way to accept one, the engine places the condition immediately above the read. Rows are produced and then rejected at once, so every step above sees fewer records while the read itself is unchanged. This edge is easy to misread as a failed rewrite; it is a partial one. **3. A step something else depends on.** If a step's output feeds two branches, or the author asked for it to be kept so it is not recomputed, it is no longer a private intermediate. Moving work across it, or removing it, changes what the other consumer receives. The rewriter treats it as fixed. ## Moving the edges The edges are not all immovable - the first one in particular is usually a consequence of how the program was written. 1. **Split the body.** Most author-supplied bodies do two things: something the engine has a named operator for - selecting a field, testing a value, computing an arithmetic expression - and something genuinely bespoke. Declare the first part in named operators and leave only the bespoke remainder inside the body. Everything declared becomes visible to the rewriter again. 2. **Write the narrowing and the condition above the body yourself.** If the rewriter will not move them down, put them down. Select the fields the body needs before it runs, and reject the records it does not need to see. This is the single highest-value habit on this subject, and it works on every engine - including one that rewrites nothing. 3. **Do not force an intermediate to be kept unless something really reuses it.** A result pinned for reuse is a step the rewriter must respect; pinning one that is read once buys nothing and costs the edits that would have crossed it. 4. **Prefer a named operator wherever one exists.** A body that reimplements a standard operation is invisible work the engine already knew how to do. ## What is not this question Two neighbouring subjects get pulled in and should be pushed back out: - **What the body costs to run** - its per-record invocation, and the cost of records leaving the engine's own runtime so a second language runtime can process them and come back - is its own subject. Here the body matters only because it blocks rewriting. - **Changing the plan mid-run on statistics measured from finished work** is a different mechanism entirely. It happens after the rewriter has finished, it exists in some engines for some operations only, and in a continuous job largely not at all. ## The claim to keep honest 'The engine optimises your program' is true only of what the program declared in named operators. Written as author-supplied per-record bodies, the same logic is executed close to literally, and in a model that runs one grouping step at a time and writes every intermediate to disk before the next begins, essentially nothing is rewritten at all. The edge does not move on its own; the author moves it by choosing what to declare.
- Why does an opaque body block read-time column narrowing as well as the condition?Narrowing works by collecting the field references of every step above the read. A body exposes none - the engine knows a function runs, not which fields it touches - so the rewriter must assume any field could be needed and the read produces the whole record. Declaring the field selection above the body restores the information and the narrowing with it.
- The author pinned an intermediate result for reuse and the rewrites stopped crossing it. Is that a bug?No, it is the contract. A pinned result is promised to other consumers, so the rewriter must produce exactly that step's output and cannot fold work across it. It is only a mistake when nothing actually reuses the result: then the promise buys nothing and costs the edits that would otherwise have crossed.
- How do you tell a blocked rewrite from one that simply had nothing to do?Compare what the job was asked to produce with what the graph says the read produces and where the condition sits. If a job reading a wide input still produces every column, or the condition is still above a body near the top of the graph, something blocked the edit. Reading that rendering is its own skill, but the habit of checking belongs here.
saying these in an interview costs you the question
- Assumes the rewriter analyses the body and moves conditions through it
- Says a condition that cannot reach the input is left where the author wrote it
- Treats every failed rewrite as an engine defect rather than a missing proof
- Pins intermediate results for reuse that nothing else reads
- Claims runtime replanning on measured statistics will fix a blocked rewrite