What does a cluster engine know about a named operator that it does not know about a function you hand it to run per record?

## The step graph and what goes into it A distributed processing engine does not run your program line by line. It first derives a **step graph** - the ordered set of steps it takes from your program, each step naming the steps whose output it reads - and only then decides how to run it. What the engine is able to do with that graph depends entirely on what each step *says*. Two authoring surfaces put very different things into it. - A **declared-operator surface** is one where the author names operations the engine already understands - keep the rows matching a condition, produce these fields, group by this key, join on that key, sum this column - over a **distributed collection**, the engine's handle on a set of records spread across the cluster. What lands in the graph is *discard rows whose amount is below one hundred*. - A **per-record function surface** is one where the author hands the engine a function and the engine calls that function once per record. What lands in the graph is *call this body on each record*. The engine knows the body will run. It knows nothing about what the body does. ## Meaning is what buys freedom Because a named operator carries meaning, an engine that has a **plan rewriter** - the component that edits the declared graph into an equivalent, cheaper graph before running it - can reason about that step rather than merely schedule it. At the level that matters here, four freedoms follow from being able to read a step: - **reorder** it against another step, where the result is provably the same; - **narrow** what the read produces, because the fields the named steps reference are visible; - **fuse** it with adjacent per-record work into a single pass, so no intermediate collection exists between them; - **skip** it, where it provably changes nothing. Each of those rewrites has conditions and limits of its own, which is a separate subject. The point on this axis is earlier and simpler: **none of them is available for a step whose content the engine cannot read.** A handed-over body is a box with exactly one contract - *call me once per record, in this position* - so the only plan the engine may legally make for it is to call it once per record, in that position. | What the engine holds | Named operator | Handed-over body | |---|---|---| | In the graph | the operation's meaning | "call this per record" | | Fields touched | visible | unknown; it must assume any | | Can be reordered | yes, on engines that rewrite at all, where equivalence is provable | no | | Can be dropped | yes, where it provably does nothing | no | | Checked before the run | usually, against the known record shape | only as it runs, on a worker | | Expressiveness | limited to the named vocabulary | anything the language can do | ## What varies between engines Do not carry one engine's behaviour into this as if it were the model of the class. - **Not every engine rewrites anything.** A **two-phase disk-handoff model** - one that runs a single grouping step at a time and writes every intermediate result to storage before the next begins - leaves a rewriter almost nothing to work with. There, a declared surface buys checking and readability rather than a cheaper plan. - **Which surface is idiomatic differs.** In a **continuous record-at-a-time model**, where one fixed graph of steps stays running and each record passes through it as it arrives, a per-record body is the normal way to express logic that remembers something between records for a key; often no named operator exists for it. In a **repeated-small-batch model**, which runs continuous work as a fast succession of small finite jobs, the declared surface is usually the default one. - **The freedoms are not all-or-nothing.** Some engines can still place, connect or fuse a body with its neighbours without reading it. What none of them can do is change what the body *means*. ## The mixed job is the normal job Real jobs use both surfaces. The named vocabulary is deliberately small - it is small precisely so the engine can reason about it - and plenty of honest work has no name in it: parsing an awkward text format, a bespoke scoring rule, a call out to something else. Handing over a body for that is correct, not a defect. The question is never "named good, body bad"; it is **where the seam between them falls**, and how much of the job you leave on the side the engine cannot see. ## What an interviewer is listening for "Named operators are faster" is a memorised slogan and invites an immediate follow-up the candidate then cannot answer. The model is: a named operator tells the engine *what* you want, so it stays free to choose *how*; a handed-over body tells it only *that* something happens, so it must do exactly what you wrote, where you wrote it. A candidate holding that can reason about a job, and an engine, they have never seen.