Which facts in the text an engine prints about a job exist only because the work is spread over machines?

## What the printed text is An engine of this class does not run your program as written. It derives a **step graph** from it — the ordered set of steps the engine will run, each step naming the steps whose output it reads — and most such engines can render that graph as text. That rendering is the **printed plan**: the engine's own description of what it intends to run. About half of what it contains is ordinary query-plan material that would look much the same for the same computation on a single machine: a read at the bottom, some conditions, a grouping, a join, a write. Reading that half — walking the tree, comparing what an operator was predicted to produce against what it produced, judging per-operator cost — is a general skill that belongs with database execution plans and is not what this subject is about. The other half exists **only because the work is spread over many machines**, and it is the half a data-engineering interviewer is pointing at. ## The four distributed facts 1. **Network crossings.** A point in the plan where records must be sent between workers, because the next step needs records some other worker is holding. The industry word for this movement is a *shuffle*. Grouping by a key, joining two inputs on a key, producing one globally ordered result, and deduplicating across the whole input all tend to force one: records that have to meet must end up in one place. The exception worth knowing is an input that already arrives arranged by that key, in which case the engine may need no movement at all. 2. **Phases.** The crossings cut the listing into runs of steps that can be executed without moving any records. Most engines print the listing already grouped that way, and many call such a run a *stage*. A long run that never splits is telling you something concrete: every step inside it builds each output piece from exactly one input piece — a *narrow* dependency — so the whole run can be walked over each piece in one pass. 3. **Step width.** How many pieces of the input a step processes at the same time. This is the number that says how much of the cluster is working during that step, and it is not constant down the listing: a step after a crossing usually has the width the redistribution produced, while a step before it has the width the input was cut into. 4. **What the read was told to skip.** Two rewrites end up at the bottom of the plan and both are visible there. **Read-time column narrowing** tells the read to produce only the columns some later step actually uses. **A filter at the read** moves a condition down to the point of reading, so the rows it rejects are never produced at all. The printed text normally shows the read's column list and the conditions handed to it. ## Why those four and nothing else - They are the only facts in the text whose existence depends on there being more than one machine; everything else in the plan has a single-machine twin. - Each of the four is **checkable against the program you wrote**: you know how many times your job changes grouping key, so you know how many crossings to expect; you know which columns you use, so you can see whether the read was told about them. - They are all statements of **intent**. The text says what the engine means to do, not what will happen: it carries no record counts it has observed, no measure of how unevenly records will land, and no duration. ## What varies between engine models | Engine model | What its printed text looks like | |---|---| | A model that runs one grouping step at a time and writes every intermediate to disk before the next begins | Almost nothing is rewritten, so the printed shape is essentially the shape that was submitted, with one crossing per grouping step | | A model that keeps one fixed graph running and passes each record through it as it arrives | The text describes a graph that is already running; crossings are connections records are pushed across as they are produced, and widths are usually fixed for the life of the job | | A model that runs continuous work as a fast succession of small finite jobs | The text describes one of those small jobs; the same crossings and widths recur on every one of them | So "the engine prints a rewritten plan with pushed-down filters" is true of some engines here and false of others — in particular, a program written as per-record functions, where the author hands the engine a body to call once per record rather than a named operation, gives the engine nothing to push down and nothing interesting to print. ## How to read it in practice 1. Count the crossings, and compare that count with the number of times your program genuinely changes grouping key. More crossings than key changes is the interesting case. 2. Read the widths down the listing and look for a collapse — a step much narrower than its neighbours is where the cluster stops being used. 3. Look at the read's column list and its conditions, and check they match what you thought you asked for. 4. Stop. The text will not tell you how many records flow, how evenly they are spread, or how long anything takes; those come from measuring the run.