Job as a Dataflow

Nothing in the job asked for an order, so none was kept. The work is cut into pieces processed by many workers at once, and rows appear in whatever sequence those pieces finished and were collected. Only a declared ordering step makes row order stable.

A describing call only adds a step to the graph the engine will run and computes nothing. A demanding call asks for an answer or for output to be written, and that demand is what makes the recorded steps execute.

Two: read-time column narrowing, which tells the read to produce only the two columns later steps use, and filter at the read, which moves the date condition down so rejected rows are never produced. Both shrink what enters the job.

Four: the points where records must be sent between workers, where the listing breaks into runs of steps between those points, how many pieces each step processes at once, and what the read was told to skip.

One pass over each record that applies all three operations in sequence. Step fusion removes the intermediate collections between the steps, not the work each operation does. The fused chain must end wherever records have to move between workers.

Both rewrites need to know what the unreadable body reads and produces. A condition on a value the body computed cannot be evaluated before the body has run, and a read cannot be narrowed to fields nobody can prove the body ignores, so whole records still arrive.

Look for where the listing breaks: a crossing is a point where records must be sent between workers, and most engines cut the printed steps into runs around it. An unbroken chain means every step builds each output piece from one input piece.

Do by hand what the engine cannot deduce: apply the conditions and select the fields before the step, hand the body named fields instead of the whole record, move the parts expressible as named operators out of the body, and call it after a reduction rather than on every raw record.

The engine knows only that this step must be called once per record, where the author put it, along with what it was handed and the shape of its result. It cannot see which fields the body reads, whether repeating it is safe, or what it costs.

Two things become visible: which fields the step needs, and whether those names and types exist. The engine can then check the step before any record moves, and can tell the read to produce only the fields some step actually references.

Only a program whose steps the engine understands can be improved. Work handed over as a function body is executed close to literally, and some engine models rewrite next to nothing on either surface, so the claim is conditional on both the surface and the engine.

Partial results were combined in a different order, and each addition in the usual binary floating-point format rounds. Nothing ever promised a combination order, so the two totals are both correctly rounded and simply not the same number.

The output becomes a function of when and where the body ran, not of the input. Two executions disagree, and one execution can disagree with itself when a lost piece is recomputed and stamped with later values than its first attempt produced.

Because the parse step only recorded itself; no record reached it until the write demanded an answer, so the failure surfaces there, carried back from a worker. Localise it by demanding answers from shorter prefixes of the graph over a small sample.

It measured graph construction: recording the steps, and perhaps resolving names and types. No records were read inside the timed region, so the number describes the program's description of the work, not the work.

A condition naming only grouping keys may move below, because rejecting whole records cannot change a surviving group's result. A condition on an aggregate must stay above: the value does not exist yet, and dropping its inputs would change it.

That the work has been funnelled through a single worker thread. Step width is how many pieces of the input a step processes at the same time, so a width of one means the cluster's size stops mattering from that step onward.

Re-express anything with a named equivalent - selecting, discarding, combining, grouping, joining, aggregating, sorting - and keep as a body only work with no name: bespoke parsing, custom scoring, per-key logic that remembers. Split fat bodies so the nameable half becomes visible.

Pin the input records, the code and the settings that shape the plan, the step width, and any clock or random read inside a step body. Then assert a chosen equality — same rows ignoring order, same order, exact values, or values within a stated bound — because some variation survives pinning.

Because a declared step holds no result of its own: each demand runs the graph again from its sources. Models that write every intermediate to storage, and models that keep one graph running and fan records out to both consumers, do not re-read.

Written order becomes executed order: no columns are dropped at the read, no condition moves down, and nothing is fused across a grouping step. The author has to narrow the read, filter first and fold partial results before records move.

Rewriting stops at three edges: a body the rewriter can only call and never read, a read with no way to accept a condition, and a step whose output something else depends on. Authors move the first by declaring the readable part of the work in named operators instead.

Each record leaves the engine's own runtime and comes back: it is converted into a form both sides read, crosses a process boundary twice, and is converted back, on top of the body's own work. Handing many records over at once pays the crossing cost once for the batch instead of once per record.

Everything about volume and evenness. The printed text states the shape the engine intends to run: how many records flow along each edge, how unevenly they land across pieces, how much a crossing moves and how long anything takes are all measurements it does not contain.