Two runs of one job over the same input write the same rows in a different order — why?

## Say which equality you mean "The two runs produced the same result" hides at least four different claims, and a job promises different things about each. | claim | what it asserts | promised by default | |---|---|---| | same rows, ignoring order | the outputs are the same multiset of records | usually yes, for a job with no unstable source inside a step body | | same rows in the same order | row *n* of one output equals row *n* of the other | **no**, unless the job declares a total order | | bit-identical values | every number matches to the last digit | **no**, where the order values were combined in affects the arithmetic | | same output files, same names | the output is laid out identically in the shared store — the cluster-visible storage a job writes to | **no**; how many output pieces there are follows the run | Row order is the second line, and it is the one people assume they already have because a small test on one machine appeared to give it to them. ## Where output order actually comes from The engine derives a **step graph** from the program — the ordered set of steps, each naming the steps whose output it reads — and runs each step over the input in pieces, one worker thread to a piece. Nothing in that description mentions a sequence. The order rows land in the output is set by: - **which piece each record was in** — fixed by how the input was cut, and then changed by any step that needs records held by other workers, which redistributes records so that all records sharing a grouping key meet on one worker (a **shuffle**); - **which pieces finished first** — a worker that finishes early contributes early, and a piece that runs far longer than its peers (a **straggler**) contributes last; - **the order the collecting worker, or the coordinating process that receives whatever the demanding call asked for, took delivery.** None of those three is stated anywhere in the program, so none of them is promised. ## The order that looks stable until it isn't Many jobs appear to produce a stable order for months. That is an **incidental** order, and it holds only while nothing moves. It changes when: 1. the step width changes — how many pieces a step processes at once; 2. a piece is lost and recomputed, so it arrives later than it did before; 3. the set of input files changes, or the same records are laid out differently; 4. the engine adjusts the not-yet-run part of the plan using measurements from work already finished. A consumer that quietly depends on the incidental order breaks on the day one of those four happens, which is rarely the day anyone is looking. ## Where engines genuinely differ - **Within one piece**, most engines feed records to a per-record function surface — a surface where the author hands over a function to call once per record — in the order the piece was read. That is a property of the piece, not of the output. - **Across a redistribution**, engines differ: some assign keys to destinations by hash and emit groups in no meaningful order; others sort keys as part of the redistribution, so their output looks sorted. Relying on the second is relying on one engine's method. - **In a continuous record-at-a-time model**, where one fixed graph stays running and each record passes through as it arrives, output order also reflects the interleaving of concurrent sources, which nothing reproduces. - **In a repeated-small-batch model**, which runs continuous work as a fast succession of small finite jobs, each small job emits its own block of output and the boundary between blocks moves with timing. ## Order inside a group is not promised either This is where unpromised order stops being cosmetic. A step that takes *the first record per key*, or *the last status per account*, is reading an order nobody declared: whichever record reached the grouping first this time wins. The rows are the same both runs; the **values** are not. The repair is to define "first" in the data — order by a timestamp with a unique field as a tiebreaker, then take the top — so the answer is a function of the records rather than of the race. ## Comparing two runs Compare as multisets, or sort both outputs by a key you choose before comparing. A line-by-line difference of two unsorted outputs reports thousands of differences that mean nothing and hides the one that means something. Checking an answer far too large to read at all is a separate subject with its own owner, and producing a genuinely ordered result at scale is another; here the point is narrower — state the equality you are claiming before you claim it.