Checking the Answer

## What a stored expected result actually asserts A **stored expected result** is a file of rows committed next to the test, and comparing the job's output to it line by line asserts three things at once: 1. the same **values** were produced; 2. they appear in the same **order**; 3. they are laid out in the same **shape** — the same number of objects, the same columns, the same text rendering of each number. Only the first is a statement about the job's logic. The other two describe how one particular distributed execution happened to arrange itself, and a distributed run promises neither. ## What a distributed run does not promise - **Order across workers.** The input is cut into **pieces** — one chunk of the data that one worker processes independently of the others — and each worker finishes when it finishes. Whichever worker's output is committed first sets the order in the combined result; next time a different one wins. A job that ran over a single piece during development can look perfectly stable and then reorder the first time it runs wide, which is why this bites in production rather than in the test. - **Order within a group.** A step that needs records currently held by other workers forces a **redistribution**: every worker writes its records out, and every worker fetches the ones addressed to it. The order those fetched records arrive in is a race between senders. - **The layout of the output.** Many runtimes write one output object per piece, so the file count tracks the width of the last step rather than the size of the answer; others commit one set through a single writer. Change the width and the identical answer arrives as a different set of files. - **Bit-identical arithmetic.** Approximate numeric types are not associative under addition: the same amounts added in a different grouping give a total that differs in the last places. The value is right to the precision the type carries, not to the last bit. *Why* two runs can disagree at all is a separate subject; here it is enough that they may. - **Columns minted during the run.** Load timestamps, run identifiers, and surrogate numbers handed out as rows are written differ on every execution by design. ## What "the output" even is depends on the model | the job | what the output is | what an expected-result comparison can mean | |---|---|---| | a finite run over bounded input | one complete set of records, committed at the end | compare the whole set, once | | continuous work run as a rapid succession of small finite runs | a series of increments, one per small run | compare one increment, or compare a time bucket once its completeness claim has passed | | a record-at-a-time runtime holding a key-bound retained set | a running result per key that may be re-emitted as the group is revised | compare the **latest value per key**, never the sequence of emissions | A test that captures an expected file from the first model and re-uses the comparison style against the third will fail forever, because the third legitimately emits the same key more than once. ## Comparing so that only the values matter 1. **Declare the grain** — the column set that identifies exactly one output row. If you cannot name it, you cannot diff; assert uniqueness on it as its own check. 2. **Match by key, not by position.** Join the produced rows to the expected rows on the grain and classify each key as present-in-both, only-produced, or only-expected. 3. **Compare numbers with a tolerance** that has both an absolute floor and a relative allowance, so small values are not swamped by the floor and large ones are not failed by the last digit. 4. **Normalise the volatile columns** — exclude them, or overwrite them with a fixed value before comparing. 5. **Decide the multiset question.** If the grain is genuinely not unique, compare counts per key rather than pretending each key has one row. ## What this style of comparison still cannot prove - A stored expected result is usually produced by running the job once and blessing the output, which freezes whatever the job did that day, bug included. It detects **change**, not **correctness** — which is why totals reconciled against the input and invariants asserted on the result carry the correctness argument instead. - It says nothing about volume. A comparison over a handful of rows proves the shape of the logic and nothing about what happens at production width. - Executing the whole job inside the test's own process, on one machine with no network and no second worker, cannot surface record redistribution, worker loss or recovery at all — that limitation, and how to work around it, is the neighbouring subject of testing logic outside the cluster.