Before calling two runs of one job equal, what has to be pinned and what equality should you actually assert?

## Why the claim needs two halves "The two runs match" is not a statement until you say what was the same going in and what sameness you are testing coming out. Most arguments about a job's repeatability are two people asserting different halves at each other: one means the rows, the other means the bytes. ## What has to be pinned 1. **The input records.** Not "the same table" but the same set of records. If the source can receive late arrivals or be rewritten under you, the second run read something else. Pinning which version of an input a run read belongs to the table layer, not the engine; the job's part is to record which version it used. 2. **The code and the settings that shape the plan.** Anything that changes which rewrites fire, or whether a small side is sent whole to every worker rather than redistributed, changes the work performed and can change the combination order of aggregates. 3. **The step width** — how many pieces each step processes at once. Width changes which records share a piece and how many partial results exist, so it changes both the incidental row order and the last digits of a fractional aggregate. Choosing a good width is a different subject; here you are only holding it constant. 4. **Every unstable source inside a step body.** A clock read, a draw from an entropy source, a generated identifier, a lookup into a system that is changing. Fix a single as-of timestamp in the coordinating process and pass it in; derive seeds from a stable field of the record. 5. **The environment where it changes results, not just speed.** A different numeric type, a different locale or collation for string ordering, a different time zone for a date derivation. These look like infrastructure and behave like code. ## The equality, chosen before the comparison | equality | what it tolerates | when to use it | |---|---|---| | same rows, ignoring order | any row order, any file layout | the default for a job that declared no ordering | | same rows in the same order | file layout only | only where the job declares a total order, and a consumer truly depends on it | | values equal exactly | nothing | exact types — integers, exact decimals, identifiers | | values within a stated bound | last-digit drift from combination order | fractional aggregates over many values | Writing the equality down is most of the work. A comparison that fails every morning and is re-run until it passes has taught the team to ignore it. ## What you cannot pin - **Recomputation.** A worker dies, its piece is recomputed, and that piece's records reach the collector later than before. With pure bodies the values are unchanged, but the order is not, and a fractional aggregate's combining order is not. - **Runtime adjustment of the plan.** Where an engine changes the not-yet-run part of the plan from measurements of finished work, the number of partials can differ between two runs with identical settings. That mechanism has its own owner; its effect here is that width is a request, not always a guarantee. - **Arrival interleaving.** In a continuous record-at-a-time model, where one fixed graph stays running and records pass through as they arrive, the interleaving of concurrent sources is not reproducible, so per-run ordering never is. - **External systems read at run time.** Anything joined against live state was a different thing at the second read. So the strongest honest claim from a well-pinned job is **deterministic up to the stated equality**. That is a useful claim — it is exactly what lets a change be reviewed by comparing a before and an after — and it is not the same as byte-identical output. ## Doing the comparison Sort both outputs by a key you choose, or compare as multisets, and report differences as records rather than as lines. Compare numeric aggregates under the tolerance you declared. When the output is far too large to inspect at all, the technique for checking it is a separate subject with its own owner; what this leaf owns is the precondition — you cannot check anything until you know what was held fixed and what equality is being asserted. ## Where engines differ A finite job re-run from scratch is the easy case: pin the five things above and the equality usually holds. A continuous job cannot be "re-run" in the same sense at all — it is resumed from a saved picture of its progress, so the comparable unit is the output for a given range of input, not the run. And a model that executes continuous work as a succession of small finite jobs draws its own boundaries between those small jobs by timing, so the blocks the output arrives in will not line up between two executions even when the records do.