Two versions of a job's logic ran over the same input and thousands of grouping keys differ — how do you decide which differences are the change working?

## The comparison is supposed to be non-empty Two senses of *difference* collide on this subject and they want opposite outcomes. A comparison of a job's output against a stored expected result must come out empty, or the job is broken. The two-version comparison discussed here must come out non-empty whenever the change was meant to change a number, and its rows are exactly the thing you have to explain before you are allowed to cut over. Counting them proves nothing on its own. A thousand differing keys can be a perfect change and three can be a disaster. The mechanics of comparing large outputs — comparing by key rather than by file position, tolerating floating-point drift, normalising before comparing — are general output-checking craft and apply here unchanged. What is specific to a change under way is everything that follows the comparison: attribution. ## Predict before you look Write down, before reading a single row: - **which keys** the change should touch, stated as a rule you can evaluate — for example, every key whose records include the one source whose unit handling changed - **roughly what share** of all keys that is, as an order of magnitude - **in which direction** the affected values should move, and roughly by how much - **what should not move at all** This is the whole method. A prediction written afterwards is not a prediction; it is a rationalisation, and it will accommodate whatever you find. ## Classify, then attribute Sort every key into four classes and read the counts against the prediction: 1. **Only in the current output.** The candidate dropped it. Intended for a change that filters; alarming otherwise. 2. **Only in the candidate's output.** The candidate created it. Intended for a change that stops over-filtering; alarming otherwise. 3. **In both, values differ.** The main body. Split it further by whether the key was in the predicted set. 4. **In both, values equal.** The part you already understand; it carries no information and should not be read. Then attribute class three. Join the differing keys back to something about their input — the upstream source, the period, a category, whether a field was null. A change doing what you think produces moved keys concentrated on the attribute you changed handling for. A change doing something else produces moved keys spread uniformly, which is the single clearest signal that you have touched a shared step rather than the one you meant to. ## When everything moved If you predicted two percent and a hundred percent moved, the first hypothesis is not that the logic is catastrophically wrong. It is that something shared and cosmetic changed underneath: a rounding rule, a precision, a period label rendered in a different time zone, a trailing-space or null-versus-empty convention, a currency or unit formatting step every row passes through. These move every row while changing nothing meaningful, and they are far more common than a logic change that genuinely reaches every key. Normalise for them and compare again before drawing conclusions. ## Reading a comparison too large to read You will not read a million rows, so read summaries in this order: counts by class; the distribution of relative change among the differing keys; the top contributors by absolute change, because those are the ones a reader will notice; and a small random sample of each class, because the top contributors are systematically unrepresentative. ## What the comparison still cannot tell you Two hard limits, and an interviewer is listening for both. - **It cannot say which version is right.** It is evidence for a conversation, not a verdict. The published version is the baseline, not the truth — it is frequently the thing being fixed. - **It cannot surface defects that only appear under failure or under rare data.** A healthy comparison window never exercises a worker lost mid-run with its piece recomputed, a write retried after a failure, a **redistribution** under one enormous key — the point where a step needs records currently held by other workers, so every worker writes its records out and every worker fetches the ones addressed to it — or the period-close path that runs once a month. Non-determinism inside the job itself is a further limit: if the logic breaks ties by arrival order, or sums floating-point values whose order depends on how records were delivered, then two executions of the *same* version can disagree, and that noise floor has to be known before any of the two-version rows mean anything. Runtime model changes what settles, too. Some engines emit a group's result once when it closes; others emit a speculative result early and corrections afterwards; one common model emits an updated running result on every input. Compare settled final values per key, or you will be comparing one arm's intermediate emissions against the other's final ones and explaining a difference that does not exist.