Both join inputs were written with an identical division on the join key, yet the run still redistributes both sides - why?

## Two different statements 'The matching records are already together' and 'the run can prove the matching records are already together' are different claims, and only the second one saves anything. The first is a fact about bytes on disk. The second is a fact about the plan the engine built, and it is the one that decides whether a step is executed locally or as a **wide step** - a step a worker cannot finish from the records it already holds, so records are redistributed by key first. This is the part of the subject that interviewers use to separate people who have read about arranged layouts from people who have tried to get one to pay off. Almost everyone who has tried has had the experience of writing both sides carefully and watching the move happen anyway. ## Where the claim can come from | Source of the data | Can the run establish the layout? | Usual outcome | |---|---|---| | Produced by an earlier step of the same run | Usually yes - the engine knows how it divided it | the property is carried forward | | Read from storage that records key, rule and count | Only if the plan reads that metadata and trusts it | may skip the move | | Read from storage that records nothing about layout | No - the arrangement is invisible | always redistributes | | A continuous job's input at a declared width | Only while the width and key extraction match on both sides | holds until a restart changes the width | The third row is the one people trip over. An arrangement that exists only because a previous job happened to write it that way, with nothing recorded anywhere, is indistinguishable from a random layout to the run that reads it. ## What destroys the claim between the read and the join Even when the property is established at the read, it is fragile across the plan. Common losses: - **A step the plan cannot see into** that receives the key and may return a different one. If the engine cannot tell that the key is unchanged, it must assume it changed. - **A concatenation or union** with an input that is not divided the same way; the result as a whole no longer satisfies the conditions. - **A change in the piece count** anywhere upstream, including merging pieces together in place to reduce their number. - **A rewrite of the key column** - a cast, a trim, a concatenation of two columns - which produces a key the recorded rule was not applied to. - **Joining on a subset of a composite key**, where the recorded division used more columns than the join matches on. By contrast, steps that remove records or columns without relocating anything usually preserve the property: filtering leaves each surviving record in the piece it was read into, and dropping columns changes the record width and nothing else. Exactly how far a given plan can see through such steps varies, which is why the reliable habit is to keep the key untouched between the read and the join. ## Why the default is to move The asymmetry is the whole reason engines behave conservatively here. If the engine assumes a layout that does not actually hold, the join silently misses matches: worker three never sees the rows for its keys that landed on worker nine, and the result is wrong with no error. If it redistributes when it did not strictly need to, the job is slower and still right. Given a choice between an unchecked speed-up and a guaranteed correct answer, engines take the answer, and a good candidate says that out loud rather than treating the redistribution as a bug. ## How you find out which case you are in The practical move is to read the job's plan and look for the redistribution before the join. Two outcomes: 1. **It is absent** - the property was established and carried. Record what made that work, because it is easy to lose in the next change. 2. **It is present** - work backwards through the steps between the read and the join, looking for the first one that could have relocated or rewritten the key. That step is where the property died. A useful discipline on a job that depends on this: assert the expectation somewhere the failure is visible. A job whose whole cost model assumes no redistribution should not be allowed to quietly become a job that redistributes, because nothing else in its output will tell you. ## Continuous jobs are a different shape of the same problem Where the runtime is continuous, the division comes from a declared operator width rather than from the stored bytes, so the question becomes whether both streams enter at the same width with the same key extracted and the same rule applied. A stored layout can still decide how the first read is divided, but it does not survive a redistribution imposed by a width change, and restarting at a different width re-shuffles everything. The general lesson is the same in both shapes: the saving belongs to whatever the runtime can establish, not to what is true.