In a job's printed plan, how do you locate the points where records cross the network, and what does a long unbroken chain mean?

## What you are looking for A **network crossing** is a point in the plan where records have to be sent from one worker to another, because the step above needs records that a different worker is holding. The industry term for that movement is a *shuffle*. It is the single most consequential thing the printed text tells you about a distributed job, because it is the one part of the work whose cost has nothing to do with how clever the computation is. A **phase** is a run of steps between two crossings that the engine can execute without moving any records at all. Many engines call such a run a *stage*, and most print the listing already cut into them. Locating the crossings and locating the phases is the same act seen from two sides: the crossings are the cuts, the phases are what lies between them. ## Two ways to find them 1. **Structurally, from the printed text.** Engines mark the cut. Depending on the engine you may see the listing grouped into numbered sections, a named node standing between two runs of steps, or an indentation break with the redistribution described on its own line. Whatever the spelling, the pattern is the same: the steps above the mark cannot begin on a piece of data until records have arrived from below. 2. **Semantically, from the operations.** You can predict the crossings before you look, which is why this question is asked. Records that must meet have to be moved to one place, so a crossing is forced by: - grouping by a key, where every record sharing a key has to be brought together; - joining two inputs on a key, unless both are already arranged by it or one side is small enough to be sent whole to every worker; - producing one globally ordered result, which needs a range of values per worker; - deduplicating or counting distinct values across the whole input; - any explicit re-arrangement of the data into a different number or arrangement of pieces. The interesting reading is the **difference between the two counts**. If your program changes grouping key twice and the plan shows four crossings, two of them came from somewhere you did not intend, and that gap is the finding. ## What an unbroken chain tells you A long run of steps with no cut in it is a positive fact, not an absence. It says that for every step in the run, each output piece is built from exactly one input piece — a *narrow* dependency, as opposed to a *wide* one where a step's output piece draws on many input pieces. Consequences worth stating out loud: - The whole run can be executed as one pass over each piece; engines that do **step fusion** collapse several adjacent per-record steps into a single pass, so no intermediate collection exists between them. - The number of steps printed in such a run says very little about cost. Ten fused per-record steps in one phase are usually cheaper than one crossing. - Its width is inherited. A chain with no crossing in it cannot change how many pieces are being worked on, so everything in it runs at the width the input arrived with. ## Where the models genuinely disagree What a crossing *is made of* is not the same everywhere, and this is the part candidates get wrong by generalising from the engine they know. | Engine model | What a crossing looks like at run time | |---|---| | One grouping step at a time, every intermediate written to disk | The producing side writes its output out, the consuming side fetches it afterwards; the crossing is a materialised handover | | One fixed graph kept running, each record passed through as it arrives | Records are pushed across the connection as they are produced, with no materialisation; the crossing is a live connection, not a handover point | | Continuous work as a fast succession of small finite jobs | Each small job crosses the network the same way a finite job does, once per crossing per small job | Two cautions follow. First, do not assert that everything downstream waits for everything upstream: whether a crossing paces the job that way depends on the model, and the rule about what a phase boundary does to pacing is a subject of its own. Second, a crossing is not automatically present just because a grouping is: an input that already arrives arranged by the grouping key can be grouped with no movement at all, and a good plan reading notices when the engine has spotted that. ## Using the finding Once you have the crossings located, the reading is comparative rather than absolute: - Count them, and account for each one against something in your program. - Note which are between the read and the first aggregation, since work done before a crossing travels less far than work done after it. - Note where two crossings sit adjacent with little between them, which usually means the data was re-arranged twice by different keys. What the plan will not give you is how much each crossing actually moves. That is a measurement, and costing a movement is a separate subject with its own owner.