A grouping that ran inside one process is now run across twenty machines. What work does only the distributed run do?

## One address space, then twenty Inside one process — an **in-process engine**, a library that does the same relational or numeric work inside a single address space, where no two records are ever separated by a network — a value handed from one step to the next is handed by reference. Nothing is copied, nothing is encoded, and both steps see the same bytes in the same memory. The moment the same computation is split across machines, two steps that live on different **worker processes** — each one process on one machine, running pieces of the job and holding their data in its own memory — cannot do that. A step that needs records currently held by other workers forces every worker to send records to every other: a redistribution, the thing the field usually calls a **shuffle**. Every value that crosses has to be turned into bytes, moved, and rebuilt on the far side. That work is manufactured by the split. It is not part of the answer, and the single-machine run pays none of it. ## Three meters, not one **Per record.** Each value crossing a boundary is encoded on the way out and decoded on the way in. The encoded form usually carries something the in-memory form did not need — type information, a field layout or a length — so the bytes grow. The decode side allocates a fresh value for every record, which the memory manager must then reclaim; on a hot path that reclamation is frequently a larger cost than the network itself. **Per unit of work.** Each transfer carries framing: a destination, a length, often a checksum and an index of what it contains. Where a redistribution is organised as every producing piece having something for every consuming piece, the number of these grows as the product of the two counts, so the bookkeeping grows faster than the data does. Cut the input into very many tiny pieces and the framing can outweigh the records inside it. **Per program.** The code has to reach every worker: the functions themselves, their libraries, and — the part candidates forget — the values a function captured from the surrounding program. A lookup structure captured by a function is not free; it travels, and engines differ in whether it travels once per worker or once per unit of work, which is a difference of orders of magnitude. | what moves | one process | across machines | |---|---|---| | a value between steps | a reference | encode, transfer, decode, reclaim | | a unit of work | a call | framing, routing, accounting | | the code and captured values | already present | copied to every worker | | grouped records | rearranged in memory | rearranged across a network | ## This is not the same question as which encoding to pick How compactly a scheme writes values and how fast it reads them back is a subject in its own right and it certainly changes the size of this bill. The point here is prior to that: the bill exists at all, per record, in both directions, at every boundary, purely because the work was spread out. Compression is the same trade one level down — fewer bytes on the wire bought with processor time on both ends — and whether it wins depends on how the cluster's network capacity compares with its processor capacity, which is why the same setting helps one cluster and hurts another. ## What varies across engines - **Record representation.** Some engines keep records in a compact layout they manage themselves and can move those bytes with very little per-record work. Others hold ordinary language objects and pay a full encode and decode at every boundary — a several-fold difference in both footprint and processor time over the same records. - **Where the bytes rest in transit.** In the older batch lineage a redistribution is written to the producing machine's local disk and fetched by consumers afterwards, adding a durable write and a read to every transfer. A runtime that handles each record on arrival instead pushes records across the network as they are produced, with nothing written down — which is exactly why its recovery story is a different subject. - **How far the tax reaches.** Where a chain of rounds each writes its whole output to durable storage before the next round reads it, the tax is paid in full between every pair of rounds; where steps are fused into one pass over the data, values that never leave a worker are never encoded at all. ## What a candidate should say Say that the distributed run does the same logical work plus a tax with three meters, name the meters, and say where the tax falls: at every boundary between steps that must exchange records, and never inside a chain of steps that stays on one worker. Then give the measurement — bytes written and read for redistribution against bytes of input. When they are the same order of magnitude, the run is spending as much moving data as reading it. Add the shape: because two of the three meters are per record and per piece rather than per byte, the same volume in many small records costs materially more than in fewer, larger ones.