When does compressing a step's output before it crosses the network make a redistribution slower rather than faster?

## The trade, stated plainly A redistribution — every worker sending each record it holds to whichever worker will handle that record's key — spends bytes on the network and processor time turning records into those bytes and back. Compressing the output before it is sent moves cost from the first budget to the second. That is the whole of it: **fewer bytes on the wire, more processor time on both sides**. Whether it is a win therefore depends on which of the two the job is actually short of, and no general answer exists. ## When it pays - The network link, not the cores, is what the movement is waiting on. Workers with idle processor capacity and a saturated link are the textbook case. - The data genuinely compresses. Repetitive text, low-cardinality string fields repeated across millions of records, sparse or defaulted fields — all shrink substantially. - The producing side's output is **materialised**: written into one bucket per destination on local disk and collected afterwards. Then compression saves the local write and the local read as well as the transfer, so the same processor spend buys three savings instead of one. - The same bytes are read more than once. Paying the encoding cost once and the decoding cost several times can still come out ahead. ## When it loses 1. **The job is already processor-bound.** If every core is busy, compression's work does not overlap with anything; it extends the producing side directly, and the bytes it removed were not the constraint. 2. **The ratio is near one.** Random identifiers, hashes, encrypted values, and payloads that arrived already compressed do not shrink. Both sides still pay to run the codec over them. Re-compressing an already-compressed document is not impossible, as is sometimes claimed — it simply returns almost nothing while charging full price, and can even grow the stream slightly. 3. **The records are tiny and numerous.** Per-block framing and per-message overhead can exceed the saving when each unit is a few dozen bytes. 4. **The choice was made for maximum ratio on a fast network.** Ratio and speed are separate axes, and past some point extra ratio costs more processor time than the removed bytes were worth. ## Where the cost lands | side of the movement | what it pays | when it shows up | |---|---|---| | producing side | encoding each record, then compressing the bucket it writes | while the producing step is still running, directly on its wall-clock | | the network | fewer bytes, in proportion to the ratio actually achieved on this data | during the transfer | | collecting side | decompressing, then decoding every record it reads | as it pulls its share, before any of the downstream work begins | The important point in the table is that the saving is one row and the cost is two. A candidate who only names the middle row is reasoning about half the ledger. ## What varies between engines Describe the mechanism, not one product's behaviour: - Some runtimes compress intermediate output by default and others do not, so "the bytes on the wire" may already be the compressed figure — or may not be — before you change anything. - Where the runtime cuts the job at barriers and materialises output, the local-disk saving is real; where it hands each record downstream as it is produced, there is no local write to save and only the wire term is in play. - Runtimes that hold records in a compact binary layout they manage have already removed much of the redundancy that a general-purpose codec would otherwise find; runtimes holding ordinary language objects produce a byte stream with more slack in it, and compress better for the same effort. ## How to decide in an interview answer Say which resource is scarce, and say how you would know. The honest sequence is: establish the byte volume of the movement and the processor utilisation of the workers while it runs; if cores are idle and bytes are large, compression is the lever; if cores are pinned, look at removing bytes instead of re-encoding them — dropping fields nobody downstream reads, or not widening the record before the move at all. Which particular codec trades ratio against speed how is a serialization question rather than a movement question, and a good answer hands it off rather than reciting it. The failure mode to avoid is treating a compression setting as a performance knob with an obviously correct position. It has two positions, each correct for a different cluster, and the data in front of you decides.