Why does grouping a billion rows by a two-value status column leave most of a hundred-worker cluster idle?

## Two separate properties of the key Work spreads evenly across a step that groups by key only when the key satisfies **both** of these: 1. **Cardinality** — there are at least as many distinct values as there are destinations, so every destination can be given something. 2. **Frequency shape** — no single value, and no small handful of values, accounts for a large share of the rows. These are independent, and an interviewer will happily test either. A two-value status column fails the first spectacularly. A customer identifier with fifty million distinct values passes the first and can still fail the second if one guest-checkout identifier carries a third of the traffic. ## Why cardinality is a hard ceiling A **redistributing step** — a step that cannot be computed from what one worker already holds, so every record is first sent to the worker that owns its key — assigns destinations with a **key-to-destination rule**, the function from a key value to the number of the destination that receives it. The same value always yields the same destination. Therefore the number of destinations that can receive **any** record is at most the number of distinct key values. With eight distinct values and two hundred destinations, at most eight destinations do work and at least 192 do nothing at all. Raising the number of pieces to four hundred changes the figure 192 to 392 and nothing else. The number of pieces controls how finely the *key space* is divided; it cannot divide a key. ## Key-frequency shapes and what each looks like | Key-frequency shape | Destinations doing work | What you see | | --- | --- | --- | | few distinct values, evenly spread | as many as there are values | most of the cluster idle; the busy pieces are all equally slow | | many distinct values, evenly spread | effectively all of them | the healthy case | | many distinct values, one dominant | all, but one is enormous | one piece runs for hours; the rest finish in seconds | | many distinct values, a handful dominant | all, with several enormous | a few long pieces and a long tail of quick ones | Row one is the case in the question, and it is worth noticing that it *is* a heavy-key case: if a column has two values each covering half the data, both of them are heavy keys by definition. The symptom differs from the single-outlier case — several equally slow pieces rather than one — but the cause is identical, and so is the fact that adding machines does nothing. ## Cardinality above the destination count is still not a guarantee The mapping **assigns**, it does not **balance**. It is chosen so that unrelated values tend to land on different destinations, but nothing arranges for equal counts. With exactly as many distinct values as destinations, some destinations will be assigned two values and others none — the same arithmetic that makes a random allocation of a hundred items into a hundred boxes leave roughly a third of the boxes empty. This matters in practice when the key cardinality is only a small multiple of the destination count: a key with 300 distinct values across 200 destinations will not be evenly spread even if every value has the same number of rows. The practical rule of thumb an interviewer likes to hear is that you want distinct values comfortably exceeding the destination count — a multiple, not a margin — *and* a frequency distribution without a dominant value. ## What varies between engines - Whether an unused destination costs anything at all differs. In a finite job, an empty destination is usually a piece that starts and finishes immediately; in a job over an endless input, the width is typically fixed for the life of the job, so an instance that owns no key simply sits there holding resources until the job is restarted. - Where an engine re-decides the part of a plan it has not run yet from the sizes a finished step actually produced, it may merge many small or empty destinations into fewer pieces after the fact. That trims the overhead of the empty ones; it does not create parallelism that the key's cardinality never permitted. ## What an interviewer is listening for That you check the key before you check the cluster. "How many distinct values does the grouping key have, and what does the commonest one cover?" answers both failure modes in one question, and it is the answer that distinguishes someone who has debugged an idle cluster from someone who has only read about parallelism.