Splitting the Work

One. A stored form that cannot be decoded from an arbitrary offset yields exactly one piece of the input — one slice a single worker thread reads end to end — so every other thread in the cluster stays idle during that read.

Dependency shape decides: if each output depends only on records already in the thread's piece, it finishes alone — a narrow step. If one output needs records spread across every piece, the step is wide.

Eight. The piece count is the ceiling on how many worker threads can be busy; the machine count only decides how many threads exist, so the other 192 lanes stay idle.

Roughly one file per piece, so about 2,000. Each worker thread writes the records it already holds, where it is; nothing gathers them first. The piece count at the final step, not the machine count, set the file count.

A target size per piece and the total worker-thread count. Size keeps each piece inside one thread's memory and worth its fixed cost; thread count keeps the lanes busy in even waves. Reconcile them rather than using either alone.

It varies by engine and by job kind. Some derive and freeze the count at submission; some let the author set it per redistribution point and adjust it between step groups while running; a continuous job's declared width usually stands until a restart, and some platforms never expose it at all.

Whole directories are matched against the filter while the plan is being built, so their files never become pieces of the input at all. Both the piece count and the bytes read fall, before a single file is opened.

The filter and the derived column are narrow: each output depends only on the record in hand. The per-country distinct count is wide, and so is the global ordering — their outputs depend on records that may sit in any piece.

A finite job derives it: the stored input is measured and cut into pieces before the run. A continuous input has no end to measure, so the author declares how many copies of each operator run.

Because the two numbers multiply. Each worker thread writes what it holds into the directory each record's value names, so a piece holding all 200 values leaves 200 files. Across 2,000 pieces that is 2,000 x 200 files, each tiny.

Each piece carries a fixed cost that does not shrink with its bytes: it is created, dispatched, tracked and collected, and its input is opened and closed. At 40 KB per piece that bookkeeping outweighs the reading, and the coordinating process becomes the bottleneck.

Both writes must have used the same key, the same function from key to piece number, and piece counts the engine can relate - normally the same count. Only then does every matching pair already sit on one worker.

It depends on how long the piece will run. Waiting buys a cheap local read but burns an idle lane now; starting further away pays a remote read for the piece's whole life. Short pieces should never wait long.

Regroup once on the account key, then keep that division. After the first wide step every account's records already sit together, so later steps keyed on the same account finish locally — unless something in between re-divides or hides the key.

No extra concurrency. One piece is read by one worker thread from start to finish, so the run's ceiling is one lane and the rest of the cluster idles no matter how many machines are added.

Bytes per piece grew fifteenfold while the count stayed put, so every piece is now oversized: uniformly longer runtimes, new disk spilling, memory pressure and expensive retries. The tell is that the whole duration distribution shifted, not that one piece is slow.

Later writes that did not follow the agreement: a backfill or a second pipeline appending files divided some other way, a piece count re-tuned on one side, or a key column quietly redefined. Nothing fails; the saving just disappears.

Because the run must know the layout holds, not merely benefit from it. Unless the key, rule and piece count are recorded where the plan reads them and survive every earlier step, it moves the records.

It disappears, on purpose. No compute machine holds the bytes, so every candidate is equally far and placement collapses to free capacity. The industry traded that optimisation for compute it can resize or discard independently of the data.

Planning dominates, not reading. Listing 300,000 objects and turning each into a piece of the input costs far more than the 15 GB of payload, and that work is serial, so idle worker threads cannot absorb any of it.

On a runtime that schedules the work between regroupings, three wide steps mean four separately scheduled units of steps. On a runtime where every operator runs at once, they mean three routing hops each record may make in flight.

Because the two numbers have different origins. The finite run derives its pieces from the stored bytes, so more data means more pieces and more busy lanes; the continuous job runs at a width the author declared, which no change in the input can move.

The limit is now the destination's, not the cluster's. Three things change: rows go in batches rather than one request each, the number of simultaneous writers is capped independently of the piece count, and the run holds a rate ceiling the store was measured to survive.

Judge it against the population of reading jobs, not one query: how often each candidate column is filtered on, how many directories and how large a file each granularity leaves, and how skewed the column is. Fine granularity buys elimination for one access pattern and charges everyone else.

Two things: bytes a machine wrote to its own disk during the run, and work for a key whose accumulated state one worker already holds. Both make that machine non-interchangeable, which is the real cost.

Only if that key dominates the joins and the table is read far more often than written. The standard freezes a piece count into shared data, taxes every writer, and needs an owner and a check to survive.

Make the write budget a property of the destination, not of each job: a measured concurrency and rate allowance per store, divided among the jobs that write to it, with batching required, writer concurrency decoupled from the piece count, and a defined behaviour when the allowance runs out.