A job's only input is one 40 GB file compressed as a single stream — how many worker threads can read it?

## What "a piece of the input" means Before a **finite job** — a run over an input that ends, so the engine can measure the stored bytes before it starts — executes anything, the input is cut into **pieces of the input**: one slice of the stored input that a single **worker thread** reads and processes from start to finish. A worker thread is one lane inside a **worker process**, which is one operating-system process on one machine holding its own memory and several such lanes. The **piece count** is how many slices the run is divided into, and it is the ceiling on how many lanes can be busy. The machine count only decides how many lanes exist. That cut is not a free choice. It is derived from what is on disk, and the first thing the reader must establish about each file is whether it can be entered anywhere other than at the beginning. ## Why a stored form can or cannot be divided To make two pieces out of one file, a reader must be able to (1) begin decoding at some offset and (2) find the first whole record boundary at or after it. Forms that allow this build in entry points: - **Block framing** — the file is a sequence of independently decodable blocks, each with its own header and its own compressed payload, so a reader that lands mid-file scans forward to the next header and starts there. - **Sync markers** — a short, statistically unlikely byte pattern written between record groups, which a reader can scan for from any offset. - **Fixed-width uncompressed records**, where the boundary is arithmetic. A file compressed as **one continuous stream** has none of these. The decoder's state at byte N — its dictionary of earlier byte sequences — depends on every byte before N, so there is no offset a second lane could start from. The same holds for a file encrypted as one stream, for a single serialized document (one enormous array or one document tree) whose record boundaries cannot be located without parsing from the root, and for a text form whose records span lines with no framing to tell a mid-file reader whether it landed inside a record or between two. | stored form | pieces from one file | what decides it | |---|---|---| | whole-file stream compression | 1 | no offset the decoder can start at | | block framing, compressed per block | many | block headers are entry points | | records separated by sync markers | many | scan forward from any offset | | one serialized document | 1 | boundaries exist only relative to the root | ## What the single piece costs - **Concurrency** — one lane works and the rest idle; wall clock for the read is the single-lane decode rate, not the cluster's aggregate rate. - **Retry granularity** — the unit that fails is the whole piece, so a failure at 39 GB re-reads from zero. - **Memory** — whatever that pass accumulates accumulates inside one process instead of being spread across many. - **Machine time billed** — you pay for lanes that are doing nothing. What it does *not* cost is the whole job. The single piece constrains the read pass and the **narrow steps** fused into it — steps each worker finishes alone out of the piece already in its hands. As soon as the job performs a **wide step** — one a worker cannot finish from the records it already holds, because it needs records currently sitting on every other worker — the records are redistributed, and the steps after it run at whatever piece count that redistribution creates. A common deliberate shape is therefore: read the undividable file on one lane, redistribute immediately, then run everything expensive at full width. That pays one pass over the data to buy the parallelism back. ## What varies between engines - Deriving the pieces from stored bytes is a **finite job's** behaviour. In a **continuous job** — a run over an input with no end, where nothing about the input can be measured in advance — the author states a **declared operator width** instead, and it stands until the job is restarted. An undividable file read by a continuous job is read by one operator instance, and no width declaration changes that. - Some readers **pack several whole files into one piece** up to a byte target. That helps a population of small files and does nothing here, because one large undividable file is already one whole file. - Some engines plan one piece per file up front; others plan from measured sizes and discover at run time that only one piece produced records. The observable symptom — one lane busy, the rest idle — is the same either way. ## What to do about it 1. Store the data in a **block-framed** form. This is the only change that makes the file itself divisible. 2. Failing that, write **many moderate files** rather than one, so the piece count comes from the file count instead. What one job leaves behind is the layout the next job inherits. 3. Failing both, accept the single-lane read, redistribute immediately afterwards, and shape that first worker process for one heavy lane rather than for many idle ones.