Who fixes the piece count of a distributed job, and when - at submission, mid-run, or only on restart?

## Why this question has four answers The **piece count** - how many slices the input is divided into, each processed by one **worker thread** from start to finish - is not owned by the same party in every engine. A candidate who answers with one regime is describing the engine they happen to know. The four regimes actually in the market: | regime | who sets it | when | cost of changing it | |---|---|---|---| | derived and frozen | the engine, from the stored bytes | before the run starts | resubmit the job | | author-set per redistribution | the author, in the program | at submission | edit and resubmit | | revised while running | the engine, between step groups | mid-run | none to the author | | declared width | the author, once | at submission | restart, with state moved | ## Derived and frozen The **two-phase disk-to-disk model** - the older execution model that materialises every intermediate result to disk between the two halves of a job - computes its splits from the stored input before a single record is read, and the number stands for the run. You influence it only through the input: how the files are laid out and how large they are. Nothing is adjustable once the run is in flight. ## Author-set at submission Most finite jobs on modern engines take a count from the author at each point where records must be redistributed. A **wide step** is one a worker cannot finish from the records it already holds, because it needs records currently sitting on every other worker; every such step ends one **step group** - the run of steps between two wide steps, scheduled and retried as one unit - and begins the next. The count applies to the pieces the next step group works on. Crucially, in several engines one number applies to *all* such points in the job unless the author overrides them individually, which is why one badly chosen value can be wrong in two directions inside the same run. ## Revised while running Some engines measure what a finished step group actually produced and change the count for the next one. That capability is **runtime replanning**, and it exists for some operators in some engines and largely not at all in continuous jobs. It is owned by Replanning at Runtime; the only thing to carry here is that it makes the initial number a starting point rather than a verdict, and only on the engines that have it. ## Declared width, standing until restart A **continuous job** is a run over an input with no end, so nothing about the input can be measured in advance and nothing can be derived from it. The author states a **declared operator width**: how many copies of each operator run. That number usually stands for the life of the job. Changing it means stopping and restarting, and that is where the real cost sits: 1. The job has to be stopped at a point whose position in the input is recorded, so it can resume without losing or repeating work. 2. Any state held per key has to be redistributed: the rule mapping a key to a lane depends on the width, so at a new width most keys belong somewhere else. 3. The job restarts behind the live input and has to catch up. Some engines make this a supported operation with tooling; others make it an outage. Either way it is a planned event, not a knob. ## Exposed to nobody A fourth case is growing: a platform that accepts work and derives the division itself, exposing no control at all. There your only levers are indirect - how much data the step actually has to handle, and the shape of what the previous job left on disk. ## What to say in an interview Three moves make this answer strong: - **Name the regimes rather than a setting.** *Fixed at submission here, per redistribution point there, restart-only for a continuous job* beats naming the one place your engine keeps the number. - **State the asymmetry.** A finite job can usually be corrected by resubmitting it; a continuous job cannot, so its width deserves more care up front. - **Say what you would still control** when you cannot set the number: the volume the step has to handle, and - by handing it to the right owner - the layout the previous job wrote, which is The Output Shape. ## The trap in the question The word *count* is doing three jobs, and only one of them is being set here. The **piece count** is the decision. The **worker-thread count** is a property of the cluster you were given. The **machine count** only decides how many lanes exist. An answer that drifts from the first to the third - *you change it by adding machines* - is the exact confusion this subject exists to correct: more lanes cannot help when the number of pieces is what is limiting the run.