A workflow scheduler is given one step per input file and there are 40,000 files - what breaks, and where does that fan-out belong?

## Two different units of work The error is a units error, and naming the units is most of the answer. A **workflow scheduler** - a system whose only job is to start other programs in a required order and record whether each one finished - has exactly one unit: a step, meaning a whole program it starts as its own process (sometimes a unit run inside a shared worker instead, but the accounting is the same). Every step carries: - a fixed start-up before any of your code runs; - a durable record of its existence and its outcome, written and updated by the scheduler; - a slot against whatever concurrency limit the scheduler enforces; - a box in a picture that a human is expected to read. A **cluster execution engine** - a system you hand a whole program to, which splits that program's work across many machines - has a different unit: one piece of input, handled by one worker thread inside one submission. A piece costs no program launch, no durable bookkeeping row and no diagram box. Its cost is scheduling a small unit of work onto a thread that is already running. ## What 40,000 steps actually does 1. **Fixed cost times 40,000.** Whatever a launch costs - and how much it costs depends entirely on how the machines are supplied - it is now the dominant term. A few seconds of real work per file against a fixed launch cost per file means most of the wall clock is spent starting things. 2. **The bookkeeping becomes the workload.** The scheduler writes and updates state for every step. Its own storage and its own loop were sized for graphs of tens or hundreds of steps, and 40,000 of them makes the scheduler the slow component in a pipeline that computes almost nothing. 3. **The concurrency limit serialises it.** A scheduler caps how many steps run at once, for the entire installation or per queue. That cap is now your parallelism, and it is shared with everybody else's pipelines - so this graph both runs slowly and starves its neighbours. 4. **The picture stops being a picture.** Nobody can look at 40,000 boxes. The one thing the workflow graph is genuinely good at - showing a human the shape of the pipeline - has been destroyed by using it as a loop. 5. **Per-file outcomes become 40,000 decisions.** Every step reports individually, so the pipeline's outcome is now an aggregate somebody has to compute and interpret. What the scheduler then does about a failed step is its own subject; the point here is that you created 40,000 of those events out of one logical computation. ## Where the fan-out belongs | the fan-out is over... | belongs to | why | |---|---|---| | input files, records, keys, time slices of one dataset | one submission to the engine | it is data parallelism; the engine's unit is a piece of input and it costs no launch | | distinct programs, different runtimes, different owners | the workflow graph | each really is a separate thing to start, and there are few of them | | a handful of independent datasets, each with its own pipeline | the workflow graph | tens of steps, not tens of thousands, and each box means something to a reader | The shape that works is one submission handed the *list* of 40,000 files - or the directory - so the engine reads them as one input and runs the same logic over every piece in parallel. How that input is then cut into pieces, and whether many tiny files should be coalesced first, is the engine's own splitting question and a different subject; what matters here is that the count stopped being 40,000 *programs*. ## The same error one level down Once you see it, the family is obvious: a step per customer, a step per hour of history, a step per row of a control table, a step that starts a program which processes exactly one record. Each of them hands per-record or per-item work to something that can only launch programs. The signature is always the same - **the number of boxes in the workflow graph is proportional to the size of the data.** A workflow graph should have the number of boxes a person could draw on a whiteboard; if it grows when the input grows, the fan-out is in the wrong system. There is a legitimate middle case worth conceding: a small, bounded fan-out over a handful of genuinely independent datasets - five sources, five steps - is fine and often clearer than one submission that reads all five. The line is not 'never fan out in the scheduler'. It is that the fan-out must be bounded by something structural, not by the data's cardinality.