In a chained generator pipeline, when does each stage actually run?

Composing generators into a pipeline is the standard Python idiom for transforming a stream, and its execution model surprises people the first time a print statement shows up in an order they did not expect. ## Building is not running ```python def read(rows): for row in rows: yield row def clean(rows): for row in rows: yield row.strip() def keep(rows): for row in rows: if row: yield row pipeline = keep(clean(read(source))) ``` After that last line, **not one line inside `read`, `clean` or `keep` has executed**. Calling a generator function does not run its body; it creates a generator object with a suspended frame parked before the first statement. Three calls produce three generator objects, each holding a reference to the one upstream. That is the entire cost of construction. ## Pull, one item at a time Execution starts when something consumes the last object in the chain. `for row in pipeline:` calls `next()` on `keep`'s generator. `keep` resumes and immediately needs an item, so it calls `next()` on `clean`. `clean` resumes and calls `next()` on `read`. `read` resumes, pulls one row from the source, and yields it - and now the value climbs back: `clean` strips it and yields, `keep` tests it and yields, the loop body receives it. Then the loop asks again and the whole sequence repeats for item two. So the ordering is **item-major, not stage-major**. Stage one does not process the entire input before stage two begins. At any instant, exactly one item is in flight, plus whatever each stage keeps in its own locals. That is the mechanical reason a chain of ten generators over a ten-gigabyte input still runs in flat memory: nowhere in the chain does a full collection exist. The direction of control is the "pull" in pull-based: the consumer sets the pace and demand travels upstream. Nothing is pushed at the consumer; a slow consumer simply calls `next()` less often, and the upstream stages sit suspended doing nothing at all. Backpressure is free, because there is no buffer to overflow. ## What defers with the data Because the bodies run at consumption time, everything they do runs then too. **Exceptions.** If `clean` raises on row 4000, the traceback surfaces from the `for` line that was draining the pipeline, not from the line that built it. A `try`/`except` wrapped around construction catches nothing. This regularly confuses people who put pipeline setup in a defensive block and then wonder why errors escape. **Side effects.** A stage that opens a connection, writes a log line or increments a counter does so during consumption, so the ordering of those effects interleaves with the consumer's own work rather than happening in a neat block. **Early exit is a real saving.** `break` out of the loop, or take a few items with `itertools.islice`, and the upstream stages never compute the rest. A pipeline that would have processed a million rows might do a hundred rows of work. The list-building equivalent has already paid for all of them. **Cleanup.** When a pipeline is abandoned and collected, each generator is closed, and a `finally` in a stage runs then - which may be much later than the code that built it expects. If a stage owns a resource, drive it under a `with` block or close it explicitly rather than trusting the timing. **Profiling.** Time appears to be spent in the consuming loop, since that is where the frames resume. Reading a flat profile of a lazy pipeline without knowing this leads people to blame the wrong function. ## Where the model breaks Any stage that must see everything before it can emit anything destroys the property for the whole chain: a stage that sorts, that dedupes into a set, that groups by key, or that simply appends to a list and yields at the end. Such a stage becomes a barrier - upstream runs to completion, memory spikes to the size of what it holds, and only then does the downstream half start. Sometimes that is required, and the right answer is to bound it: sort within batches, or cap the buffer, rather than pretending the chain is still lazy. ## The interview answer Say: building the chain runs nothing; the consumer pulls; one item travels through all stages before the next one starts; therefore memory is flat, errors and side effects appear at consumption, and stopping early skips the remaining upstream work entirely.