Where a producing operator pushes each record downstream as it is made, what replaces the bucket a consumer would otherwise collect?

## The half that does not change This regime is often described as if it were a different exchange. It is not. The record's key is still put through the **routing rule** — the rule that names exactly one destination per key — and equal keys still converge on one consumer, which is the entire point of a redistribution. Buckets still exist as a *concept*: the producer still separates its outgoing records by destination. What differs is where a bucket lives and who moves it. ## What a bucket is when nothing is stored - It is a **small send buffer per destination**, held in the producing operator's memory. - It is flushed when it fills, or after a short interval so a trickle of records does not sit waiting forever. - It is addressed to a **specific running instance** of the consuming operator, over a channel that stays open for the life of the exchange. Designs vary in whether each producer-consumer pair gets its own network connection or many logical channels are multiplexed over fewer physical ones. - It is **gone once acknowledged onward**. There is no offset to come back to and no file to reopen. The fan is the same arithmetic as before — one channel per producer-consumer pair, so the product of the two widths — but it is paid as concurrent channels and buffers rather than as transfers requested later. ## What the consuming side does instead of collecting It receives. It does not decide when its input arrives, it cannot ask for a particular producer's contribution, and it cannot ask twice. That produces four properties an interviewer is listening for: 1. **Both sides are alive at once.** The producing and consuming steps hold capacity simultaneously for as long as the exchange lasts, rather than the producing side completing and its workers being handed back. 2. **No second read.** A consumer that fails cannot re-gather what it was sent. The job returns to a recorded point and re-derives the input from the source — a recovery mechanism of its own, and a different subject from the exchange. 3. **A lost machine takes something different.** There is no written share to lose. What is lost is the records in flight and whatever the consumer had accumulated, which is why recovery here is about restoring a consistent picture rather than re-making a bucket. 4. **The sides are coupled.** A consumer that cannot keep up stops draining its channel, the producer's buffers for that destination fill, and the resistance propagates upstream — backpressure, the mechanism by which a slow step throttles the ones feeding it. Reading and acting on that signal is an operating subject in its own right; what matters here is that a materialised exchange does not have it, because a slow collector simply asks later. ## Three regimes side by side | | shares written down first | pushed as produced | run as repeated small finite jobs | |---|---|---|---| | where a bucket lives | the producing machine's local disks | a send buffer in the producer's memory | local disks, once per little job | | how the consumer gets it | it requests its numbered share | it is sent, on an open channel | it requests, within that little job | | can a share be read twice | while those bytes survive | no | within that little job only | | both steps alive at once | not required | required | required within each little job | | what a machine loss takes | the written shares on it | in-flight records and accumulated consumer state | that little job's shares | The third column is the one people forget, and it is a large part of this market: a continuous computation can be executed as a rapid succession of small finite jobs, each of which performs the written-down version at small scale. It therefore inherits materialised exchange mechanics *and* the ability to re-read a share within a little job, while looking from the outside like a continuous pipeline. ## How to answer without asserting one engine's model The failure mode in an interview is to state one regime as *the* way exchanges work. Both statements below are true only of their own regime: - *a redistribution writes to local disk and is collected afterwards* — the batch lineage's design; - *records cross the network as they are produced, with nothing materialised* — the record-at-a-time lineage's design. Name the regime with the claim: **where the shares are written down**, collection can outlive the producer and a share can be read again; **where records are pushed as they are produced**, there is nothing to re-collect, both sides must be alive, and the price of that is a coupling between them that has to be operated.