Between the stages of a concurrent processing chain you can place an unbounded queue or a fixed-capacity one. Which do you choose, how do you pick the capacity, and what happens to the neighbouring stages when that queue is full or empty?

## Why bounded A queue between two stages exists to decouple them so a brief slowdown in one does not immediately idle the other. It does **not** exist to reconcile a permanent rate difference — nothing can do that except making the slow stage faster or the fast stage slower. If the producing stage is persistently faster than the consuming stage, an **unbounded** queue grows forever. Three bad consequences: memory grows until the process dies; per-item latency grows linearly with elapsed time (an item entering after an hour of imbalance waits an hour); and the system looks *healthy* the whole time — every worker busy, no errors — so the true fault (a bottleneck) is invisible until a hard crash. Worse, at crash time you lose everything buffered. A **bounded** queue makes the mismatch visible instantly and locally: the queue sits at capacity, the upstream stage spends its time blocked, and the queue-depth metric points straight at the constraint. Memory is capped by design, and latency is capped at depth ÷ throughput. ## Full and empty: the two stall modes - **Full queue → upstream stalls.** The producing stage blocks when it tries to hand off. It stops consuming from its own inbox, which fills, which blocks the stage before it. This backward propagation is the pipeline's self-regulation: within one round trip the whole chain runs at the bottleneck's rate, and pressure arrives at the source, which is the only place that can meaningfully decide what to do about excess work (block the caller, shed, spill, slow the poll). Choosing among those overflow policies is a separate design topic; the pipeline-internal mechanism is simply that a bounded handoff blocks. - **Empty queue → downstream stalls.** The consuming stage has nothing to do — a **bubble**. Bubbles are pure lost capacity: the bottleneck idling for 10% of the time costs 10% of maximum throughput and cannot be recovered later. Causes include an upstream stage with high variance (an occasional slow item), buffers too small to cover that variance, expensive handoffs, and start-of-stream fill. A healthy pipeline shows queues that oscillate: partially full, occasionally empty. Persistently full means the downstream stage is the constraint; persistently empty means the upstream stage (or the source) is. ## Choosing the capacity Two forces pull in opposite directions: - **Lower bound (avoid bubbles).** The buffer must cover normal variation. If the producer occasionally pauses for t and the consumer serves at rate r, roughly r · t items of buffer keep the consumer fed. Batchy producers need at least one batch of room. - **Upper bound (latency).** By Little's law the delay a full queue adds is depth ÷ throughput. At 100 items/s, a 1000-slot queue adds 10 seconds. Derive the maximum from the latency budget: max in-flight = target latency × throughput, then divide that allowance across the stages. In practice, start small (tens), measure starvation (time the consumer spends blocked on empty) and latency, and grow only if starvation is real. A large buffer that never drains is a bottleneck in disguise. ## Batching When per-item handoff cost is significant relative to per-item work, move items in **batches**: one lock acquisition, one wakeup, one network or disk operation for k items. Batching raises throughput (amortized overhead, better cache and I/O behaviour) and is often the difference between a pipeline that helps and one that is all overhead. The costs: latency grows by up to a batch-fill time, and a partly filled batch must be flushed on a timeout or it can sit forever at low load — so batching policies are always "k items **or** t milliseconds, whichever first". Batch sizing is the same tradeoff as buffer sizing: bigger batch, more throughput, more latency, more work lost or replayed on failure. ## Deadlock hazards specific to bounded buffers Bounding introduces blocking, and blocking introduces deadlock: - **Cycles.** If a later stage feeds work back to an earlier one over a bounded queue, both can block on full queues waiting for each other. Break it with a cycle-free graph, a dedicated monitored feedback path, or non-blocking offer-with-drop on the back edge. - **One worker on both sides.** A worker that drains queue A and pushes to queue B must never be the only worker that also drains B; if B fills, it blocks and can never drain B. - **Fan-out/fan-in join.** A stage that must take one item from each of two bounded inputs deadlocks if one input's producer is blocked behind the other's full queue. - **Shutdown.** Draining requires ordered termination: stop the source, send end-of-stream markers downstream, let each stage finish and forward the marker. Killing a middle stage while its neighbours block on it wedges the chain.