A three-step processing chain runs each step on its own worker with queues between them, taking 10 ms, 50 ms and 10 ms per item. What throughput and per-item latency do you expect, and what would you change to raise throughput?

## The arithmetic In steady state a pipeline emits one item per **slowest stage** time, because every item must pass through that stage and it can serve only one at a time. So: - **Throughput** = 1 / max(10, 50, 10) ms = 20 items/s - **Latency** = 10 + 50 + 10 = 70 ms minimum, plus time spent waiting in buffers - **Speedup over sequential** = sum / max = 70/50 = 1.4x, despite using three workers - **Balance efficiency** = sum / (n · max) = 70 / 150 ≈ 47% A perfectly balanced 3-stage split of the same 70 ms of work (23.3 ms each) would give 43 items/s — over twice as much from the same three workers. That gap is the whole point of balancing: an unbalanced pipeline pays for n workers and gets sum/max of them. ## Why the fast stages don't matter Optimizing a 10 ms stage to 5 ms changes throughput not at all: the bottleneck still gates the rate, and the fast stage simply idles more. This is Amdahl-flavoured reasoning applied to stages — effort spent off the critical stage returns nothing. It also means measurement must identify *which* stage is slowest before any tuning starts. The cheapest instrument is **queue depth**: the buffer feeding the bottleneck is persistently full (or growing) while the buffers downstream of it are persistently empty. That signature localizes the bottleneck without profiling any code. ## What happens if arrivals exceed 20/s If the source pushes 30 items/s into a pipeline that drains 20/s, 10 items/s accumulate. With an **unbounded** buffer the queue and the memory footprint grow without limit and per-item latency grows linearly with time — the system looks healthy (no errors, workers busy) until it dies. With a **bounded** buffer the producer blocks when it is full, which pushes the pressure back to the source where it can be handled. Bounding turns a silent memory leak into a visible slowdown at the entrance. ## Ways to rebalance **Reduce the work.** Caching a lookup, batching the I/O the stage performs, or moving per-item work out of it is always the first move: it costs no extra worker and no ordering. **Replicate the stage (widen).** Run k copies of the slow stage pulling from the same queue and pushing to the next; effective service time becomes 50/k ms. This is the standard fix and the reason real pipelines are stages-of-pools rather than stages-of-single-workers. Caveats: (a) the underlying resource must scale — five copies of a stage that all hit one saturated database gains nothing and may lose; (b) **ordering is lost**, since copies finish out of order, so if downstream needs order you must attach sequence numbers and re-sequence, which reintroduces a buffer and head-of-line blocking; (c) if the stage has shared mutable state, replication needs synchronization that can eat the gain. **Split the stage (deepen).** If the 50 ms decomposes into two 25 ms halves with a clean handoff, you get two stages and a 25 ms cycle while keeping strict ordering with single-threaded stages. It only works if the work genuinely divides and the intermediate value is cheap to pass. **Merge cheap stages.** Two 10 ms stages that use the same resource can become one 20 ms stage: one fewer handoff, one fewer queue, one worker freed to be spent on the bottleneck. Handoff cost is real, and stages that are much cheaper than a handoff are net negative. ## The bottleneck always exists After fixing the 50 ms stage you have 10/10/10 and a 100 items/s pipeline whose bottleneck is now every stage at once. The next constraint may be the source, the sink, the queue handoff cost, or a shared resource behind several stages. Balancing is iterative, and the honest answer in an interview names the measurement loop — instrument stage service times and queue depths, fix the top one, re-measure — not a one-shot fix.