After a team reorganized a request handler into a chain of concurrent stages with queues between them, total requests per second went up but the time an individual request takes got worse. Explain why that is the expected outcome and how you would reason quantitatively about the per-item time.

## Two different metrics **Throughput** is items completed per unit time (λ). **Latency** is the time one item spends from entry to exit (W). They are related but not interchangeable, and pipelining moves them in opposite directions. A pipeline raises throughput because several items are in flight at once, one per stage, so all the workers are busy simultaneously. It does nothing for a single item, which still visits stage 1, then 2, then 3, in exactly the order it would have visited them in a sequential implementation. Its best case latency is the same sum of service times as before. ## Why latency gets actively worse Three additions: 1. **Handoff cost.** Each stage boundary means enqueue, dequeue, and usually a worker wakeup, plus a scheduling delay before the next stage's worker is actually running. The item's data also migrates between cores, so the next stage starts with cold caches. For stages that are individually short, handoff can be a large fraction of the stage. 2. **Queue waiting.** An item that arrives while the next stage is busy waits. Waiting time dominates everything else once the bottleneck approaches saturation. 3. **Variance amplification.** Any burst in arrivals or any slow item at the bottleneck delays every item queued behind it — head-of-line blocking. This shows up as a much worse p99 than p50, even when average throughput looks excellent. ## Little's law: the quantitative handle **L = λ · W** — for any stable system, the average number of items inside equals arrival (= completion) rate times average time inside. It assumes nothing about distributions, only stability (nothing accumulating forever). Applied to a pipeline: - Whole pipeline: if 200 items are resident across all queues and stages and it completes 100 items/s, mean latency is W = 200/100 = **2 seconds**. If your latency budget is 200 ms at 100/s, you may have at most 20 items in flight — that is a hard cap on total buffer capacity, and it is the principled way to size queues. - Per stage: a stage serving 20 items/s with a queue averaging 40 deep contributes 2 s of waiting on its own. Sum the per-stage L/λ values to see which stage owns the latency. The useful reframing: **buffer depth is latency**. Adding queue capacity in front of a saturated stage does not add capacity; it converts a would-be rejection or upstream block into seconds of waiting. Throughput is set by the bottleneck's service rate and nothing else. ## Utilization makes it non-linear As a stage's utilization ρ (arrival rate ÷ service rate) approaches 1, queue wait grows roughly like ρ/(1−ρ) for random arrivals. At 50% utilization the wait is about one service time; at 90% it is nine; at 99% it is ninety-nine. This is why a pipeline that is "fully utilized" has terrible latency and why latency-sensitive systems deliberately run the bottleneck stage well below saturation. It also explains an observed cliff: raising load from 80% to 95% of capacity barely improves throughput but multiplies latency. ## How to diagnose Instrument each stage with two separate numbers: **service time** (how long the stage held the item) and **queue wait** (how long it sat in the inbox). The sum across stages is the end-to-end latency, and the split immediately says whether the fix is faster code or less queueing. Also record queue depth over time; a queue whose depth is high and stable is pure added latency, and its depth divided by throughput is exactly the delay it contributes. ## What to do about it - **Shrink buffers** to the smallest size that keeps the bottleneck fed through normal jitter. Latency falls immediately; throughput is unaffected as long as no stage starves. - **Raise bottleneck capacity** (replicate or make it cheaper) so utilization drops and waiting collapses. - **Merge trivially short stages** so handoff cost stops dominating. - **Separate classes of work** — a latency-critical path should not queue behind a bulk path in the same buffer. - **Accept the tradeoff explicitly.** If per-request latency is the product requirement, pipelining is the wrong tool: reduce per-request work, or overlap the request's own steps if they are independent. If total system rate is the requirement, pipelining plus small buffers is right and you defend the latency number with Little's law.