How do you decide the capacity of a work queue sitting between producers and consumers, and what changes about the system's behaviour if you make that queue effectively unbounded?

## The core fact: queues do not create capacity In steady state, throughput = min(production rate, consumption rate). Adding buffer slots changes neither. A queue converts *variance* into *latency*: it lets a burst be served later instead of being dropped or blocking the source. That single sentence rules out the most common wrong answer, "make it big so we don't block". ## Little's law as the sizing tool Little's law states that for any stable system, **L = λ · W**: the average number of items resident equals the arrival rate times the average time each spends there. Rearranged for a queue: **W = L / λ**. If a queue holds 10,000 items and items arrive at 1,000/s, each new item waits ~10 seconds before service. That is not a tuning detail — it decides whether the work is still worth doing. Run it backwards to size the queue: pick the maximum queuing delay you can tolerate, multiply by the *service* rate, and that is your maximum useful capacity. ``` capacity_max_useful = acceptable_queue_delay × consumer_service_rate example: 200 ms × 500 items/s = 100 slots ``` Anything beyond that stores work that will breach its deadline before it is touched. ## The burst budget The other bound comes from *why* you wanted a buffer at all: absorbing short-lived mismatch. ``` capacity_for_burst = (arrival_rate − service_rate) × burst_duration example: (900 − 500) items/s × 100 ms = 40 slots ``` Typical stalls worth covering: a GC pause, a slow downstream call, a scheduler hiccup, a batch flush. Note that this formula only makes sense for *transient* excess. If arrival rate exceeds service rate on average, no finite capacity helps — the queue fills and you are choosing an overflow policy whether you admit it or not. The working capacity is `min(capacity_max_useful, capacity_for_burst)` sanity-checked against memory: capacity × bytes-per-item × number of queues × number of instances. Queues are frequently the largest single heap consumer in a pipeline, and this is easy to underestimate when items hold buffers or response objects. ## What unbounded actually changes **1. The overload signal disappears.** With a bound, sustained overload manifests as producers blocking or offers being rejected — early, local, and attributable. Unbounded, the system looks healthy right up until it isn't. **2. Memory becomes the implicit bound.** You still have a bound; it is "all available memory", it is discovered at runtime, and hitting it kills the process — losing every queued item at once, rather than shedding a few at the edge. **3. Latency grows without limit.** The Little's-law consequence: deep queue, stale work. In a request-serving system this becomes a pathological equilibrium — clients time out at 2 s, the server is 30 s behind, so every response the server produces is discarded, the client retries, arrival rate rises, and the queue deepens further. The system does maximum work and delivers nothing. This is the classic queue-induced collapse, and the fix is a shallow queue plus fast rejection, not more buffer. **4. Timeouts stop working the way you think.** A client timeout only helps if the server also abandons expired work. With a deep queue, cancelled work is still sitting there waiting to be processed. Effective designs stamp each item with a deadline and let the consumer drop expired items on dequeue. ## The overflow policy is the real decision Once you accept a bound, you must choose what happens when it is reached — and this is where the design judgment lives: - **Block the producer.** Correct when the producer is a pull-based source (reading a file, a partition, a batch) — blocking naturally throttles reading. Wrong when the producer thread is servicing a live client, because you convert queue pressure into held connections. - **Reject fast.** The producer gets an immediate failure and translates it into a 503 / retry-after / caller-side backoff. Best for request-serving; makes overload visible and bounded. - **Drop.** Discard the oldest (keep fresh data — metrics, telemetry, live positions) or the newest (keep history — audit trails). Only for loss-tolerant data, and always with a counter so the loss is observable. - **Run on the caller.** The submitting thread executes the work itself. This is elegant throttling — the producer physically cannot produce while it is processing — but it destroys latency isolation and can deadlock if the work depends on the queue. Always instrument: current depth, high-water mark, time-in-queue percentiles, rejection/drop counters. Queue depth is the earliest and cleanest saturation signal a pipeline has, and it should have an alert on it. ## Small capacities are usually right A counterintuitive but well-supported conclusion: in latency-sensitive systems the best queue is a shallow one — often a handful of slots, sometimes zero (a direct handoff, where a producer waits for a consumer to be ready). Shallow queues keep latency tight, surface overload immediately, and push the buffering decision to a layer that can make a policy call. Deep queues belong where the work is throughput-oriented and deadline-free — batch ingest, offline processing — and where the buffer is genuinely absorbing scheduled bursts. ## What to say when asked for a number Never a bare number. Say: what the consumer service rate is, what queuing delay the end-to-end budget allows, what burst you intend to absorb, what the item costs in memory, what happens at overflow, and how you will observe depth in production — then give the number that falls out, and say you will re-derive it from measurements once it is running.