Your LLM server's waiting queue keeps growing under load — how do you diagnose the bottleneck?

## The mental model to start from A continuously batched server is a queueing system with an unusual service discipline. There is a **waiting set** of accepted-but-unscheduled requests, a **running set** participating in the current forward pass, and an admission decision taken before every iteration. A growing waiting set means admission is being denied step after step. Diagnosis is therefore the question: *which* admission condition is failing? There are only a handful of candidates, which is what makes this tractable in an interview. ## Candidate 1 — the concurrency cap binds The engine will not run more than its configured maximum number of sequences at once. If the running count is sitting exactly at that number, that is your answer. The tell that it is *artificial* rather than real is the KV-cache utilization gauge sitting comfortably below full: you have memory you are not allowed to use. Raising the cap converts that headroom into throughput, up to the point where cache becomes the binding constraint instead. This is the happy case, because it is a one-line fix. ## Candidate 2 — KV-cache memory binds Every running sequence holds cache blocks proportional to its current length, and the pool is fixed at startup. When utilization pins near full, the scheduler stops admitting, and under pressure it will evict running sequences to keep the ones it has making progress. The signature is a full cache gauge, a running count *below* the configured cap, and a waiting set that grows. No scheduler tuning fixes this, and raising the concurrency cap actively makes it worse. The real levers are elsewhere: reduce the maximum context you promise, quantize the cache to a smaller dtype, shard the model so more memory is available per replica, or add replicas. This case is worth naming precisely because it is where candidates reach for the wrong knob. "The queue is long, so let more requests in" is exactly backwards when the cache is the constraint. ## Candidate 3 — prefill is eating the step With chunked prefill on, each iteration's token budget is shared between the decode tokens of running sequences and a slice of some waiting request's prompt. A workload dominated by very long prompts spends most of its budget on prefill, so requests take many iterations to become fully prefilled and start decoding. Throughput in *output* tokens looks poor while the GPU is genuinely busy. The tell is a workload profile — median prompt length in the tens of thousands — combined with healthy cache and a running count below the cap. Here the lever is the token budget, plus anything that avoids the prefill entirely, such as prefix reuse for shared system prompts. ## Candidate 4 — you are simply out of capacity If the running count is at the cap *and* the cache is full *and* step times are what you would expect for this model and hardware, then nothing is misconfigured: arrival rate exceeds this replica's service rate. Queueing theory then guarantees the waiting set grows without bound; there is no setting that makes a saturated server keep up. The response is horizontal — more replicas, routed by a signal that reflects the real state — or it is admission control: shed or reject work above a threshold rather than letting queue time inflate first-token latency for everyone. ## Getting the data Use the server's own metrics endpoint before reaching for GPU-level tools. vLLM's `/metrics` publishes `vllm:num_requests_running` and `vllm:num_requests_waiting`, plus gauges for cache utilization and histograms for request timings; TGI and Triton each expose their own Prometheus surface. A GPU utilization percentage is close to useless for this — a decode step keeps the GPU "busy" while doing very little math, so utilization can read high on a badly under-occupied server. The single most useful derived signal is the split between **queue time** and **service time** for a request. If first-token latency is mostly time spent waiting to be admitted, you have a capacity or admission problem. If it is mostly time spent being prefilled, you have a prompt-size or scheduling problem. Those two lead to completely different fixes, and a candidate who separates them is demonstrating the thing the question is testing. ## What overload looks like to the client Worth stating explicitly, because it distinguishes an LLM server from a stateless service. There is usually no fast failure. Requests are accepted, sit in the waiting set, and eventually stream slowly; the user's experience degrades as a long silence before the first token rather than an error. If you want an error instead — and often you should, for interactive traffic — you have to impose it yourself with a queue-depth threshold or a request timeout, so that the system fails visibly rather than turning everyone's latency into minutes.