Output tokens/s went flat but TTFT p95 keeps climbing — what is happening?

## What the curve looks like Sweep concurrency (or arrival rate) against a served model and plot two lines: output tokens per second, and TTFT p95. You get a shape that recurs on every engine. - **Under-loaded region.** Few requests in flight. The GPU spends decode steps reading the entire model weights to produce a handful of tokens, so per-token cost is dominated by weight loading and hardware is badly under-used. Throughput climbs nearly linearly with concurrency; latency barely moves. - **The knee.** The batch is large enough that the fixed cost of a decode step is amortised across many sequences. Throughput growth flattens. Latency starts to rise noticeably. - **Saturated region.** Throughput is flat — the ceiling is set by memory bandwidth per decode step and by how many sequences fit in KV cache. Additional arrivals queue. TTFT rises roughly linearly with the backlog. - **Collapse.** If KV-cache capacity binds hard, the scheduler starts preempting running sequences (recomputing or swapping their cache), and throughput can *fall* while latency continues to climb. ## Why the queue lands on TTFT specifically TTFT is the interval between the request arriving and its first token appearing. It decomposes into: network, admission/queue wait, and the prefill forward pass. Once the server is saturated, queue wait dominates and it grows with backlog. Every second a request spends waiting for a batch slot is a second added to TTFT and to nothing else. Inter-token latency behaves differently. A request that has been admitted is being decoded alongside its batch mates; it degrades as the batch grows, but it degrades smoothly. This asymmetry is the diagnostic signal: **TTFT climbing steeply while inter-token latency rises mildly and throughput is flat means a queueing problem, not a decode-speed problem.** When inter-token latency *also* jumps and throughput drops, you are into preemption or memory pressure territory, which is a different and worse failure. ## Confirming the diagnosis on a live server - Watch queue depth / number of waiting requests. Growing monotonically under steady arrivals means arrivals exceed service rate — the queue will never drain on its own. - Watch running-batch size. If it is pinned at its cap while requests wait, you are concurrency-limited by configuration. If it is *below* the cap while requests still wait, you are memory-limited: there are no free KV-cache blocks to admit anyone. - Watch preemption counters. Non-zero preemption under steady load means the engine is thrashing. - Check GPU utilisation. Near 100% with flat throughput confirms you have found the hardware ceiling rather than a software stall. Those two sub-cases matter because the remedies differ: a configuration limit can sometimes be raised; a memory limit means you need more KV cache, a smaller context budget, a smaller model footprint, or another GPU. ## Little's law, applied In steady state, `average in-flight requests = arrival rate x average time in system`. If throughput is capped at T requests/s and you offer more than T, the excess accumulates: time in system rises without bound and there is no equilibrium. This is the formal reason "just raise concurrency" is not a fix. Concurrency is not a capacity knob past the knee; it is a decision about *where the waiting happens* — in your queue, or in the client's connection pool. ## Choosing the operating point Do not run at the knee. The knee is peak throughput, which means peak latency-per-token and zero headroom: a traffic burst puts you straight into the collapse region. Instead: 1. Define the latency SLO first (TTFT p95, per-token p95). 2. From the sweep, find the highest load at which those percentiles still hold — that is the SLO-bounded capacity, usually meaningfully below peak throughput. 3. Provision replicas so expected peak traffic sits below that point, keeping headroom for bursts and for a replica being unavailable during a rollout. 4. Autoscale on a signal that leads the cliff — queue depth or TTFT — rather than on GPU utilisation, which is high across both the healthy and the saturated regions and therefore cannot distinguish them. ## The wrong answers "Increase the batch size" — past the knee the batch is already as large as memory allows, and forcing it larger buys preemption. "Scale up the GPU" — helps only if you were memory-capacity-bound, and it is slower and often costlier than another replica. "Raise the client timeout" — hides the symptom while the backlog keeps growing. The correct response to demand above capacity is more capacity, or shedding load with an explicit rejection so clients fail fast instead of waiting behind an unbounded queue.