TTFT, Throughput and SLOs

## The unit-of-work problem Every capacity metric assumes requests are roughly interchangeable. That assumption holds for a stateless REST service where a request is a database lookup and a JSON serialization. It fails badly for an LLM server, where one request may be a 200-token prompt with a 30-token answer and the next may be a 32k-token document summarised in 1,500 tokens. Those two differ by two orders of magnitude in GPU work, yet both count as "one request". A capacity claim of "we do 20 requests/s" therefore carries no information until somebody says what a request was. ## Prefill and decode scale on different inputs An LLM request has two phases with different cost drivers. **Prefill** processes the whole prompt in one pass. Its cost grows with the number of **input** tokens (super-linearly once attention over long contexts dominates). Prefill is compute-heavy: it runs large matrix multiplications over many tokens at once and keeps the GPU's math units busy. **Decode** produces output one token at a time, each step reading the full model weights and the accumulated key/value cache. Its cost grows with the number of **output** tokens, and per token it is memory-bandwidth-heavy rather than compute-heavy. So a request's cost is roughly `a x input_tokens + b x output_tokens`, with very different constants. A workload that is all long prompts and one-word answers stresses prefill; a workload that is short prompts and long generations stresses decode. Both can be described as "10 requests/s", and they will saturate the same GPU at wildly different rates. ## What to report instead Three numbers, together: 1. **Output token throughput (tokens/s)** — the headline capacity number. It is what users perceive as "the server is producing text", it is what your decode-phase capacity is spent on, and it is the denominator for cost per million tokens. 2. **Total token throughput (input + output tokens/s)** — includes prefill work. Useful when prompts are long, because output-only throughput understates how loaded the GPU is on a RAG or long-document workload. 3. **Request throughput (requests/s)** — still worth reporting, but only as a derived number, and only next to the length distribution. And one qualifier that is not optional: **the input/output length distribution**. "1,024 in / 256 out, 500 requests, concurrency 32" turns a bare number into a reproducible claim. ## Reading a real benchmark summary vLLM's built-in load generator (`vllm bench serve` in vLLM 0.27, which replaced the older `benchmarks/benchmark_serving.py` script) prints exactly this trio — request throughput, output token throughput, total token throughput — above the TTFT / TPOT / ITL percentile tables. That layout is deliberate: the throughput block tells you how much work the server did, the latency block tells you what each user felt, and neither is interpretable without the other. GenAI-Perf, the load generator now living in the `triton-inference-server/perf_analyzer` repository, reports the same split (output token throughput plus per-request latency statistics). ## Where requests/s is still the right unit Requests/s is fine — and often better — in three places: - **Admission and rate limiting.** Quotas and queue depth are naturally per request, and a client's fair share is easier to express that way (usually alongside a token quota). - **Autoscaling signals**, where queue depth in requests is a direct measure of backlog. - **Fixed-shape workloads**: classification, embedding, or reranking traffic where every request really is the same size. There, requests/s is a legitimate capacity metric because the length distribution is a constant. For open-ended chat or summarisation, none of those conditions hold. ## The failure this causes in practice The common production incident is a capacity plan built from a benchmark whose outputs were short. The team measures 25 requests/s on 128-in/128-out synthetic traffic, provisions for 20 requests/s of real traffic, and then discovers that real answers average 600 tokens. Decode work is roughly five times what was budgeted, the scheduler starts queueing, and time to first token degrades for everyone — because the capacity number was denominated in a unit that did not track the work. The fix is procedural, not clever: measure with the length distribution your production traffic actually has (sample it from real logs), publish capacity in output tokens/s, and treat any requests/s figure without an attached length profile as unverified.