Latency, Throughput and Batching

## The three clocks Every generation request has three latency numbers, and conflating them is the most common source of useless service-level objectives. **Time-to-first-token (TTFT)** is the wall-clock gap between sending the request and receiving the first output token. It contains two things: how long the request waited in the server's queue, and the forward pass over the entire prompt that must finish before any token can be emitted. Because that work grows with the number of input tokens, TTFT grows with prompt length — a 200-token prompt and a 100,000-token prompt do not have the same first-token wait on the same hardware. **Inter-token latency (ITL)**, often called time-per-output-token (TPOT), is the steady-state gap between consecutive output tokens once generation is under way. Its reciprocal is the per-user streaming rate in tokens per second. For a given model, hardware and server load it is roughly constant, and it is largely independent of how long the prompt was. **End-to-end completion time** is what a non-streaming caller experiences: approximately TTFT + ITL x (number of output tokens). Note the third term — output length is a latency variable. A prompt change that makes the model twice as verbose doubles the tail of every response even though nothing about the server changed. ## Why the two front metrics move independently The first token and every token after it are produced by work with different shapes. Producing the first token requires consuming the whole prompt; producing token N+1 requires one more step on top of state the server already holds. That is why the two numbers respond to different levers. Shortening the prompt, or splitting a giant document into a retrieval step, moves TTFT and leaves ITL alone. Choosing a smaller or faster-decoding model moves ITL. Nothing about a shorter prompt makes tokens stream faster once streaming has begun. Load couples them from the other direction. More concurrent traffic means requests wait longer before they start, which inflates TTFT, and means each generation step is shared with more sequences, which inflates ITL. This is why both numbers must be quoted at a stated load, and why a benchmark run against an idle server tells you almost nothing about production. ## Perceived versus measured latency Streaming changes none of these numbers — the model does not generate faster because you show tokens as they arrive — but it changes the experience completely. With streaming, perceived latency collapses to TTFT plus whether tokens outrun reading speed. Comfortable reading is on the order of a handful of tokens per second, so a stream at 30 tokens per second feels instantaneous after the first token appears; the user is never waiting on the model. Without streaming, the same request is a 12-second blank screen, which reads as a hang even though the total generation time is identical. This is the first-token illusion, and it cuts both ways. A regression from 400 ms to 2 s TTFT makes a chat product feel broken while total completion time is unchanged, so a team watching only end-to-end time will see a flat graph and a support queue full of complaints. ## What belongs in the SLO Write them as separate objectives, at tails, per route class: - p95 (and p99) TTFT, because the tail is what users remember and the mean hides queueing spikes entirely. - p95 ITL, or equivalently a minimum tokens-per-second floor per stream. - A completion-time budget only where a caller genuinely blocks on the full answer — an API integration, a synchronous backend step, a non-streamed job. Measure at the client, not inside the server, so network and any intermediary are included. Slice TTFT by prompt-length bucket, because one route with enormous prompts will otherwise drag the whole distribution and look like a general slowdown. And keep the interactive and non-interactive routes on separate objectives: a background summarization path has no meaningful TTFT requirement at all, and holding it to the chat target wastes capacity. ## Common mistakes Reporting a single mean latency number is the classic one — it is the average of a bimodal mixture of short and long prompts, and it moves for reasons nobody can attribute. Assuming faster hardware fixes ITL is another; per-token generation speed is bounded by how fast weights can be read from memory, not by raw arithmetic throughput. And treating output length as free ignores that it multiplies straight into completion time.