For a streamed LLM endpoint, which latency numbers do you track besides total time?

## Why one number is not enough For a buffered request, total latency is the whole story: the user waits, then gets the answer. Streaming splits that single wait into phases with different causes, different owners and different fixes, and a single average hides all of it. Two endpoints can both average nine seconds, one showing text after 500 ms and streaming smoothly, the other sitting blank for eight seconds and then dumping everything. Only a decomposition tells them apart. ## The four numbers **Think time.** With a reasoning model, the run begins with an internal reasoning phase before any answer token exists. This is real elapsed time during which the response is open but nothing readable is coming. It can be seconds to tens of seconds, and it scales with how much reasoning effort the request asks for. As of mid-2026 this is a large, separately-budgeted component rather than a rounding error, so it deserves its own line. **Time to first token (TTFT).** The classic serving metric: how long until the first token of output is produced. Measured at the server or the provider boundary. **Time to first *visible* token (TTFVT).** How long until something appears in front of the user. This is the metric that governs whether the feature feels alive, and it is not the same as TTFT for two reasons. First, with extended thinking the first produced token may be a reasoning token the user never sees, so the answer's first visible token comes much later — unless you deliberately surface a thinking summary, which resets TTFVT to something small. Second, everything between your server and the user's screen can add delay: a buffering proxy, a compression layer, a client that waits for a complete parse before rendering. TTFT is a server number; TTFVT is a user number, and only client-side instrumentation can measure it. **Inter-token latency (ITL).** Once the stream is running, the gap between successive chunks — usually reported as its inverse, output tokens per second. This decides smoothness. The key discipline is to look at the distribution, not the mean: a stream averaging 40 tokens/s that contains one 2.5-second stall reads as broken, and the mean will not show it. Track the p99 gap, or simply count stalls over a threshold. **Time to last token / total.** Still worth tracking, because it drives connection lifetime, capacity, and the cost side of the ledger. But it is the wrong headline for a streamed UI, since the user has usually been reading for most of it. ## Where to measure Instrument on the client wherever the metric is about perception. A server-side histogram cannot see a proxy that buffered your flushes, a slow network, or a renderer that batches DOM updates. The usual shape is: the client records timestamps for request start, first visible chunk, each chunk arrival, and stream end, then reports a compact summary; the server records its own TTFT and total for capacity work. When the two disagree, the gap between them is your infrastructure problem, and that gap is the single most useful diagnostic this decomposition gives you. ## Percentiles, not averages Every number here should be reported at p50 and p99. Streaming latency distributions are heavy-tailed — a queue at the provider, a cold path, or an unusually long reasoning phase produces occasional very slow runs, and the mean absorbs them silently. A p99 TTFVT of 20 seconds against a p50 of 900 ms says something a nine-second average never will. ## Turning the numbers into targets The decomposition maps to different levers. A bad TTFVT with a good TTFT is a delivery problem: buffering somewhere in the path, or a client that renders too late. A bad TTFVT with a matching bad TTFT on a reasoning model is a think-time problem: the fix is either to surface progress during the thinking phase or to ask for less reasoning. A good TTFVT with bad inter-token latency is a serving or network problem: contention, throttling, or chunk framing overhead. Bad total time with everything else healthy usually means the answer is simply long, which is a prompt or output-budget question, not a latency bug. One more calibration point: past a certain output rate, faster stops helping. People read a few words a second, so once the model comfortably outruns reading speed, additional tokens per second buys nothing perceptible, while first-visible-token improvements keep paying all the way down.