Which TGI /metrics series separate queueing time from GPU inference time?

## Why an LLM server needs its own signals On a stateless web service, a p95 latency chart plus CPU utilisation gets you a long way. Neither works here. GPU utilisation on a busy inference server sits near 100% almost all the time - a single decode step keeps the device nominally busy - so it cannot tell a comfortable server from a drowning one. And end-to-end latency lumps together two completely different problems with completely different fixes. TGI's Prometheus endpoint at `/metrics` exists to separate them. ## The series to know **Per-request timers** - `tgi_request_queue_duration` - how long the request sat before the router admitted it into a batch. - `tgi_request_inference_duration` - how long the model actually spent on it. - `tgi_request_validation_duration` - time in request validation, before any scheduling. - `tgi_request_duration` - the end-to-end figure the client experiences. - `tgi_request_mean_time_per_token_duration` - the inter-token pace, which is what a streaming user perceives as 'speed' after the first token arrives. **Batch and queue gauges** - `tgi_queue_size` - requests waiting right now. - `tgi_batch_current_size` - requests decoding right now. **Counters and size distributions** such as `tgi_request_count` and `tgi_request_generated_tokens` round out the picture, letting you turn latency into per-token economics. ## The triage decision tree Start from bad end-to-end latency and split it: 1. **Queue duration dominates.** Requests are waiting, not computing. Either the server is genuinely at capacity - add replicas - or the admission policy is too lazy and queued work is not being folded into the running batch quickly enough. Cross-check `tgi_queue_size` against `tgi_batch_current_size`: a long queue beside a small batch points at admission or at a memory budget that will not let the batch grow. 2. **Inference duration dominates.** The model step itself is slow. Look at prompt lengths (prefill cost scales with them), at precision and kernel path, at whether the batch has grown so large that per-request decode has slowed, and at whether output lengths are longer than you assumed. 3. **Both look fine but users complain.** Check `tgi_request_mean_time_per_token_duration`. Frequent admission pauses show up here as inter-token jitter even when averages look healthy. ## Overload behaviour TGI's router enforces `--max-concurrent-requests`. Past that limit it does not silently queue forever - it rejects with HTTP **429** and an error type of `overloaded`. This matters for two reasons. First, it is the correct load-shedding contract: a fast rejection lets a caller retry elsewhere or degrade, where an unbounded queue converts overload into timeouts everywhere. Second, clients must actually handle it - retry with backoff and jitter, and treat a sustained 429 rate as a scaling signal rather than a bug. An overload rejection is distinct from a validation rejection: a prompt that exceeds the configured input length fails validation with a 4xx that no amount of retrying will fix, and it never reaches the queue at all. ## Choosing the autoscaling signal When you wire these into a scaling policy, the useful signals are queue depth and time-to-first-token, because both track saturation in the direction users feel. GPU utilisation does not. Whatever mechanism you scale with, the constraint that dominates an LLM deployment is that adding a replica means loading tens of gigabytes of weights, so the reaction is measured in minutes - scale on a leading indicator, not a lagging one. ## Streaming complicates error handling One practical wrinkle worth carrying into an interview: once a streamed response has started, the HTTP status is already 200. A failure part-way through therefore cannot come back as a status code - it arrives as an event in the stream carrying an error payload. Clients that only check the response status will silently treat a truncated generation as a complete one. Metrics show these as failures; naive clients do not. ## What interviewers listen for Name real series rather than saying 'we'd monitor latency'. Then show the split - queue versus inference - and say what each direction implies. Finishing with the 429 contract and the observation that GPU utilisation is a useless saturation signal here is what separates someone who has run one of these from someone who has read the README.