Which vLLM /metrics series show whether the server is queueing or just slow?

## Where the numbers come from A vLLM server started with `vllm serve` (or the `vllm/vllm-openai` image) publishes Prometheus-format metrics at `/metrics` on the same port as the API. No sidecar or exporter is involved; you point a scrape at the pod. Collection is on by default and is switched off by `--disable-log-stats`, which is a real trap: the endpoint still responds, but the `vllm:` series are simply absent and every dashboard built on them goes blank. ## The three layers worth wiring up **Scheduler state.** `vllm:num_requests_running` is how many sequences the engine is decoding this iteration; `vllm:num_requests_waiting` is how many are admitted but not yet running. `vllm:num_requests_waiting_by_reason` breaks the wait down further. These two gauges are the single most useful pair on the dashboard, because they distinguish "the server has more work than it can start" from "the work it started is slow". **Memory pressure.** `vllm:kv_cache_usage_perc` reports how much of the KV block pool is occupied, on a 0-to-1 scale. `vllm:num_preemptions` counts sequences the scheduler had to evict to free blocks — a counter that should be flat and whose slope is a direct measure of over-subscription. When usage sits near 1.0 and preemptions are climbing, the fix is memory-shaped: fewer concurrent sequences, a shorter context ceiling, an fp8 KV cache, or more GPUs. There are also `vllm:kv_offload_cpu_cache_usage_perc` for CPU-offloaded blocks and `vllm:external_prefix_cache_hits` / `vllm:external_prefix_cache_queries` for external prefix-cache tiers. **Latency and work.** `vllm:time_to_first_token_seconds` covers everything before the first token — queue wait plus prefill. `vllm:inter_token_latency_seconds` and `vllm:request_time_per_output_token_seconds` cover the streaming phase. `vllm:iteration_tokens_total` shows how much work each scheduler step actually did, which is the honest measure of batching efficiency. `vllm:prompt_tokens` and `vllm:generation_tokens` are the token counters that feed cost accounting, and `vllm:prompt_tokens_cached` shows how many prompt tokens were served from the prefix cache rather than recomputed. `vllm:request_success` counts completed requests, labelled by finish reason. ## Reading them together The diagnostic value is in the combination, not any single series: - Waiting high, cache usage high, preemptions rising: **capacity-bound on memory.** More replicas, less context, or a cheaper cache dtype. - Waiting high, cache usage moderate, iteration tokens flat: **compute-bound.** The GPU is doing all it can per step; a faster card or a smaller model is the lever. - Waiting near zero, TTFT high: **not saturation.** Look at prompt length distribution, prefix-cache hit rate, and whether one enormous request is dominating each prefill. - Everything calm inside the server, users complaining: the problem is outside vLLM — the ingress, the routing layer, or a client that is not consuming the stream. ## Naming is a versioned claim The metric namespace has changed over vLLM's life, and copying a dashboard from a two-year-old blog post produces panels that quietly display nothing. The specific one to know: the KV-cache occupancy gauge was `vllm:gpu_cache_usage_perc` under the V0 engine and is `vllm:kv_cache_usage_perc` in current versions — the old name does not exist any more. Because a Prometheus query for a missing series returns empty rather than erroring, this failure mode is invisible until an incident, when the panel everyone trusts is blank. The habit that prevents it is to scrape `/metrics` once by hand after any version bump and diff the series names against what your dashboards and alert rules reference. ## What these metrics are not They are per-replica engine internals. They do not tell you about GPU temperature, ECC errors or device utilization — that is DCGM's job — and they say nothing about what is happening between replicas. GPU device memory in particular is useless as a load signal for vLLM, because the engine claims its budget at startup and holds it flat regardless of traffic; the occupancy of the block pool is the number that actually moves.