In vLLM, what does --gpu-memory-utilization control, and what does raising it buy?

## What the flag is `--gpu-memory-utilization` (engine argument `gpu_memory_utilization`) takes a float between 0 and 1 and defaults to `0.9`. It is documented as the fraction of GPU memory to be used for the model executor, and it is a **per-instance** limit: it describes how much of one GPU a single vLLM instance intends to occupy, not how much of the GPU is currently free. Two details trip people up. First, the fraction is of **total** device memory, so `0.9` on an 80 GB card is roughly 72 GB regardless of what is already resident. Second, the budget covers the **whole instance** — model weights, activations, CUDA graph memory and the KV cache — not the KV cache alone. ## What vLLM does with it at startup The sequence at server start is roughly: 1. Load the model weights onto the GPU (or onto each GPU when sharded). 2. Run a profiling forward pass at the configured maximum batch shape to measure the peak transient memory the model needs — activations, workspace, and the memory CUDA graph capture will hold. 3. Compute `budget - (weights + peak transient)` and convert that leftover into a fixed number of KV-cache blocks. 4. Allocate that block pool up front and never grow it. This is why the number printed in the startup log — the KV-cache size in GiB or in tokens — is the number that actually matters operationally. The block pool is the server's real capacity: it caps how many sequences can be resident at once and how long each of them may become. ## The arithmetic, in the shape an interviewer wants Take a 13B model in bf16: about 26 GB of weights. On a 40 GB A10G-class card at `0.9`, the budget is ~36 GB; subtract weights and a couple of GB of activation/graph headroom and you have roughly 8 GB of KV cache. On an 80 GB card at `0.9` the same model leaves roughly 44 GB — five times the concurrency for the same model and the same flag value. The flag is a lever on the *remainder*, so its effect is highly non-linear in the weights-to-VRAM ratio: the closer the weights come to filling the card, the more dramatic a small change in utilization becomes, and the more likely it is that the honest answer is "this model does not belong on this GPU". ## Why you cannot just set 1.0 Raising the value trades safety margin for blocks. Things that live outside the profiled estimate: fragmentation in the allocator, communication buffers when tensor parallelism is on, the CUDA context and driver overhead, anything else sharing the card (a metrics exporter, another framework, a stray notebook), and transient spikes on unusual request shapes that the profiling pass did not represent. `1.0` reliably fails; values above ~0.95 are for cards you fully own and workloads you have already profiled. When you must claw back memory, the more surgical levers are lowering `--max-model-len`, shrinking bytes-per-token with `--kv-cache-dtype fp8`, serving a quantized checkpoint, or disabling CUDA graph capture with `--enforce-eager` (which costs decode throughput). ## Co-tenancy Because the value is a per-instance budget, running two vLLM instances on one GPU means setting each to about `0.45` — two instances each claiming `0.9` are each planning to occupy 90% of the same physical card. A non-vLLM process on the card is not represented in vLLM's accounting at all, so co-tenancy with anything else means leaving explicit headroom and validating under load, not at idle. ## What it does not do It does not speed up individual tokens: per-token latency is set by weights, memory bandwidth and batch size, not by how many spare blocks exist. It does not prevent out-of-memory failures caused by the weights themselves — if the weights do not fit under the budget, the server fails immediately and no value of the flag saves it. And it is not a soft ceiling that the server grows into on demand; the block pool is claimed at startup, which is exactly why a vLLM pod's memory graph is flat and boring even as load varies.