In vLLM, what does --max-model-len cap, and why can it block startup?

## What the flag caps `--max-model-len` (engine argument `max_model_len`) is the maximum sequence length vLLM will handle for one request, counted in tokens and covering **prompt plus completion together**. It is not a batch-level knob and not a throughput knob: `--max-num-batched-tokens` bounds tokens per scheduler iteration, `--max-num-seqs` bounds concurrent sequences, and this one bounds a single conversation's total length. If you do not pass it, vLLM reads the model's own configuration and uses the context length declared there. That default is the source of most surprises: a checkpoint advertising 128k context makes vLLM plan for 128k-token sequences on a card that may only have a few gigabytes free for KV blocks. ## Why it can prevent the server from starting vLLM allocates its KV-cache block pool once, at startup, from whatever the memory budget leaves after weights and profiled activation peaks. It then performs a sanity check: can that pool hold at least one sequence of `max_model_len` tokens? If not, it aborts with an error stating that the model's maximum sequence length is larger than the number of tokens that can be stored in the KV cache, and suggesting you increase `gpu_memory_utilization` or decrease `max_model_len`. That refusal is deliberate. The server advertises a context limit to clients; accepting a request at that limit and then discovering mid-decode that no blocks exist would be an unrecoverable failure per request rather than one clear failure at boot. Failing closed at startup is the right behaviour and is worth saying out loud in an interview. ## The three levers, and how to choose **Raise `--gpu-memory-utilization`.** Cheapest if you have headroom, but it is a small multiplier — moving 0.90 to 0.95 on an 80 GB card is roughly 4 GB, which at typical KV sizes is thousands, not hundreds of thousands, of tokens. **Lower `--max-model-len`.** Usually the correct answer, because the advertised limit is almost always inherited from the checkpoint rather than chosen. If your product truncates inputs at 8k anyway, serving with `--max-model-len 8192` frees the planning constraint and lets the same pool hold many more concurrent sequences. The tradeoff is a hard contract change: requests longer than the limit are rejected by the API with an error about exceeding the maximum context length, not silently truncated, so the client must enforce the same ceiling. **Shrink bytes per token.** `--kv-cache-dtype` accepts `auto` (match the model dtype), `fp8`, `fp8_e4m3` and `fp8_e5m2`. Moving from 16-bit to 8-bit KV roughly halves the per-token cost, so the same pool holds about twice the context — at the price of needing a supported attention backend and a quality check, since the cache is now lossy. A fourth, structural lever is serving a quantized checkpoint or adding GPUs: smaller weights leave a bigger remainder for blocks, and tensor parallelism splits both weights and cache across devices. ## Sizing it honestly The useful discipline is to set `--max-model-len` from your measured request distribution rather than from the checkpoint. Look at the p99 of prompt tokens plus your `max_tokens` cap, add margin, and serve that. Every token of advertised context you do not need is capacity you have promised to reserve for a request that never arrives. Teams that leave the default in place routinely find that a server which "cannot handle more than four concurrent users" is simply planning for sequences a hundred times longer than any client sends. ## Going the other way Setting `--max-model-len` **above** the checkpoint's native length is a different operation entirely. It requires a RoPE-scaling configuration so the positional encoding covers the longer range, and quality degrades outside the range the model was trained or fine-tuned on. vLLM will generally reject a value above the model's declared maximum unless that scaling is configured. Extending context is a modelling decision, not a serving flag; the serving-side flag only ever lets you serve less than the model claims, safely. ## What it interacts with Because the block pool is fixed, `--max-model-len` and `--max-num-seqs` compete for the same tokens: a large context ceiling with a large concurrency cap describes a worst case the pool cannot satisfy, and the scheduler resolves the conflict at runtime by queueing or preempting rather than by failing. Long-context and high-concurrency are two ways to spend one budget, and the flags are how you declare which you care about.