Your TensorRT-LLM engine rejects a 32k-token request; max_seq_len was 8192. Now what?

## Why the limit is hard A TensorRT engine contains optimization profiles: declared min/opt/max shape ranges that the builder used to pick kernels and to plan memory. A request outside the compiled range has no valid execution path, so it is rejected rather than handled slowly. This is the sharpest behavioural difference between a compiled engine and a Python-level server, where the same limit is usually a configuration value the process reads at startup. The four ceilings that bite: - **`--max_input_len`** — longest prompt admitted. - **`--max_seq_len`** — longest prompt-plus-generation. - **`--max_batch_size`** — most concurrent sequences. - **`--max_num_tokens`** — most tokens processed in one forward pass, which bounds how much prefill can be packed into an iteration. ## Diagnosing which one you hit The error text points at a shape, but the mapping is worth knowing. A long prompt that fails immediately is `max_input_len` or `max_num_tokens`. A request that starts generating and then stops short is bumping `max_seq_len` — the prompt fit, the sum did not. Concurrency that plateaus below what your GPU memory should allow is `max_batch_size`, or the serving-layer value you configured under it. Rejections that arrive under load but not in isolation are usually queue bounds, not engine ceilings, and no rebuild will help those. ## Why not just build everything huge The temptation is to build with the largest numbers the model supports and never think again. Three costs argue against it. **KV-cache math.** The bytes a sequence can consume scale with its maximum length. The runtime reserves and plans against the compiled maximum, so an engine built for very long contexts leaves less room for concurrency at the same VRAM budget. If your median request is 2k tokens, building for 128k means paying for a tail you rarely serve. **Activation memory.** A larger per-pass token budget means larger intermediate tensors, which the builder must plan workspace for. That workspace is memory not available as cache. **Tactic selection.** The builder tunes against the shapes you declared. A profile spanning an enormous range gives it a harder optimization problem than a narrow one, and the kernels it settles on may be less good for the shapes you actually serve. `--multiple_profiles` exists for exactly this tension: build several profiles so different shape regimes get separately tuned kernels, at the cost of a longer build and a bigger engine. ## Sizing from data, not from the model card The defensible method is to take the request-length distribution from your logs — prompt tokens and output tokens separately — and build for a high percentile plus margin, not for the model's theoretical maximum context. Then decide explicitly what happens to the tail: reject it, route it to a second deployment built with longer limits, or truncate it upstream. "Route long requests to a long-context deployment" is a perfectly good production answer, and it is often cheaper than making every replica carry long-context overhead. ## Make the rebuild boring The reason this question tests seniority is that the correct answer — "rebuild" — is only acceptable if rebuilding is routine. That means the build lives in CI, parameterised by the ceilings; the artifact is versioned with those ceilings in its key; accuracy validation runs against the new engine; and deploying it is a normal rollout with the old artifact still available. Teams that build by hand on a login node experience a limit change as an outage; teams with a pipeline experience it as a ticket. ## What to say about the request that failed There is also an immediate-response half to this answer. While the rebuild is in flight, you decide what the client sees: a clear error naming the limit, truncation at a documented boundary, or a redirect to a deployment that can serve it. Silently truncating a user's 32k prompt to 8k and answering from the first quarter of it is the worst of the three, because it produces a confidently wrong answer instead of an honest failure. ## The interview shape Weak answers reach for a runtime knob that does not exist. Adequate answers say "rebuild". Strong answers say rebuild, explain why the ceiling is structural, explain why building for the maximum is not free, and describe how the ceilings were chosen from traffic data in the first place — plus what the client sees in the meantime.