In TGI, what do --max-input-tokens and --max-total-tokens each cap?

## Two limits, one request Every request TGI serves occupies a slot sized in tokens. Two launcher flags bound that slot: - **`--max-input-tokens`** — the maximum length of the tokenized prompt. - **`--max-total-tokens`** — the maximum of prompt plus generated tokens combined. The relationship is arithmetic, not independent: for a given request, the most tokens it may generate is `max_total_tokens` minus its actual input length. A 4,000-token prompt against `--max-total-tokens 4096` leaves 96 tokens of output, regardless of what `max_new_tokens` the caller asked for. The launcher requires `max_input_tokens` to be strictly less than `max_total_tokens` — a configuration where they are equal would admit prompts with zero room to answer, so it is refused at startup rather than at 3am. Both are also bounded above by the model itself. You cannot configure a slot longer than the checkpoint's positional limit; asking for more is a startup failure, not a silent clamp. ## Rejection, not truncation The router — the component in front of the shards — tokenizes and validates each request before it is queued. An over-length request comes back as **422 Unprocessable Entity** with a JSON body carrying an `error` message and `error_type` of `validation`. It never occupies GPU time. This is worth defending in an interview, because the naive expectation is truncation. Truncating a prompt is the worse behaviour: the model answers confidently from a mutilated context and the caller has no signal that anything was dropped. An explicit 422 pushes the decision back to the client, which is the only layer that knows whether to summarize, drop old turns, or split the work. Application code in front of TGI should therefore count tokens with the same tokenizer and manage its own context budget, treating 422 as a bug in its budgeting rather than a condition to retry. Note the distinction from the overload path: a request that is *valid* but arrives when the server has no capacity is a queueing/backpressure concern and surfaces differently (TGI signals an overloaded server with 429). Validation is about the shape of the request; 429 is about the state of the server. ## What happens when you leave them unset TGI 3.x does not force you to specify these. When unset, the launcher derives them: it reads the model's configuration for the architectural ceiling and, during the warmup phase, measures how much memory it can actually claim on the device, then settles on values that fit. This is convenient and it is also a trap for reproducibility — the same image and the same model id can settle on different limits on an 80 GB card than on a 24 GB one, or when another process is already holding memory on the same GPU. So: read `/info` after every deploy. It reports the limits the server actually chose. "It works on the A100 and rejects requests on the L4" is nearly always this, and it is invisible unless you look. ## Setting them deliberately The reason to pin these is that they are a capacity dial. Every admitted request reserves cache space proportional to its slot length, so raising `--max-total-tokens` makes each request more expensive and reduces how many can run concurrently on the same card. Lowering it packs more conversations in. The right value comes from your traffic, not from the model's maximum: - Measure the real distribution of prompt lengths — the p95 and the tail, not the mean. - Set `--max-input-tokens` just above the p99 you intend to support, so genuinely oversized requests fail fast and loudly. - Set `--max-total-tokens` to that plus the longest answer the product actually needs. A team that sets `--max-total-tokens` to the model's full advertised context "because we paid for it" and then serves 500-token chats has quietly bought a much smaller concurrency ceiling for no user benefit. Conversely, a team that sets it too tight discovers it through a stream of 422s from exactly the users with the longest, most valuable documents. ## Making the limits visible to callers Because the limits are server-side and per-deployment, they belong in whatever contract your clients read. Surface the configured input and total limits in your API docs or a capabilities endpoint of your own, and give the 422 a client-facing message that names the number. A caller that has to discover the limit by bisection is a caller that will hard-code the wrong one. ## Changing them These are launcher flags, so changing them means restarting the server: there is no runtime knob. On a rolling deploy that is a full model reload per replica, with the cold-start cost that implies. Plan limit changes as deployments, and validate the new value on the same GPU SKU you run in production — a limit that fits on one card may not fit on another.