Which tensorrtllm_backend settings does Triton need to serve a TensorRT-LLM engine?

## Why there are five models, not one A TensorRT-LLM engine takes token ids in and produces token ids out. Nothing in it turns text into ids. So the standard deployment is a small pipeline of Triton models: - **`preprocessing`** — a Python-backend model that tokenizes the request text. - **`tensorrt_llm`** — the TensorRT-LLM backend model that actually executes the engine. - **`postprocessing`** — detokenizes generated ids back into text. - **`ensemble`** — wires the three together so a client calls one endpoint. - **`tensorrt_llm_bls`** — a Business Logic Scripting alternative to the ensemble, used when the glue needs real control flow. You pick either the ensemble or the BLS front door; both sit over the same `tensorrt_llm` model. The templates ship as `config.pbtxt` files with `${placeholders}`, and a `fill_template.py` helper substitutes values, which is why deployment guides show a string of `key:value` pairs rather than hand-edited configs. ## The parameters that matter on the tensorrt_llm model **`engine_dir`** — the path to the engine directory. This is the parameter people get wrong from memory: the 0.x-era spelling was different, and current documentation uses `engine_dir`. For encoder-decoder models there is a companion `encoder_engine_dir`. **`batching_strategy`** — `inflight_fused_batching` turns on in-flight batching, where new requests join the running batch at iteration boundaries instead of waiting for the current batch to drain. Setting it to `V1` selects the older static behaviour. Note this is *not* Triton's generic `dynamic_batching` block: the TensorRT-LLM backend schedules internally, at iteration granularity, and the generic batcher is the wrong tool here. **`triton_max_batch_size`** — the batch ceiling Triton enforces for this model. It must not exceed the `--max_batch_size` compiled into the engine; you can go lower, never higher. **`decoupled_mode`** — set `true` when the model should emit many responses for one request, which is what token-by-token streaming requires. Leave it false and clients get a single response at the end. A deployment that "doesn't stream" is usually this flag. **`max_num_tokens`** — the runtime token budget per iteration, again bounded by what the engine was built with. **`max_queue_size` and `max_queue_delay_microseconds`** — how many requests may wait and how long the backend will hold a request before scheduling it. A bounded queue is what lets an overloaded server reject fast instead of accumulating an unbounded backlog whose latency nobody can meet. **`batch_scheduler_policy`** — `max_utilization` packs aggressively and accepts that a request may be paused and resumed when cache pressure spikes; `guaranteed_no_evict` only admits work it can carry to completion, trading some throughput for predictability. **`kv_cache_free_gpu_mem_fraction`** — the share of free VRAM after weights are loaded that becomes KV cache. Raise it for more concurrency, lower it if something else on the GPU needs room. **`gpu_device_ids`** — which GPUs this model's ranks occupy, which matters when several models share a node. **`instance_count`** — how many instances of the backend model run. For a multi-GPU engine, the mapping between ranks and instances is not the naive one; treat instance count for a TensorRT-LLM model as something to verify against the template guidance rather than something to increase reflexively. **`logits_datatype`** — the datatype of returned logits, when a client asks for them. It is not a knob for compute precision, which is fixed in the engine. **`tokenizer_dir`** — set on the preprocessing and postprocessing models, pointing at the original Hugging Face directory. Mismatch it with the checkpoint the engine was built from and you get garbled text out of a perfectly healthy engine. **`return_perf_metrics`** — turns on per-request performance metrics, useful when you want the backend's own view of queueing versus execution rather than only client-side timings. ## What the engine does not tell Triton None of the build ceilings are auto-discovered into your config. If the engine was built with `--max_batch_size 64` and you fill in `triton_max_batch_size: 128`, you have created a mismatch that surfaces as a load or runtime error rather than as graceful clamping. Keeping the build command and the config template beside each other in the same repository — generated from one source of truth — is the practice that prevents this class of bug. ## Diagnosing a deployment Three symptoms cover most first deployments. The model fails to load: check `engine_dir`, the world size versus `gpu_device_ids`, and that the engine's library version matches the container. Responses arrive all at once: `decoupled_mode`. Text comes back mangled: `tokenizer_dir` on pre/post. After that, the interesting questions become tuning ones — cache fraction against concurrency, scheduler policy against tail latency, queue bounds against overload behaviour. ## The interview shape Interviewers are checking whether you have deployed this rather than read about it. Naming `engine_dir` and `inflight_fused_batching` correctly, knowing that the tokenizer lives outside the engine, and knowing that `decoupled_mode` is what makes streaming work are the three signals that separate hands-on from hearsay.