TensorRT-LLM Engine Build

## Two ways to run a model on a GPU Most inference servers execute the model graph at runtime: Python code walks the layers, calling into kernels chosen while the request is in flight. TensorRT-LLM's classic flow does the opposite. Before any request arrives, it *compiles* the network into a TensorRT **engine** — a serialized binary that already knows which kernels to call, in what order, with which fusions, for a bounded range of input shapes. Running the model is then largely a matter of feeding tensors into a prebuilt execution plan. ## What the build step consumes and emits `trtllm-build` does not read a Hugging Face repository directly. It reads a **TensorRT-LLM checkpoint** — a directory holding a `config.json` describing architecture, dtype and parallel degrees, plus one weight file per rank. It emits an **engine directory**: one serialized engine per rank plus the build configuration that produced it. The important flags are the ones that fix capacity: - `--max_batch_size` — the largest number of concurrent sequences the engine can execute. - `--max_input_len` and `--max_seq_len` — the longest prompt and the longest prompt-plus-output the engine admits. - `--max_num_tokens` — the token budget for a single forward pass, which bounds how much prefill work can be packed into one iteration. - `--gemm_plugin` — which plugin implementation backs the matrix multiplies. - `--kv_cache_type` — in-flight batching requires a paged KV cache rather than one contiguous buffer per sequence. Everything in that list is a compile-time decision. None of it is negotiable per request. ## Why the engine is pinned to one GPU During the build, TensorRT auto-tunes: for each layer it benchmarks candidate kernel implementations ("tactics") on the GPU present at build time and records the winners in the plan. Those winners depend on the compute capability, the SM count, the cache sizes and the tensor-core instructions available. A serialized plan therefore carries an architecture stamp, and loading it on a different architecture fails rather than silently falling back. This is why teams building for a mixed fleet end up with an artifact matrix: `model x precision x tensor-parallel degree x GPU SKU`. Two more pins are easy to overlook. The engine is tied to the **TensorRT-LLM / TensorRT version** that built it, so bumping the serving image is a rebuild, not just a redeploy. And it is tied to the **precision** it was compiled for: a BF16 engine and an FP8 engine of the same model are two different binaries, produced from two different checkpoints. ## What this buys The compiler can do things a runtime dispatcher cannot afford to do per step: fuse attention and normalization patterns, pick a specialized GEMM per shape bucket, remove framework overhead from the decode loop, and lay out memory once. On NVIDIA hardware this typically shows up as higher tokens/second at a given latency and lower per-iteration CPU overhead, which matters most in the decode phase where the per-step work is small and framework overhead is proportionally large. ## What this costs operationally The build itself is not instant — expect minutes to tens of minutes for a large model, longer with many optimization profiles — so it belongs in CI, not in a pod's startup path. A container that builds on boot turns an already slow cold start into a much slower one. The normal pattern is: build in a job, publish the engine directory as a versioned artifact, and mount or bake it into the serving image. You also inherit a validation obligation. Because the engine is a compiled artifact rather than the checkpoint you evaluated, "the same model" at a new precision or a new library version is a new binary whose output quality you have not yet measured. Rebuild pipelines that skip an accuracy check are how a quiet regression reaches production. ## Where this sits in TensorRT-LLM 1.x TensorRT-LLM 1.x ships two flows. The ahead-of-time flow described here compiles an engine with `trtllm-build` and serves it from the C++ runtime, including through Triton's TensorRT-LLM backend. Alongside it, the project ships a PyTorch backend that runs a checkpoint without an ahead-of-time engine build, trading some peak throughput for a much shorter path from checkpoint to running server. Knowing that both exist is part of answering "why compile at all" — the compiled engine is the right answer when the model is stable, the fleet is uniform, and you are paying for GPU-hours at volume. ## The interview shape When this comes up, interviewers are checking whether you understand that a compiled engine is a *deployment artifact with a lifecycle*, not a portable file. The strong answer names what gets frozen, names the three pins (architecture, version, precision), and follows through to the pipeline consequence: a build matrix, an artifact store, and a validation gate.