In a Triton config.pbtxt, what do max_batch_size and dims declare together?

## Two fields, one shape contract A Triton model configuration describes each tensor with a `name`, a `data_type` and `dims`. What makes `dims` subtle is that it does not stand alone: it is interpreted relative to `max_batch_size`, and the two together are the shape contract between Triton and the backend. Getting this pair wrong is the single most common reason a model loads cleanly and then fails on the first request. ## max_batch_size > 0: the implicit leading dimension Setting `max_batch_size: N` asserts two things. First, the model accepts a **variable-size first dimension** — this is what "the model supports batching" means in Triton's vocabulary. Second, Triton will never assemble a batch larger than `N` for that model. Because the batch dimension is implied, you must **omit it from `dims`**. A classifier that takes `[batch, 3, 224, 224]` is configured as: ``` max_batch_size: 8 input [ { name: "data_0" data_type: TYPE_FP32 dims: [ 3, 224, 224 ] } ] ``` A client sending one request supplies a tensor of shape `[1, 3, 224, 224]` — the batch dimension is present on the wire, it is only absent from the configuration. If Triton combines four such requests, the backend sees `[4, 3, 224, 224]`. The same rule applies to outputs: `dims: [1000]` with `max_batch_size: 8` means the model returns `[batch, 1000]`, and Triton splits that back into per-request responses. ## max_batch_size: 0 — batching off `max_batch_size: 0` says the model does **not** support a variable leading dimension. Now `dims` is the literal, complete shape the model expects, batch dimension included if the model has a fixed one: ``` max_batch_size: 0 input [ { name: "input" data_type: TYPE_FP32 dims: [ 1, 3, 224, 224 ] } ] ``` This is the correct configuration for a graph exported with a hard-coded batch of 1, and for models whose first dimension means something other than batch — sequence length, for instance. The trade-off is real: with `max_batch_size: 0` Triton executes exactly one request per model execution, and the scheduling features that combine requests are unavailable to that model, because they all require a variable batch dimension to combine along. ## Variable dimensions Any entry in `dims` may be `-1`, meaning "variable, supplied per request". Text and sequence models lean on this heavily: `dims: [-1]` for a token-id vector of unknown length. Use it deliberately rather than everywhere — a fully variable shape gives Triton and the backend less to validate, so genuine client mistakes surface later, inside the framework, with a worse error message. The `reshape` field exists for the mismatch case: when the model's tensor has no batch dimension at all but you still want Triton to batch, `reshape { shape: [ ] }` lets Triton insert and remove the dimension around execution. ## Data types and names `data_type` uses Triton's own spelling — `TYPE_FP32`, `TYPE_FP16`, `TYPE_INT32`, `TYPE_INT64`, `TYPE_BOOL`, `TYPE_STRING`, `TYPE_UINT8` — which is then mapped onto the backend's native type. `name` must match the tensor name in the artifact exactly (the ONNX graph input name, or the Python backend's `pb_utils.get_input_tensor_by_name` argument). A typo here fails at load for backends that can introspect their model, and at execute for those that cannot. ## An input may be optional An input block with `optional: true` is one clients may omit; the backend must handle its absence. This is useful for models that take an optional mask or per-request parameter, and it is preferable to inventing a sentinel value. ## When you can skip the whole file For TensorRT, ONNX, TensorFlow SavedModel and OpenVINO models the artifact already carries its shape metadata, so Triton's auto-complete can generate the configuration and `config.pbtxt` becomes optional; the Python backend can do the same by implementing `auto_complete_config`. Auto-complete is on by default and `--disable-auto-complete-config` turns it off. Note what auto-complete infers: shapes and types, and a batching setting derived from whether the artifact's first dimension is variable. It cannot guess your throughput intent, so in production most teams write the file explicitly anyway — it is the place where shape, batching cap and scheduling live where a reviewer can see them. ## Choosing the number `max_batch_size` is a ceiling, not a target: it bounds memory per execution and bounds the worst-case latency of a batched execution. Set it from what one execution's activations cost on your GPU and from the latency you can accept for the last request in a full batch — then verify under load rather than trusting the arithmetic.