In Triton, how do preferred_batch_size and max_queue_delay_microseconds shape dynamic batching?

## What the dynamic batcher is A GPU is far more efficient running one kernel over 16 inputs than 16 kernels over one input each. But your clients are independent — a browser, a mobile app, a batch job — and none of them knows about the others. Triton's **dynamic batcher** closes that gap on the server: it accepts single-request inferences, queues them for a few microseconds, and hands the backend one batched execution. This is purely a scheduler. It does not change the model, and clients need no code change. It is enabled per model, in that model's `config.pbtxt`, by adding a `dynamic_batching` section. The model must actually support batching — its first tensor dimension is the batch dimension and `max_batch_size` must be greater than zero — otherwise the scheduler has nothing to combine. ## preferred_batch_size `preferred_batch_size` is a repeated field listing batch sizes that are "good" for this model, for example `[ 8, 16 ]`. When the queue contains enough requests to form one of those sizes, the scheduler dispatches immediately rather than waiting for anything better. If you do not specify it, Triton chooses preferred sizes on its own based on the model configuration, so it is an optimisation hint, not a requirement. Why would some sizes be better than others? Compiled backends often have kernels or optimisation profiles tuned around particular shapes, and a TensorRT engine built with an optimisation profile centred on batch 8 will run batch 8 more efficiently than batch 7. Padding, tiling and occupancy all favour round numbers. If you have no such structure, leaving the field out is fine. ## max_queue_delay_microseconds This is the knob interviewers actually probe, because it is the one that spends latency. It is the maximum time the scheduler will hold a batch that has not reached a preferred size, hoping more requests arrive. The default is **0**: form a batch from whatever is queued and dispatch as soon as a model instance is free. The key asymmetry: the delay is an upper bound, not a fixed cost. Under heavy load, batches fill before the timer expires and the added latency is near zero. Under light load, nothing arrives during the wait, so you pay the full delay on every request and gain almost no batch size. That is why blindly setting a 50 ms delay on a low-QPS endpoint makes the service strictly worse — a very common wrong answer. A reasonable starting point is a delay on the order of one model execution time (single-digit milliseconds for most vision models), then measure. The thing you are measuring is not just throughput: watch the queue-time component of latency and the achieved average batch size. If batch size did not move, the delay bought nothing. ## How it interacts with instances Batches are dispatched to a free model instance. Batching and instance count are complementary but pull in different directions on a saturated GPU: more instances add concurrent executions and memory pressure, larger batches add work per execution. On a GPU-bound model, a bigger batch is usually the cheaper win. ## Ragged inputs and queue policy Requests with different input shapes cannot always be stacked. Triton pads or refuses depending on the backend; for backends that support it, `allow_ragged_batch` on an input lets variable-length inputs be concatenated instead of padded, with a `batch_input` telling the model where each element starts. Separately, `default_queue_policy` inside the `dynamic_batching` block bounds the queue itself: `max_queue_size` caps depth, and `timeout_action` decides whether an over-aged request is rejected or delayed. Under overload, rejecting fast is often better than letting a queue grow beyond any useful latency. `priority_levels` plus per-level queue policies let you run interactive traffic ahead of bulk traffic through the same model, which is a cheap alternative to standing up a second deployment. ## Where it does not apply The dynamic batcher forms a batch, runs it, and returns — a request/response shape. Generative LLM serving needs a batch whose membership changes every decoding step, which is a different scheduler entirely: Triton's TensorRT-LLM and vLLM backends do their own in-flight/continuous batching internally, and you do not configure it with `dynamic_batching`. Similarly, stateful models where request N depends on request N-1 need the sequence batcher, not this one.