How does TGI's --waiting-served-ratio decide when to pause decoding for queued requests?

## The scheduling decision this flag answers TGI's router keeps a running batch that is producing one token per sequence per step, plus a queue of requests that have arrived but not yet been prefilled. Prefill and decode cannot run in the same forward pass on the same batch, so admitting a newcomer means stopping decode, running a prefill, and concatenating the result into the batch. That pause is paid by every request already streaming. So the router needs a rule for when the pause is worth it. TGI's rule has two parts. ## --waiting-served-ratio This is a ratio of *queued* requests to *running* requests. If it is 0.3 and eight requests are decoding, the router starts considering an admission pause once roughly three are waiting. The intuition: when the queue is small relative to the batch, the newcomers are a minor share of demand and the interruption cost outweighs their gain; when the queue grows relative to the batch, those waiting requests represent enough of your traffic that leaving them queued is the bigger latency problem. The ratio form is deliberate. An absolute queue threshold would behave completely differently on a server running batches of four versus batches of sixty-four. A ratio scales with whatever batch size the memory budget currently supports. Note also that admission is still bounded by memory: there must be room in the batch under `--max-batch-total-tokens` for the new sequences' cache. Crossing the ratio expresses intent, not permission. ## --max-waiting-tokens Ratio alone has a starvation hole. Imagine one long generation running alone and a single request queued behind it: one waiting over one running is above the threshold, but if the batch were larger the same lone request would sit indefinitely. `--max-waiting-tokens` (default 20) forces the issue - after that many decode steps have elapsed, waiting requests get admitted if the batch has room. The useful way to read this number is in wall-clock terms. If a decode step takes about 20 ms, 20 steps is roughly 0.4 seconds, which is the worst-case extra queue delay this backstop imposes. That converts directly into a TTFT budget line: if your SLO says first token within one second and your model decodes at 25 ms per step, a `--max-waiting-tokens` of 20 spends half a second of that budget in the worst case. ## Tuning by workload shape - **Interactive chat, TTFT-sensitive**: lower both. Admit early and often. You are deliberately paying inter-token jitter for existing streams to keep new users from staring at an empty box. Chunked prefill in recent TGI releases makes that stall shorter than it used to be. - **Batch/offline scoring**: raise them. Nobody is watching a stream, so long uninterrupted decode runs maximise tokens per second per GPU. Queue delay is irrelevant when the job is measured in total wall clock. - **Mixed traffic on one endpoint**: this is where candidates usually get stuck, and the honest senior answer is that a single scheduler cannot satisfy both SLOs. Split into separate deployments with separate settings rather than searching for a ratio that pleases everyone. ## How you observe the effect Don't tune these blind. TGI's Prometheus endpoint exposes queue-time and inference-time series separately, so you can see which side of the tradeoff you are on: if queue duration dominates end-to-end latency, admission is too lazy; if queue duration is near zero but inter-token latency is spiky, you are interrupting decode too often. Change one flag at a time and re-measure under a fixed load profile - these settings interact with arrival patterns, so an A/B under synthetic uniform load can mislead you about bursty production traffic. ## What interviewers listen for The signal is whether you can state the tradeoff in both directions and name who pays. Weak answers describe the flag as 'batching efficiency' with no mention that the cost lands on requests that are already streaming. Strong answers translate `--max-waiting-tokens` into milliseconds against a TTFT SLO and admit that mixed workloads want separate deployments.