Serving Concepts

Size the per-iteration token budget by how long a step may take, since step duration is every user's inter-token latency, and size the concurrent-sequence cap by how much KV cache you actually have. Then measure at your target load rather than guessing.

Suspect something outside the engine's own budget. Serving engines claim their KV pool at startup and then queue or preempt rather than OOM, so a steady-state failure usually means another process on the card, an activation spike from an unusually long prompt, or a utilization fraction set too high.

Bigger blocks mean fewer table entries and longer contiguous reads, so kernels and bookkeeping get cheaper. The cost is coarser granularity: more wasted space in each sequence's partly-filled tail block, and prefix sharing that only matches in larger chunks.

Goodput counts only the requests that finished inside the latency SLO — for example TTFT under 500 ms and per-output-token latency under 50 ms. Raw throughput counts a request that took 30 seconds as a success, so it can rise while every user's experience gets worse.

Expert parallelism places whole experts on different GPUs instead of slicing every matrix. Each MoE layer then does an all-to-all: tokens are dispatched to the GPUs owning their chosen experts and the results are gathered back. Uneven routing makes one GPU the straggler.

Keep tensor parallelism inside each node — TP=8 over the intra-node fabric — and cross the network with pipeline parallelism, PP=2. The node boundary is the slow link, and only pipeline parallelism sends little enough traffic to survive it. First check the model cannot simply fit on one node.

At small batch a decode step has idle arithmetic capacity, so scoring extra drafted positions is nearly free. As concurrency rises the GPU's matrix units fill up, and the extra positions — most of the rejected ones pure waste — start costing real time and stealing throughput from other requests.

Scale to zero when idle hours dominate the bill and the first caller after an idle period can tolerate minutes — internal tools, batch pipelines, dev environments, rarely used per-tenant models. Never for interactive traffic with a time-to-first-token target, or where GPU capacity may not be reacquirable.

When sustained volume is high enough to keep GPUs genuinely busy, and when something other than price — data control, a custom checkpoint, predictable latency — is also on the table. Below that, per-token pricing wins because you are not paying for idle hardware.

Pinning blocks through a user's think time blocks capacity for every idle session, so it only pays for short gaps and high-value sessions. The usual answer is neither extreme: free the blocks but leave them in the prefix cache, where they are reused if still present and recomputed if not.

End-to-end latency scales with answer length, which the model and caller choose, so an e2e p95 target is mostly an SLO on how long answers are. Bound time to first token and per-output-token latency instead, each at a stated load level and length regime.

Default to replicas whenever the model fits on one GPU: independent copies add throughput linearly with no cross-GPU collectives and no shared failure domain. Shard only when the model does not fit, when per-token latency must drop, or when one replica needs a bigger KV-cache pool.

Standardize per GPU generation, not across the whole fleet — hardware decides which schemes have fast kernels. Allow per-model exceptions only with evidence, because every model-and-scheme pair carries its own validation, artifact and re-validation-on-upgrade cost.

When one continuously batched replica can no longer serve two SLOs at once. Disaggregation runs prefill workers and decode workers separately and ships the KV cache between them, so each can be sized, tuned and scaled independently — worth it only at cluster scale.

When the constraint is cost or capacity rather than per-request latency. Speculation raises total GPU work and consumes memory that limits concurrency, so a throughput-bound or budget-bound fleet is usually better served by more replicas, a smaller or quantized model, or better batching.