Serving Concepts

Static batching forms a batch, runs it to completion, and only then accepts new work, so every slot waits for the longest generation. Continuous batching re-forms the batch before each decode step: finished sequences leave and queued requests join immediately.

Add the four terms up. 70B weights at bf16 are about 140 GB of the 160 GB on the node, so after CUDA context and activations only a few GB are left for KV cache. It loads, but it serves almost no concurrency.

PagedAttention splits the KV cache into fixed-size blocks, each holding a set number of tokens. Every sequence keeps a block table listing which physical blocks hold its logical positions, and the attention kernel follows that table instead of striding through one range.

A contiguous allocator must reserve one slab per request sized for the longest output that request could produce. Most requests never grow into it, so the reserved tail sits idle, and the leftover gaps between slabs are too small for the next request to use.

Prefix caching reuses already-computed KV blocks when a new request starts with the exact same tokens as an earlier one, skipping that part of prefill. It misses whenever the prompt differs at the front, when the shared run is shorter than one block, or when those blocks have already been evicted.

Requests per second hides how much work each request does: one call may emit 40 output tokens, another 2,000. LLM capacity tracks output tokens per second, so report token throughput together with the input and output length distribution it was measured at.

Tensor parallelism splits each layer's weight matrices across GPUs, so all of them work on every token and must all-reduce partial results at each layer. Pipeline parallelism gives each GPU a different block of layers and passes activations forward once per stage.

Weight-only 4-bit helps only while decoding is memory-bandwidth-bound. At large batch the server becomes compute-bound, and every matmul must first dequantize weights back to 16-bit — extra work fp16 never pays, so throughput can fall.

A cheap drafter guesses the next few tokens. The expensive target model then scores all of those guessed positions in a single forward pass, keeps the longest prefix it agrees with, and emits one corrected token where it disagrees.

A new replica must wait for a GPU node, pull a multi-gigabyte container image, fetch tens of gigabytes of weights, load and shard them into VRAM, and run a warmup pass before it is useful. Each stage is minutes, and none of them is code you wrote.

Divide the hourly cost of the GPU by the tokens it actually produces in that hour. GPU $/hr divided by (tokens per second times 3600), times one million. The number that matters is measured throughput at your real utilization, not the card's peak.

The scheduler preempts a running sequence: it frees that sequence's blocks and returns it to the waiting queue, either discarding its cache to recompute later or copying it to host memory. The client sees a stall, not an error, and tokens already streamed are never taken back.

The server is past its saturation knee. Decode throughput is capped by memory bandwidth and KV-cache capacity, so extra concurrency no longer produces more tokens — it only waits in the scheduler queue, and that queue time lands entirely in time to first token.

Tensor parallelism runs an all-reduce roughly twice per transformer layer, on the critical path of every token. NVLink moves those messages at hundreds of GB/s between GPUs; PCIe offers roughly an order of magnitude less bandwidth and higher latency, so the collectives dominate the step.

Hardware decides. FP8 tensor cores exist only on Ada and Hopper generations and newer, so an H100 should serve FP8 weights and activations. An Ampere card like the A10G or A100 has no FP8 math and should serve 4-bit GPTQ or AWQ through a Marlin-class kernel, or INT8.

Validate both halves of the trade. On quality, compare against the full-precision baseline on real task evals and the failure modes quantization hits hardest. On serving, measure TTFT, inter-token latency and throughput at production concurrency — then canary traffic with the old replica warm for rollback.

Measure the acceptance rate and the per-position acceptance curve first, then the draft cost as a fraction of a target step. Low acceptance means a mismatched or weak drafter; high acceptance with no win means the drafter is too expensive or the draft length overshoots.

Round-robin equalizes request counts, but LLM requests differ in cost by orders of magnitude and run for seconds to minutes. One replica ends up holding all the long generations while another idles, so tail latency blows up even though every replica got the same number of requests.

Chunked prefill splits a long prompt's prefill into pieces that fit the scheduler's per-step token budget, so each step can carry prefill tokens and ongoing decode tokens together instead of stalling every generating sequence behind one huge prompt.

Capacity gates the choice, bandwidth sets the speed, and price decides between what is left. An 8B model at bf16 needs ~16 GB, so a 24 GB L4 or A10 holds it but leaves thin cache room; bigger cards buy concurrency and far faster decoding.

Identical prompts hit the server's prefix cache, uniform lengths remove the long-request tail that clogs real batches, and steady closed-loop pacing never produces bursts. The result is a throughput number production traffic cannot reproduce.

Drive the server with a token-aware load generator such as vLLM's vllm bench serve or GenAI-Perf, pinning prompt and output lengths and either an arrival rate or a concurrency cap. Report TTFT, per-output-token latency and inter-token latency percentiles alongside output token throughput.

The degree must divide the model's structure: the attention head count above all, plus the MLP intermediate size, and with grouped-query attention the much smaller KV-head count becomes the binding limit. Hardware adds its own cap — the GPUs must share one fast interconnect domain.

Prefer a prequantized checkpoint for production: calibration is already done, the layout matches a fast kernel, and startup stays short. Startup quantization is convenient for experiments but adds load time and usually lands on slower kernels — the exception is FP8, which quantizes online cheaply.

Use prompt lookup when the output largely copies the input — summarisation, document QA, code edits, structured rewrites. It costs no extra weights or VRAM and drafts almost instantly, but on open-ended generation with nothing to copy its acceptance rate collapses to near zero.

The verification rule accepts a drafted token with probability capped by the ratio of the target's probability to the drafter's, and resamples from a corrected distribution otherwise. The result is distributed exactly as the target model alone — the drafter changes speed, not output quality.

Route by adapter identity, not round-robin. A replica can only serve an adapter it currently has resident, and engines cap how many distinct adapters may appear in one batch, so hash adapters to replicas, keep each replica's hot set stable, and give a heavy tenant its own group.

A prefix cache lives inside one replica's own KV memory, so a shared preamble or a continuing conversation only skips prefill if the request lands on the replica that already holds it. Prefix-aware routing hashes the leading tokens and steers matching requests to the same replica.

Compare the running-sequence count against the configured concurrency cap and the KV-cache utilization gauge. Pinned at the cap with spare cache means the cap is throttling you; pinned cache means memory is the limit and no scheduler knob will fix it.