Do you standardize one quantization scheme across a serving fleet, or choose per model?

## Why the naive answer fails in both directions "Standardize on one scheme" fails because the fleet is heterogeneous. FP8 arithmetic exists only on Ada/Hopper-generation cards and newer; mandating FP8 fleet-wide means every Ampere node runs a dequantizing fallback that saves memory and buys no speed. "Choose per model" fails because the cost is not the choosing — it is everything that follows each choice. The workable position is a **default per hardware class, with an evidence-gated exception process**. ## The real cost of a long tail of schemes For each (model, scheme, GPU generation) triple you are on the hook for: - **An accuracy validation run** against the full-precision baseline, on task evals that stress structured output and long context. This is human time, not just compute. - **A performance benchmark** at production concurrency, because the same scheme behaves differently per model shape and per card. - **An artifact** in the model registry, with its own storage, provenance and revision pin. - **Re-validation on engine upgrade**, because the engine may route the same checkpoint to a different kernel, or stop accepting the method entirely. Ten models times three schemes is thirty of those, and the marginal one always looks cheap. Standardization is a bet that the aggregate maintenance bill exceeds the per-model tuning gains — which it usually does, except for the two or three highest-volume models where a few percent of GPU spend is real money. ## Deprecation is an operational, not academic, risk Quantization methods churn faster than almost anything else in a serving stack, because they are close to fast-moving kernels. The concrete current example: vLLM 0.27 raises on `fbgemm_fp8` and `fp_quant` unless the operator explicitly passes `--allow-deprecated-quantization`. Failing closed is the right design — silently substituting a different scheme would change model numerics without notice — but it means an engine bump can crash-loop a deployment that nobody touched. The fewer distinct schemes in production, the smaller that blast radius, and the shorter the release-note review before an upgrade. ## Compiled engines raise the stakes vLLM and TGI resolve kernels at load time, so a checkpoint can move between SKUs and simply take a different path — degraded, perhaps, but running. TensorRT-LLM compiles ahead of time: `trtllm-build` produces an engine pinned to a GPU architecture and precision, served through Triton's TensorRT-LLM backend. Moving that workload to a different card generation is a rebuild in a build pipeline, not a scheduling decision. If your fleet uses compiled engines, quantization scheme becomes a build-matrix dimension, and the argument for keeping that matrix small gets much stronger. ## A defensible policy 1. **Default per SKU class.** FP8 on FP8-capable cards; 4-bit AWQ/GPTQ in a fast-kernel-compatible layout, or INT8 W8A8, on Ampere. Write it down, including the acceptance thresholds. 2. **Keep a full-precision reference.** At least one bf16 replica or an on-demand path for the highest-value models, so quality regressions can always be A/B'd against ground truth and a rollback exists that is not itself quantized. 3. **Exception process.** A team may deviate with measured evidence: quality delta against the baseline and throughput at production concurrency, both on the target SKU. "It felt faster" is not evidence. 4. **Pin and version.** Checkpoint revision, scheme, engine version and eval results recorded together. Quantized artifacts get republished upstream; an unpinned dependency is an unreviewed model change. 5. **Gate engine upgrades** on a scheme-compatibility check across the deployed set — a cheap script, and the difference between a routine bump and a fleet incident. ## Where per-model choice genuinely wins Do not over-rotate to uniformity. Real reasons to deviate: a model whose only credible open checkpoint exists in one scheme; an MoE model where expert-specific methods matter; a very high-volume endpoint where a measured few percent of throughput pays for the extra validation many times over; or a model that demonstrably degrades at the fleet default and passes at a wider bit width. The policy exists to make those decisions explicit and evidenced, not to forbid them. ## The framing to give an interviewer Say that hardware sets the floor (per-generation defaults are forced), that the marginal cost of a scheme is validation and re-validation rather than compute, that deprecation now fails closed and therefore couples quantization choice to engine upgrades, and that you would run a documented default plus an evidence-gated exception path with a full-precision reference retained.