How do you decide whether 4-bit serving is worth its quality cost?

## Both sides have to be measured, on your hardware Published quantization results describe someone else's checkpoint on someone else's accelerators with someone else's traffic. The two quantities that decide this — how much you gain and how much you lose — are both workload-specific. A decision made from a vendor blog post is a guess wearing a citation. ## What the memory actually buys The headline benefit is that low-precision weights occupy a fraction of the memory. What that fraction is worth depends entirely on what was constraining you: - **Fitting at all.** If the high-precision model does not fit on the hardware you have, quantization is not an optimization, it is the deployment. There is no trade to weigh, only a quality floor to check. - **Fewer or cheaper accelerators.** Dropping from multiple devices to one removes cross-device communication from every forward pass as well as cutting cost, which can improve latency independently of arithmetic. - **Headroom for context and concurrency.** Weights are a fixed cost; the KV cache grows with concurrent sessions and their context length. Freeing weight memory converts directly into how many long-context sessions a replica can hold at once, which for many services is the binding constraint rather than raw arithmetic speed. - **A bigger model on the same box.** The most valuable option and the most often missed. See below. ## The throughput side, honestly Token generation is dominated by moving weights from memory rather than by arithmetic, so shrinking the weights genuinely speeds decoding — that part is real and usually the largest single-request effect. But two caveats decide whether you see it. First, hardware support: if the accelerator executes the low-precision format natively the saving lands, while if the kernel has to dequantize to a wider type before each matmul, the overhead can eat much of it. Second, the phase mix: prompt processing is far more arithmetic-bound than generation, so a workload dominated by long prompts and short answers gains less than one that streams long outputs. Measure the end-to-end number on your traffic shape rather than reasoning about it. ## The cost side The quality loss must come from a task evaluation, sliced by the segments you care about, with a regression budget written down before the numbers arrive. Aggregate benchmark deltas and perplexity are not adequate instruments for a production decision. Sample enough to distinguish the effect from generation noise, and be explicit about which slice the budget applies to — the worst affected segment, not the mean. ## The comparison that actually matters Teams habitually frame this as "our model at 16-bit versus our model at 4-bit". Under a fixed memory budget the more useful frame is: **which model can I run at which precision in the memory I have?** A larger model quantized to 4 bits frequently outperforms a smaller model at full precision occupying the same footprint, because the capability difference between model sizes is usually larger than the degradation from careful 4-bit quantization. Answering the narrow question can leave a substantial win unclaimed. ## Tiering by surface One global decision is the wrong shape. Different product surfaces have wildly different tolerance: - **High tolerance** — draft generation, summarization for human review, ranking candidates a person will check. A point of accuracy here buys real capacity. - **Low tolerance** — anything whose output is executed rather than read: tool arguments, code applied automatically, structured records written to a system of record. These surfaces feel format and decisive-token degradation first and hardest. Running a quantized build for the tolerant surfaces and a higher-precision build for the sensitive ones is a legitimate architecture, and its cost is operational complexity you should price in, not pretend away. ## Keeping the decision reversible Whatever you choose, keep the higher-precision build deployable behind a flag, roll the quantized one out to a slice of traffic first, and watch production signals that the offline eval cannot capture — schema-error rates, tool-call retry rates, user-visible retries, escalation rates. Offline evals are a filter, not a verdict. The organizations that get this right treat a quantization change with the same care as a model change, because from the user's point of view that is exactly what it is. ## When the answer is simply no If the model already fits comfortably, the traffic is small, the surface is high-stakes, and the hardware has no native support for the format, quantization is spending quality to solve a problem you do not have. Say so.