After quantizing Llama to 4-bit, how do you verify the accuracy loss is acceptable?

## Why this needs a process Quantization does not fail loudly. A 4-bit Llama still speaks fluently, still answers general questions, and still scores respectably on public benchmarks. What it loses is *precision*: exact formatting, reliable schema adherence, correct arithmetic chains, tool arguments that match the declared JSON Schema, and stability at the far end of the context window. Those are exactly the behaviours production pipelines depend on and public benchmarks under-weight. So the verification job is to build a measurement that looks where the damage actually lands. ## Establish a control You cannot say "the 4-bit build is fine" in isolation; you can only say it is or is not different from a reference. Use the FP16/BF16 weights as the control where you can afford to run them, or Q8_0 / int8 as a practical proxy — 8-bit is close enough to lossless that a 4-bit-versus-8-bit comparison is honest. Fix everything else: same prompts, same chat template, same seeds, same decoding parameters, same stop conditions. Any of those drifting between runs will produce differences you will wrongly attribute to quantization. ## Layer 1 — perplexity as a smoke test Perplexity on a held-out corpus (llama.cpp ships a perplexity tool for exactly this) is cheap and catches gross breakage: a build with a mangled tensor mix or a bad calibration run shows a large delta immediately. Its limits matter just as much: - It averages over every token, so a model that is right 99 % of the time and catastrophically wrong on closing braces looks fine. - It is measured on generic text, which may look nothing like your traffic. - Absolute values are not comparable across corpora or tokenizers, only deltas on an identical setup are. So use it to *reject* builds, never to *accept* them. ## Layer 2 — task evals on your own data This is the load-bearing layer. Build a few hundred graded examples from real traffic, and score deterministically wherever possible: - **Schema validity.** Does the structured output parse and validate? Percentage-valid is a brutally clear metric and is one of the first things to sag at low bit widths. - **Tool-call correctness.** Right tool chosen, arguments well-formed and semantically right. - **Executable checks.** For code, does it compile and pass tests? For extraction, does the field match ground truth exactly? - **Multi-step reasoning.** Chained arithmetic or multi-hop questions, where small per-step degradation compounds. - **Long context.** Sample at the long end of your real length distribution, not just short prompts. Retrieval-from-context accuracy is quantization-sensitive. - **Non-English and domain jargon**, if you serve them — a build calibrated on English web text can regress specifically there. Use greedy decoding (temperature 0) for the comparison so you are measuring the weights, not the sampler. Then, separately, sanity-check with your production sampling settings, because greedy hides some instabilities. Be realistic about statistics: a two-point difference on 100 examples is noise. Size the eval set so the difference you care about is detectable, and report the delta with an interval rather than a bare number. ## Layer 3 — paired human review Run the same fifty to a hundred real prompts through both builds and read the pairs side by side, blind if you can. Metrics miss register shifts, verbosity creep, instruction-following slippage and repetition loops. This is also the step that catches the pathological failure mode of aggressive quantization: degenerate repetition at long generations. ## What the ladder usually looks like Run 8-bit, 4-bit and 3-bit through the same harness. Typically 8-bit is indistinguishable from the reference, 4-bit shows small but measurable regressions concentrated in the precise tasks, and 3-bit and below fall off sharply — with the fall being far steeper for small models. A 70B at 4-bit is a mild compromise; a 1B–3B model at 3-bit often is not usable for structured work at all. Model size buys redundancy, and redundancy is what quantization spends. ## Deciding and keeping it honest Pick the lowest bit width whose regressions you can state and accept, with the memory saving written next to it — the decision is a tradeoff, so present it as one. Then keep the harness: requantize when weights are updated and re-run it, because a new build of the "same" quant level is a new artifact, and if the calibration corpus or the quantizer version changed, so did the model. Finally, watch for confounds that masquerade as quantization damage: a wrong or subtly different chat template, a tokenizer mismatch, changed stop tokens, or a different sampling default in the serving stack. Confirm the control and the candidate share all of those before blaming the bits.