Int8 PTQ cost you 4 points of accuracy on device — how do you recover it?

## Do not start with QAT QAT is the expensive answer: it needs the training data, a working fine-tuning loop, GPU time, and a commitment to repeat all of it every time the float model is retrained. A four-point drop is frequently caused by something upstream that costs an afternoon to fix. Work the ladder in order. ## Rung 1: check that you measured the right thing Two mistakes produce fake numbers. Evaluating the **prepared** model tells you nothing — it is still float32, observers only record. And evaluating the converted model **on the host** can disagree with the device, because a delegate's kernels are not bit-identical to the simulated numerics. Establish the real baseline: converted model, on the target device, on your real eval set. ## Rung 2: audit the calibration set Static quantization is only as good as the ranges it observed. Ask what production inputs look like versus what you calibrated on — resolution, preprocessing, lighting, speakers, class balance. Ranges that are too narrow clip real inputs at qmin/qmax; ranges stretched by a single freak batch waste the integer grid. Two fixes: widen the calibration set to cover the real distribution, and use a histogram-style observer instead of plain min/max so a lone outlier stops dictating the scale. This rung alone often recovers most of the loss. ## Rung 3: per-channel weights With per-tensor weight quantization, one scale covers an entire convolution kernel. If channel 7 has weights ten times larger than channel 3, the shared scale is set by channel 7 and channel 3's weights round to a handful of distinct values — sometimes to zero. Per-channel gives each output channel its own scale, costs almost nothing at inference on kernels that support it, and is why `get_symmetric_quantization_config(is_per_channel=True)` is the sane default. If you shipped per-tensor, flip this before doing anything harder. ## Rung 4: localise, then exempt Quantization damage is rarely uniform; usually two or three layers cause most of it. Run the float and converted models on the same inputs and compare intermediate activations layer by layer — a signal-to-quantization-noise ratio per layer, or just relative error. Typical offenders are the first layer (raw input dynamic range), the final classifier or logits head, and any block with heavy outliers. Having found them, exempt them via `set_module_name(name, ...)` on the quantizer — a different config, or none at all — and re-run prepare/convert. You give back some speed and some size in exchange for accuracy, and because you know *which* layers you exempted, that trade is explicit rather than accidental. ## Rung 5: QAT If the ladder runs out and you still miss the accuracy floor, QAT is the tool that genuinely changes the answer, because it lets the weights adapt to rounding instead of being rounded after the fact. Mechanically it mirrors PTQ. Build the quantizer with `get_symmetric_quantization_config(is_per_channel=True, is_qat=True)`, call `prepare_qat_pt2e` instead of `prepare_pt2e`, and instead of a forward-only calibration loop you run a real fine-tuning loop with labels and an optimizer. The inserted **fake-quant** modules round the tensor in the forward pass — so the loss reflects the integer grid — while letting gradients pass through, so training can still make progress. A few epochs at a low learning rate, typically a small fraction of the original schedule, is normal; QAT is fine-tuning, not retraining. Then `convert_pt2e` produces the real quantized graph. Two cautions. **Evaluate the converted model, not the QAT model** — the fake-quant graph is still float and can look better than what ships. And **QAT is not a one-off**: every subsequent float retrain has to go back through the QAT loop, so you are committing to it as a permanent pipeline stage, with the data access, compute budget and revalidation cost that implies. ## What a good answer sounds like Name the ladder and the reasoning behind its order: measurement first because it is free, calibration second because it is the most common cause, per-channel third because it is a one-flag fix, selective exemption fourth because it converts an accuracy problem into a controlled speed trade, and QAT last because it is the only rung with an ongoing organisational cost. Mentioning that you would set an accuracy floor and a latency budget *before* starting — so you know when to stop climbing — is what separates a senior answer from a list of techniques.