In ExecuTorch int8 quantization, when do you need calibration data?

## Weights are easy, activations are the problem After training, weights are constants. You can look at them, compute a scale, and store int8 values — no data needed, ever. Activations are different: their range depends on the input, and you do not have the input until inference. Every quantization variant is really an answer to "what do we do about activation scales?", and the answer decides whether you need calibration data. ## Static PTQ: freeze the scale ahead of time Static quantization commits to one scale per activation tensor, chosen offline. To choose it you must observe realistic activations, which is what the calibration pass does: run the prepared module forward on representative inputs so its observers accumulate min/max or a histogram, then let `convert_pt2e` turn those into constants. What you get for that effort is the fastest option on device. Because the scale of a conv's output is already known, the backend can hand int8 straight to the next int8 op without a dequantize/requantize round trip. Whole chains — conv, batchnorm, relu, conv — collapse into fused integer kernels. This is why static is the default choice for CNNs and for anything where convolutions dominate the runtime. The cost is a dependency on the calibration set. It needs to look like production traffic: same preprocessing, same resolution, same lighting or speaker or language mix. A few dozen to a few hundred batches is usually plenty; the number matters far less than the representativeness. Ranges that are too narrow clip real inputs; ranges stretched by one freak outlier waste resolution on values that never occur, which is the argument for a histogram-based observer over plain min/max. ## Dynamic PTQ: compute the scale at runtime Dynamic quantization stores weights as int8 and leaves the activation scale to inference time: the kernel scans the incoming tensor, computes its range, quantizes, multiplies in integer, and rescales the output. Enable it in the ExecuTorch flow with `get_symmetric_quantization_config(is_per_channel=True, is_dynamic=True)`. Because the scale adapts per call, there is nothing to calibrate — you skip the data pass entirely, and the model cannot be embarrassed by an input distribution it never saw. That robustness is real: dynamic quantization degrades gracefully where a badly calibrated static model falls off a cliff. The cost has two parts. First, the per-tensor scan and rescale are real work on every inference; for a small op, that overhead can exceed the integer speedup. Second, the model cannot stay in the integer domain across ops, since each op re-derives its own scale, so you lose the fusion that makes static fast. Dynamic wins when the arithmetic is dominated by large weight matrices — transformer and RNN-style stacks, embedding-plus-linear models. There the memory traffic of the weights is the bottleneck, 4x smaller weights is the whole win, and the runtime scan is amortised over a large matmul. It is also the pragmatic first move when you have no calibration data available at all. ## Weight-only, the third point on the line Some flows quantize weights alone and dequantize them into float for the multiply. This buys the download and memory-footprint win with essentially no accuracy risk and no calibration, but little or no compute speedup, because the arithmetic is still float. It is a size play, not a latency play. ## Choosing, in practice A workable decision order: 1. **Convolutional vision model with a latency budget?** Static, per-channel weights, with a calibration set drawn from production data. 2. **Linear/matmul-heavy model, or no calibration data at hand?** Dynamic — cheap to try and hard to get badly wrong. 3. **Size is the constraint, latency is fine?** Weight-only. 4. **Static chosen but accuracy unacceptable?** Fix calibration coverage and per-channel weights first; escalate to QAT only after that. Always confirm the choice on device rather than from the table above: which variants a backend actually accelerates differs by backend and by operator, and a config the delegate cannot execute becomes explicit quantize/dequantize work in float — slower than the model you started with. ## Interview traps The most common wrong answer is "dynamic quantization needs a smaller calibration set". It needs none; that is the definition. The second is treating calibration as training — no labels, no loss, no backward pass; it is a forward-only instrumentation run. The third is assuming dynamic is always safer, which ignores that its per-inference scan can wipe out the speedup on small ops and that it forecloses cross-op integer fusion.