How do you choose calibration data for post-training quantization?

## Why the set matters at all Post-training quantization has exactly one channel through which your specific deployment can influence the result: the calibration pass. Whatever the algorithm — plain range estimation, error compensation, salience scaling, importance weighting — it looks at the activations produced by those samples and nothing else. The quantized checkpoint is, in a real sense, fitted to that corpus. Teams routinely spend a week choosing between quantization algorithms and thirty seconds choosing the data those algorithms will see, which is backwards. ## Size The practical range is small: on the order of 128 to 512 sequences at the model's typical serving length. The quantities being estimated — per-channel magnitudes, per-layer output error, curvature — converge quickly, and past a few hundred sequences the marginal sample changes almost nothing. There is also a downside to piling on more: methods that fit a correction to the calibration set (the error-compensating family especially) can overfit an estimate built from a finite sample, and a huge but unrepresentative set makes that worse rather than better. More data is not the lever; more *representative* data is. ## Domain match The narrower the deployment, the more domain match pays. Consider quantizing a defect-classification model to run on an on-prem factory edge box with no internet egress. The traffic it will serve is inspection reports and line telemetry in a house vocabulary, with recurring part numbers and terse operator shorthand. Calibrating with five hundred samples drawn from actual line-inspection logs gives the algorithm the activation distribution it will really encounter; calibrating on a generic web corpus gives it a plausible-looking distribution from a different world. Conversely, for a general assistant serving open-ended traffic, a broad and diverse corpus is the honest choice — matching *the* distribution means matching a wide one. The air-gapped case adds a constraint worth naming in an interview: the calibration data cannot leave the plant, so the quantization run happens inside the perimeter, and the resulting scales encode statistics of proprietary data. That is usually fine, but it is a data-governance fact, not a neutral build step. ## Format match This is the most common unforced error. An instruction-tuned model in production never sees bare prose: it sees a chat template, a system prompt, tool definitions, sometimes retrieved documents pasted into a long context. Those structural tokens produce their own characteristic activation patterns. Calibrating on raw text from a generic dataset therefore calibrates a distribution the served model does not visit. The fix is trivial — render your calibration samples through the exact same prompt template the application uses — and the omission is easy to make. Length deserves the same treatment. If production requests routinely run tens of thousands of tokens, calibrating exclusively on short snippets misses the activation behaviour of long-context positions. ## Mixture-of-experts coverage Almost all current large open-weight models are mixture-of-experts, and this changes the calibration calculus. Each token is routed to a small subset of experts, so a calibration set of a few hundred narrow sequences may leave many experts touched by very few tokens — or none. Their scales then rest on a near-empty sample. Practical responses are to enlarge and diversify the set specifically to spread routing, to check per-expert token counts during the calibration pass as a diagnostic, and to be suspicious of a run where the coverage histogram has a long empty tail. ## Contamination and leakage Two hygiene rules. Do not calibrate on your evaluation set — the quantized model is fitted to those activations, and your measured degradation will be optimistically biased. And treat calibration samples as data you have processed: if they contain personal or regulated content, the pipeline that touches them inherits that classification even though the output is only a set of scales. ## Detecting a mismatch after the fact The characteristic symptom is a quantized model that looks fine on generic perplexity or a public benchmark and degrades on your own task suite — precisely because the generic measurement matches the calibration corpus and your traffic does not. The defence is to hold out a task-specific eval set that was never used for calibration, and to compare the full-precision and quantized models on it rather than on a generic proxy. For activation quantization with static ranges, monitoring saturation rates in production is the direct signal that the calibrated ranges were too narrow. ## What a strong answer sounds like Size, domain, format, length, and — for current architectures — expert coverage; plus the discipline of keeping calibration and evaluation data disjoint. Ending on the failure symptom, rather than the recipe, is what marks the answer as coming from someone who has shipped one of these.