When does a learned clipping threshold beat a min-max calibration range in QAT?

## The two errors a range must trade Quantizing to `b` bits over a range `[lo, hi]` gives step size `scale = (hi - lo) / (2^b - 1)`. Every value in the tensor picks up one of two errors: - **Resolution error** — at most half a step, for values inside the range. Proportional to `hi - lo`. - **Clipping error** — for values outside the range, equal to their distance past the boundary. Unbounded, but only paid by values in the tail. Widen the range and you trade a small error paid by everyone for a large error paid by nobody; tighten it and you do the reverse. There is no setting that avoids both, so the range is a genuine optimization problem, not a bookkeeping detail. ## Why min-max is a bad default on skewed layers A min-max range is read off a calibration pass: run some batches, record the smallest and largest activation seen, use those as `lo` and `hi`. It has one appealing property — it clips nothing — and one severe failure mode: it is decided entirely by the single most extreme sample. Consider a layer whose activations are almost all within a small band, with a thin tail reaching two orders of magnitude higher. Min-max fixes the range from the tail. The step size becomes enormous relative to the bulk of the distribution, so the vast majority of activations land in a few adjacent levels — many are rounded to the same value, some to zero. The layer's output loses nearly all its expressive detail, and the network's accuracy falls even though the range was, in the narrow sense, "correct". Worse, min-max is not stable: it depends on which batches you happened to run, so a rare spike in one calibration batch changes the deployed model. ## The learned alternative The PACT-style approach makes the clipping threshold a **trainable parameter**, typically one per layer, applied to a bounded activation: values above the threshold are clipped to it, values below pass through, and the resulting bounded activation is quantized over `[0, alpha]`. The crucial part is that `alpha` participates in backpropagation. For an activation above the threshold, the layer's output equals `alpha`, so the derivative of the output with respect to `alpha` is one there — gradient flows to it from every clipped unit. For an activation below the threshold, the output does not depend on `alpha` and contributes nothing. So the loss itself decides: if clipping the tail is hurting the task, gradient pushes `alpha` up; if the coarse step size is hurting more, a decay term applied to `alpha` pulls it down against a diminishing benefit. The threshold settles where the marginal cost of each error is equal — which is precisely what you want and precisely what a rule based on observed extremes cannot compute, because that rule never looks at the loss. ## Static alternatives worth knowing A learned threshold is not the only escape from min-max. You can pick a **percentile** of the observed distribution instead of the maximum, or choose the range that minimizes the reconstruction error between the quantized and unquantized tensor. Both are cheap, need no extra parameter, and capture much of the benefit. The learned threshold's distinct advantage is that its objective is the *task* loss rather than a per-tensor proxy: it can discover that a particular layer tolerates aggressive clipping because a later layer compensates, which no per-tensor criterion can see. ## Ranges are not only about clipping — the granularity axis A separate lever attacks the same problem from a different direction: use **more than one range**. Per-tensor quantization gives a whole weight tensor one scale; per-channel gives each output channel its own. On a layer whose channels differ in magnitude by two orders of magnitude — depthwise-separable layers are the classic offender — a single per-tensor scale is set by the loudest channel and crushes all the quiet ones, regardless of how cleverly that single scale was chosen. Per-channel scales fix that structurally, and often recover more than any range-selection tweak can. Given a badly quantized model, checking granularity is usually the cheaper first move; learning the threshold is the refinement after it. ## When it is not worth it - **At 8 bits with well-behaved distributions.** The grid is fine enough that a sensible static range costs little, and the extra trainable parameter, its decay coefficient and the extra tuning surface are not repaid. - **When the pipeline cannot carry it.** A learned threshold is a parameter that must be trained, exported and honoured at deployment; if the target runtime resolves ranges its own way, a threshold that only exists in your training graph is a simulation that does not match reality. - **When the problem is elsewhere.** If accuracy is lost in a layer whose channel scales are wildly uneven, or because normalization was not folded before quantizing, tuning the threshold treats a symptom. ## Failure modes to watch A threshold with too strong a decay term collapses toward zero, clipping nearly everything and starving the layer; too weak a decay and it drifts up toward the min-max solution, giving back the resolution it was meant to buy. Both are visible directly: log the learned thresholds per layer through training and compare them to the observed activation percentiles.