Why are activations harder to quantize than weights in a transformer?

## Two different jobs "Quantizing a model" can mean two quite different things. **Weight-only** quantization compresses the parameters and leaves the arithmetic in floating point: the low-bit weights are dequantized on the way into the matrix multiply. **Weight-and-activation** quantization compresses both operands, so the multiply itself can execute on an integer or low-precision floating-point path. The first is primarily about the size of the thing you have to move; the second is about what instruction the accelerator gets to issue. That difference in purpose is why the two are attempted so differently, and why the second is much harder. ## Weights are a static, inspectable tensor After training, a weight matrix is a fixed array. You can compute its exact distribution, split it into groups of 64 or 128 along whichever axis suits the kernel, and choose a scale per group with full knowledge of what it must cover. If a group has an awkward spread you can shrink the group. Nothing at runtime can surprise you. Rounding error is then a one-time, offline optimization problem — which is exactly what algorithms in the weight-only family attack. ## Activations are a distribution, not a tensor An activation tensor exists only while a particular batch of tokens is flowing through. Its range depends on the input, and you are choosing scales for a distribution you can only sample. There are two strategies: - **Static scaling**: estimate ranges during calibration and freeze them. Cheapest at runtime, but any production input outside the calibrated range gets clipped. - **Dynamic scaling**: compute the scale on the fly, typically per token, from the values actually present. Robust to distribution shift, but it costs a reduction over the tensor before every quantized matmul and complicates kernel fusion. Even with dynamic scaling, the shape of the distribution is the real problem. ## The outlier structure Large transformers develop **persistent per-channel outliers**: specific hidden dimensions whose activation magnitudes sit far above the rest, consistently, across nearly all tokens and layers. They are not noise and you cannot clip them away — they carry information the model depends on, and truncating them damages quality out of proportion to their count. The phenomenon becomes pronounced as models scale past a few billion parameters, and it is why naive 8-bit activation quantization that worked fine on small vision models fails on an LLM. The geometry is unforgiving. A symmetric per-tensor scale must cover the largest absolute value present. If one channel is dozens of times larger than the typical channel, then the typical channels are represented by only a few distinct levels out of the available range, and their relative error explodes. ## Why per-channel does not simply solve it For weights, per-output-channel scales are easy: each output channel's scale multiplies a whole row of the result, so it factors cleanly out of the accumulation. For activations, the outlier structure runs along the *input*-channel axis, which is the axis being summed over inside the matmul. A per-input-channel scale therefore cannot be pulled out of the accumulator — it would have to be applied inside the inner loop. That structural mismatch, not a lack of imagination, is why per-token scaling (which does factor out) is the granularity that hardware kernels actually offer. ## Outlier migration SmoothQuant's contribution is to notice that the difficulty can be moved rather than solved where it sits. For a linear layer computing `y = x @ W`, pick a per-input-channel factor `s` and compute `(x / s) @ (diag(s) @ W)` instead. The product is unchanged — it is a diagonal rescaling that cancels — but the activation's outlier channels have been shrunk and the corresponding weight rows enlarged. Since weights tolerate a much wider spread (they are static and finely grouped), you have traded an intractable activation problem for a tractable weight one. The factor is usually chosen with an exponent that balances how much difficulty each side absorbs. Crucially the division of `x` is not a runtime cost: it can be folded into the weights and biases of whatever produced `x` — the preceding normalization or linear layer — during the offline quantization step. At serving time the model just has different weights. ## Other levers Keeping a handful of known-outlier channels in higher precision and running them as a separate small matmul is the mixed-precision alternative; it works but fragments the kernel and is hardware-unfriendly. Rotation-based methods multiply activations and weights by paired orthogonal matrices to spread outlier energy across channels before quantizing, achieving something similar to migration without a hand-picked factor. ## What an interviewer is listening for The weak answer is "activations are dynamic". The strong answer adds the structural fact — persistent per-channel outliers — explains why the outlier axis is the one the matmul sums over, and names migration and per-token scaling as the two ways out.