Why do 4-bit serving recipes keep certain layers at higher precision?

## The premise: error is not uniform It is tempting to model quantization as a single knob applied to a homogeneous pile of weights. Measured layer by layer, a transformer behaves nothing like that. Quantize one block at a time and measure the resulting drop on a task metric and you get a spiky profile: most blocks cost almost nothing, and a handful cost more than all the others combined. Every practical low-precision recipe exploits that shape. ## What makes a tensor sensitive: outliers Quantization represents a group of values with a small set of levels plus a shared scaling factor. The scale must reach the largest magnitude in the group. In several places inside a transformer — most notoriously the projections that consume attention output and the inputs to certain feed-forward projections — a few activation channels carry magnitudes one or two orders larger than the rest. The scale stretches to cover them, and everything else in the group collapses onto a couple of representable levels. The result is not gentle rounding noise; it is the near-total loss of the small-magnitude signal that most of the computation actually depends on. This is why "outlier channels" is the vocabulary of the field. Whole families of algorithms exist to move that difficulty around — that machinery belongs to the algorithms discussion — but even with them, the outlier-heavy tensors remain the least tolerant of aggressive width reduction. ## Position in the stack Two positional effects show up repeatedly. **Early layers.** The first block or two operate on the raw embedded representation. Error injected there propagates through every subsequent layer, and the network has had no opportunity to build redundancy that could route around it. **Late layers and the head.** The final block and the output projection map hidden state to logits directly. There is no downstream computation left to absorb a perturbation — a small logit shift is immediately a changed token distribution. That is precisely the regime where decisive-token errors appear. ## Small but decisive tensors A separate category is tensors that are tiny in parameter count yet control routing of information: - **Embedding and output-head matrices** — large in raw element count for big vocabularies, but degrading them hits every token directly. - **Normalization parameters and biases** — negligible memory, and quantizing them is essentially free damage. - **Router or gate weights in a mixture-of-experts model** — with fine-grained expert layouts now standard in most frontier open models, a tiny router matrix decides which experts see each token. A rounding error there does not blur an output; it sends the token to a different expert entirely. These are routinely excluded from quantization. ## Why keeping them high is cheap The memory math is the reason mixed precision works at all. The bulk of parameters in a dense transformer sit in the feed-forward stacks, and in a mixture-of-experts model in the expert weights. Sensitive tensors — a few attention projections, the first and last blocks, routers and norms — are a small share of the total. Keeping them at 8 or 16 bits while pushing the mass to 4-bit gives up only a few percent of the memory saving and recovers most of the accuracy that uniform 4-bit gave away. A representative recipe: attention output projections plus the first and last transformer blocks stay high, the MLP stacks go to 4-bit. ## Finding them Do not guess from architecture diagrams alone; the profile is checkpoint-specific. A sensitivity sweep is the standard approach: hold everything else at high precision, quantize one block or one tensor type at a time, and measure either the end-task delta or the relative error of that layer's output against the high-precision reference. Rank the layers, keep the top offenders high, and stop where the accuracy curve flattens. Aggregate perplexity is a poor readout for this sweep; use a task metric with enough resolution to see the differences. ## The cost of mixed precision, and the 2026 direction Mixed precision is not free on the serving side. Heterogeneous layouts complicate kernels, prevent some fusion, and can leave throughput below what a uniform, hardware-native format achieves — a build that is more accurate per bit but slower per token. That is one reason the field has been moving toward uniform low-precision formats executed natively on hardware, combined with quantization-aware training to recover the accuracy that hand-tuned mixed precision used to buy. Hand-selected exclusion lists remain common in open-weight tooling and remain the right move when a uniform recipe measurably breaks your task.