Why can two 4-bit LLM builds differ sharply in quality at the same bit width?

## The label is a container, not a guarantee "4-bit" states how many bits encode each weight value. It says nothing about how the values were chosen, how they are scaled, what else in the pipeline runs at low precision, or whether the model ever saw low precision during training. Two builds carrying the same label routinely land several points apart on the same task suite. Treating the width as the quality specification is the most common mistake in this area. ## Axis one: scale granularity Every low-precision format pairs a small integer or float code with a shared scaling factor. The question is how many weights share one factor. A scale shared across a whole tensor must accommodate the largest value anywhere in it; a scale shared across a small block only has to accommodate that block's range. Finer sharing means outliers damage fewer neighbours. That difference is the main reason NVFP4, which scales in 16-element blocks, tends to preserve more accuracy than MXFP4, which scales in 32-element blocks, at identical nominal width. Open-weight quantization tooling exposes the same axis under different names — per-tensor versus per-channel versus per-group scales — with the same directional effect: finer is more accurate and slightly larger, because the scales themselves take space. The internals of these formats are a topic of their own; what matters for the quality question is that granularity, not width, is doing much of the work. ## Axis two: how the checkpoint reached 4 bits There are three routes, in increasing order of typical quality: 1. **Post-training conversion.** Take released high-precision weights and map them into the low-precision format, possibly using a small calibration corpus to fit the scales. Cheapest and most common. 2. **Quantization-aware training or fine-tuning.** The model is trained or fine-tuned with the low-precision behaviour simulated in the loop, so it learns weights that survive the mapping. Vendors increasingly publish these directly — Google's published int4 QAT Gemma checkpoints are the canonical public example, and they beat post-hoc conversions of the same models at equal width. 3. **Native low-precision training.** The pretraining itself runs in the low-precision format. NVIDIA reports training in NVFP4 with no measurable quality loss versus higher-precision baselines, which is what turned 4-bit from a serving compromise into a shipping format. If a vendor QAT checkpoint exists at the width you want, it is almost always the better starting point than converting the high-precision release yourself. ## Axis three: what else is quantized "4-bit model" in casual usage often means weight-only 4-bit with activations still in 16-bit and the KV cache untouched. A build that also quantizes activations, or stores the KV cache at low precision, is a materially different system with a different quality profile — activation quantization exposes the outlier problem far more sharply than weight quantization does. Low-precision KV cache is now common because it halves that footprint and lets you hold far more concurrent context, but it is a separate quality decision from the weights. When comparing two builds, establish scope before comparing numbers. ## Axis four: hardware-native execution A format that the hardware executes natively behaves differently in practice from one emulated by dequantizing back to a wider type before each matmul. Blackwell-class hardware runs MXFP4 and NVFP4 natively; elsewhere a 4-bit checkpoint may be stored small and computed wide. That does not change the stored-quality story much, but it changes the throughput half of the trade entirely, and it changes which formats are worth choosing on a given fleet. ## The mid-2026 picture Four-bit is a mainstream production format, not an emergency measure. MXFP4 is an open standard with frontier models shipped in it; NVFP4 pairs finer blocks with hardware support and is used for training as well as serving; GPTQ, AWQ, SmoothQuant and GGUF k-quants remain the workhorses of the open-weight ecosystem. The live interview question is no longer "is 4-bit acceptable" but "which 4-bit, from which route, quantizing what". ## What to do with all this Never accept a width as a specification. Ask which format and granularity, whether the checkpoint is QAT or converted, whether activations and KV cache are included, and whether the target hardware runs it natively. Then measure on your own task — because the interaction between all four axes and your particular workload is not predictable from the labels.