llama-quantize refuses with "requantizing from type ... is disabled" when you shrink a downloaded Q4_K_M GGUF to Q2_K — why, and what should you do instead?
answer
- the input is already lossy
- refused by default on purpose
- --allow-requantize overrides it
- "can severely reduce quality"
- go back to the 16-bit source
basics
~20 sllama-quantize refuses already-quantized tensors, because rounding Q4_K_M weights again stacks a second error on the first. Quantize from a 16-bit GGUF instead, converted from the original checkpoint or published as BF16; --allow-requantize is a last resort.
solid answer
~40 sThe check is deliberate: when a tensor in the input is already a quantized type and `--allow-requantize` is not set, llama-quantize throws "requantizing from type ... is disabled". The option's own help says it "can severely reduce quality compared to quantizing from 16bit or 32bit", because the Q4_K_M weights already carry rounding error and a second quantization rounds those rounded values again. Expanding the file back to F16 first does not help, since the lost precision is gone. The right fix is a 16-bit source: convert the original checkpoint with `convert_hf_to_gguf.py --outtype bf16`, or use a BF16 or F16 GGUF from the same publisher, then quantize to `Q2_K`, ideally with an imatrix. If only the quantized file exists, `--allow-requantize --leave-output-tensor` is the least-bad option, and you should measure the result before shipping it.
go deeper
Recall that llama-quantize refuses already-quantized input by default and that the fix is to start from a 16-bit GGUF.
Explain why rounding quantized weights again stacks error, why dequantizing to F16 does not undo it, and what --allow-requantize and --leave-output-tensor do.
Tell this refusal apart from the importance-matrix one, find or rebuild a 16-bit source, and, if forced to requantize, measure the result before anyone relies on it.
Make keeping a 16-bit master per model a team rule, so every published quant can be rebuilt from clean weights and a requantized file is never shipped unmarked.
## What the refusal is `llama-quantize` processes a GGUF tensor by tensor. Before it rewrites a tensor, it checks whether the input tensor is already a quantized ggml type, such as `Q4_K`, `Q6_K` or `Q8_0`. If it is and `--allow-requantize` was not given, the run stops: ``` requantizing from type q4_K is disabled ``` The type name in the message depends on which tensor it reached first. A `Q4_K_M` file contains several quantized types, so you may see `q6_K` instead. One-dimensional tensors such as norms stay in `F32` and never trip the check. Do not confuse this with the importance-matrix refusal. A `Q2_K` file type does not require an imatrix; only `Q2_K` inside a `Q2_K_S` file does. So the error in this scenario is about the input, not the target. ## Why requantizing loses quality The option's help text gives the reason: requantizing "can severely reduce quality compared to quantizing from 16bit or 32bit". Concretely: - A `Q4_K_M` file already holds weights rounded onto a 4-bit grid; the original 16-bit values are gone. - Quantizing to `Q2_K` rounds those **already-rounded** values onto a coarser grid, so the two errors stack. - An importance matrix cannot repair this. llama-quantize uses it to choose where to put rounding error, but it starts from the damaged weights. - **Dequantizing first does not help either.** Expanding `Q4_K_M` tensors to `F16` produces 16-bit numbers that hold exactly the 4-bit values. llama-quantize treats a quantized input as requantizing whatever the target is, so even that expansion needs the flag. How much quality is lost is the quantization-tradeoff material. The operational point is that the tool refuses by default, and the warning is literal. ## What to do instead 1. **Find a 16-bit source.** Many GGUF repositories publish a `BF16` or `F16` file next to the quants. Otherwise, download the original checkpoint and convert it: ```bash python convert_hf_to_gguf.py ./original-checkpoint --outtype bf16 --outfile model-bf16.gguf ``` 2. **Build an importance matrix** from that file with `llama-imatrix`, which helps a lot at 2 bits. 3. **Quantize from the 16-bit file:** ```bash ./build/bin/llama-quantize --imatrix imatrix.gguf model-bf16.gguf model-Q2_K.gguf Q2_K ``` 4. **Consider an i-quant at a similar size.** At 2 bits, `IQ2_XS` or `IQ2_M` with an imatrix are the other choices, and they are refused without one. ## When forcing it is acceptable Sometimes the quantized file is all there is: the 16-bit weights were never published, or the build machine cannot hold them. Then: | Option | Effect | |---|---| | `--allow-requantize` | lets llama-quantize rewrite tensors that are already quantized | | `--leave-output-tensor` | keeps `output.weight` as it is in the input; the README says it "may also increase quality, especially when requantizing" | | `--dry-run` | shows the resulting size per tensor before you commit to the run | Treat the result as experimental. Compare it with the source file using `llama-perplexity` with `--kl-divergence-base` and `--kl-divergence`, or with your own task checks. Label it clearly, so nobody mistakes it for a build made from 16-bit weights. ## Recognising a requantized file Nothing in a GGUF marks it as requantized. llama-quantize copies the input's metadata and sets `general.file_type` to the new type, so a `Q2_K` made from a `Q4_K_M` reports `Q2_K` like any other. The clues are indirect: - the publisher offers no 16-bit file and no original checkpoint; - the model card, or the filename you were given, says what it was built from; - quality is clearly worse than other builds of the same size. That is why the build command and its source belong in the model card of anything you publish. ## How to keep it from happening - Keep the 16-bit GGUF you converted as the **master** for every model you publish, and build every quant from it. - Do not use a `q8_0` conversion as a quantization source. It is already quantized, so the same refusal applies. - When someone asks you for a smaller build of a file they downloaded, ask for the original checkpoint first.
- Does converting the Q4_K_M file back to F16 first avoid the problem?No. The F16 tensors would hold exactly the 4-bit values, so the precision is still gone. llama-quantize also treats any quantized input as requantizing, so even writing F16 from the Q4_K_M file needs `--allow-requantize`. Only the original checkpoint, or a 16-bit GGUF made from it, gives a clean source.
- Why is a q8_0 conversion a poor quantization master?Q8_0 is a quantized type, so llama-quantize refuses to quantize from it without `--allow-requantize`, and the tool's own warning about quality applies. Q8_0 is a good file to ship, but keep the `bf16` or `f16` conversion as the source for other quants.
- If you must requantize, how do you limit and check the damage?Add `--leave-output-tensor` so `output.weight` is not rounded a second time; the README says this may increase quality, especially when requantizing. Then measure the result against the source file, for example with `llama-perplexity` using `--kl-divergence-base` and `--kl-divergence`, or with your own task checks, before anyone relies on it.
- --allow-requantize is harmless; it only skips a safety prompt
- Converting the Q4_K_M back to F16 restores the lost precision
- An importance matrix makes requantizing as good as starting from bf16
- The error means Q2_K needs an importance matrix
- A Q8_0 GGUF is an unquantized source like bf16