In llama.cpp 0.5.0, how do you produce an importance matrix with llama-imatrix and use it when quantizing a GGUF with llama-quantize?
answer
- run the model over sample text
- -m model.gguf -f calibration.txt
- writes imatrix.gguf by default
- llama-quantize --imatrix imatrix.gguf
- IQ1, IQ2 and IQ3_XXS refuse without one
basics
~10 sRun llama-imatrix -m model-bf16.gguf -f calibration.txt, which writes imatrix.gguf, then pass --imatrix imatrix.gguf to llama-quantize. IQ1, IQ2 and IQ3_XXS types refuse to run without one; other types only gain quality from it.
solid answer
~40 s`llama-imatrix` runs the high-precision GGUF over a calibration text file and records, per tensor, how strongly its inputs were activated; `-m` and `-f` are mandatory, `-o` defaults to `imatrix.gguf`, and since recent releases the file is written in GGUF format (`--output-format dat` gives the legacy one). Offload with `-ngl` to make it fast. `llama-quantize --imatrix imatrix.gguf in.gguf out.gguf IQ2_XS` then uses it to decide where precision matters. In llama.cpp 0.5.0 any tensor bound for `IQ1_S`, `IQ1_M`, `IQ2_XXS`, `IQ2_XS`, `IQ2_S` or `IQ3_XXS`, or for `Q2_K` inside a `Q2_K_S` file, makes the tool stop with "this quantization requires an importance matrix!". K-quants such as `Q4_K_M` do not need one but still use it. The output records which imatrix and dataset were used in `quantize.imatrix.*` metadata.
code
bash · 5 lines./build/bin/llama-imatrix -m my-model-bf16.gguf \
-f calibration.txt -ngl 99 -o imatrix.gguf
./build/bin/llama-quantize --imatrix imatrix.gguf \
my-model-bf16.gguf my-model-IQ2_XS.gguf IQ2_XSgo deeper
Recall the two commands: llama-imatrix with a model and a text file, then llama-quantize with --imatrix.
Explain what llama-imatrix records, its default output, which tensor types refuse to run without it in llama.cpp 0.5.0, and how a file type trips the check through its mixture.
Compute one imatrix per model from its 16-bit GGUF, reuse it across that model's quants, regenerate it for a fine-tune, and use --dry-run to find refusals before a long job.
Decide which calibration data and which imatrix builds a team publishes, record them in the quantize.imatrix metadata, and make it auditable which build used which data.
## What an importance matrix is An **importance matrix** (imatrix) is a file of per-tensor statistics collected by running a model over sample text. For each weight tensor, `llama-imatrix` adds up the squared activations that flowed into it, which tells llama-quantize which input channels the model actually leans on. llama-quantize then spends its rounding error where it hurts least. Why this works is the subject of the quantization-method material; what matters here is producing the file and feeding it in. ## Step 1: compute it ```bash ./build/bin/llama-imatrix -m my-model-bf16.gguf -f calibration.txt -ngl 99 -o imatrix.gguf ``` - **`-m`** and **`-f`** are mandatory: the model and a text file of calibration data. - **`-o`** defaults to `imatrix.gguf`. In llama.cpp 0.5.0 the output format defaults to **GGUF**. `--output-format dat`, or an extension other than `.gguf`, writes the legacy binary format that older guides show. - **`-ngl`** offloads layers to the GPU; the README recommends it because the run is a full forward pass over the text. - **`--chunks`** limits how many chunks are processed (default all), and **`--chunk`** skips the first N. - **`--in-file`** loads existing imatrix files and combines them, which lets you merge runs over several datasets or convert legacy files to GGUF. - **`--process-output`** also collects data for `output.weight`; it is off by default because using the imatrix on that tensor is usually worse. - **`--show-statistics`** prints per-tensor and per-layer statistics for an existing file instead of running inference. Run it on the **16-bit GGUF** you are about to quantize. The calibration text should resemble what the model will be asked to do; how to choose it belongs to the quantization-method material. ## Step 2: quantize with it ```bash ./build/bin/llama-quantize --imatrix imatrix.gguf my-model-bf16.gguf my-model-IQ2_XS.gguf IQ2_XS ``` `--include-weights` and `--exclude-weights` limit which tensors use the matrix; the two cannot be mixed. llama-quantize writes `quantize.imatrix.file`, `quantize.imatrix.dataset`, `quantize.imatrix.entries_count` and `quantize.imatrix.chunks_count` into the output's metadata. Anyone reading the file can therefore tell an imatrix build from a plain one. ## Which types refuse to run without one The check runs per tensor, on the type each tensor is about to receive: | Target tensor type | Without `--imatrix` | |---|---| | `IQ1_S`, `IQ1_M`, `IQ2_XXS`, `IQ2_XS`, `IQ2_S`, `IQ3_XXS` | refused: "this quantization requires an importance matrix!" | | `Q2_K` inside a `Q2_K_S` file | refused, the one k-quant exception | | other k-quants (`Q3_K` to `Q6_K`), `IQ3_S`, `IQ4_XS`, `IQ4_NL`, `Q8_0` | allowed, but an imatrix still improves the result | Because the rule follows tensor types, a file type can trip it through its mixture: `IQ2_M` stores most weights as `IQ2_S`, and `IQ3_XS` puts some attention and feed-forward tensors at `IQ3_XXS`. The token-embedding and output tensors are exempt. To find out in advance, `llama-quantize --dry-run` prints the planned size of every tensor. When the run would need an imatrix, it ends with "dry run completed successfully, but actually completing this quantization will require an imatrix!". ## Long runs and combined files On a large model or a long calibration file, llama-imatrix runs for a long time, and it has options for that: - **`--output-frequency`** (default 10) saves the result so far every N chunks, so an interrupted run leaves a usable file. - **`--save-frequency`** (default 0, never) also keeps numbered snapshots along the way. - **`--parse-special`** parses special tokens such as `<|im_start|>` in the calibration text, which matters when the text is formatted as chat turns. - **`--no-ppl`** skips the perplexity it otherwise reports per chunk, which saves time when you only want the matrix. `--in-file` accepts several files, so separate runs over different text can be merged into one matrix without processing the text again. ## Errors you will meet 1. **"this quantization requires an importance matrix!"** names the offending tensor and its target type. Compute an imatrix and pass `--imatrix`. 2. **"imatrix size ... is different from tensor size ..."** means the imatrix came from a model with different tensor shapes. Compute it from the model you are quantizing. 3. **"missing imatrix metadata"** means the file is not a usable GGUF imatrix; regenerate it, or convert a legacy `.dat` file with `--in-file`. ## Practical defaults - Make one imatrix per model, from its 16-bit GGUF, and reuse it for every quant you build of that model. - Use it for k-quants too: the quantize README says a suitable imatrix minimises the accuracy loss. - For a fine-tune, compute a new one. The shapes match the base model, but the activations the imatrix records are the fine-tune's.
- Does a Q4_K_M build need an importance matrix?No. In llama.cpp 0.5.0 k-quants do not require one; the only k-quant exception is `Q2_K` tensors inside a `Q2_K_S` file. llama-quantize still uses an imatrix for k-quants when you pass `--imatrix`, and the quantize README says a suitable one minimises accuracy loss, so many published Q4_K_M builds include one.
- Can you reuse one imatrix for a different model?Only if tensor names and shapes match, and even then you should not. llama-quantize stops with "imatrix size ... is different from tensor size" when shapes differ, and only logs "did not find weights" for tensors whose names are missing. A fine-tune shares its base model's shapes, so the file loads, but it describes the base model's activations rather than the fine-tune's. Compute a new one from the model you are quantizing.
- How do you know before a long run whether a type will need an imatrix?Run `llama-quantize --dry-run` with the input and the target type. It prints each tensor's planned type and size without writing anything. If any tensor needs an importance matrix, it ends with a warning that the real run will require one.
An importance matrix is like a map of which streets a city's traffic actually used during a sample week. Road crews (llama-quantize) keep the busy streets smooth and patch the quiet ones roughly. A map drawn from another city, or from a week unlike real traffic, sends the crews to the wrong streets.
- The importance matrix is computed from the weights alone, without text
- Every quant type refuses to run without an imatrix
- IQ4_XS refuses to quantize without an imatrix
- Pass the calibration text straight to llama-quantize
- Any imatrix file works for any model
- llama-imatrix still writes only the legacy .dat format
Assumes llama.cpp 0.5.0; llama-imatrix writes GGUF-format imatrix files by default there, while older guides show the legacy .dat format.