skip to content

With llama-quantize, how do you keep chosen tensors at higher precision in a Q4_K_M build, and what does the file's quant label then tell its users?

level: seniorimportance: nice to knowfreq 4%

answer

  1. the recipe is already a mixture
  2. output and embedding have their own flags
  3. --tensor-type name=ggml_type, repeatable
  4. tensor types, not file types
  5. general.file_type still says Q4_K_M

basics

~20 s

Pass --output-tensor-type and --token-embedding-type for those two tensors and --tensor-type name=type (a regex, repeatable) for others, using ggml tensor types such as q6_K or q8_0. The label stays Q4_K_M, so only the tensor table shows the overrides.

solid answer

~40 s

`Q4_K_M` is already a mixture: most weights become `Q4_K`, some `attn_v` and `ffn_down` tensors and usually `output.weight` get `Q6_K`, and norms stay `F32`. llama-quantize lets you change that per tensor. `--output-tensor-type` and `--token-embedding-type` set those two tensors, `--tensor-type attn_v=q6_k` matches tensor names by regex and may repeat, and `--tensor-type-file` reads a long list. The values are **ggml tensor types** (`q4_K`, `q5_K`, `q6_K`, `q8_0`, `f16`), not file types, so `Q5_K_M` is rejected as an invalid ggml type. `--pure` goes the other way and drops the mixture. The output's `general.file_type` is still the type you named, `Q4_K_M`, so the Hub filename, LM Studio's download option and Ollama's `quantization_level` all hide the overrides. Say so in the model card, and expect users to check the tensor table.

code

bash · 4 lines
bash
./build/bin/llama-quantize --imatrix imatrix.gguf \
  --output-tensor-type q8_0 --token-embedding-type q8_0 \
  --tensor-type attn_v=q6_k --tensor-type ffn_down=q6_k \
  model-bf16.gguf model-Q4_K_M-custom.gguf Q4_K_M

go deeper

for a junior

Recall that a quant label names a recipe and that llama-quantize can set some tensors to other types.

for a middle

Explain the Q4_K_M mixture and the four override options, and why their values are ggml tensor types rather than file types.

for a senior

Write and dry-run regex overrides against real tensor names, and recognise that general.file_type hides them, so verify any build by its tensor table.

for a principal

Decide whether a team publishes stock recipes or custom mixtures, and how custom builds are labelled, documented and reproduced, given that runtimes show only the base label.

## The default recipe is already a mixture A llama-quantize **file type** is a recipe, not a single type. For `Q4_K_M`, the per-tensor rules in llama.cpp give: | Tensor | Typical type in `Q4_K_M` | |---|---| | most weight matrices | `Q4_K` | | a share of `attn_v` and `ffn_down` tensors | `Q6_K` | | `output.weight` | usually `Q6_K` | | one-dimensional tensors (norms) | `F32` | The k-quant file types ending in `_S`, `_M` and `_L` share a base type and differ in which tensors they promote. Why some tensors deserve more bits is the quantization-tradeoff material. What this question covers is the knobs llama-quantize gives you on top of the recipe. ## The override options - **`--output-tensor-type TYPE`** sets the type of `output.weight`. On architectures that tie embeddings to the output, the token embeddings follow it. - **`--token-embedding-type TYPE`** sets the type of the token-embedding tensor. - **`--tensor-type PATTERN=TYPE`** quantizes tensors whose name matches the pattern to that type. It supports regex syntax and may be given many times, for example `attn_v=q6_k`, or a regex that picks out the attention tensors of chosen layers. - **`--tensor-type-file FILE`** reads a long list of `PATTERN=TYPE` entries, separated by spaces or newlines. - **`--leave-output-tensor`** keeps `output.weight` exactly as it is in the input. - **`--pure`** works the other way: it turns off the k-quant mixture so that every quantized tensor uses the same type. The names these options match are GGUF tensor names such as `blk.12.attn_v.weight` or `blk.3.ffn_down.weight`. List them with `gguf-dump` or the Hub's tensor viewer before writing patterns. ## Writing patterns that match Tensor names follow one scheme across architectures: `token_embd.weight` for the embeddings, `output.weight` for the output projection, and `blk.N.` followed by the tensor's role for each layer, such as `blk.0.attn_q.weight`, `blk.0.attn_v.weight` or `blk.0.ffn_down.weight`. Some practical rules: 1. A bare role such as `attn_v` matches that tensor in every layer, which is what the README's own examples use. 2. To pick layers, match on the `blk.N.` prefix with a regex. The README shows odd and even layer numbers getting different types. 3. Repeat `--tensor-type` for each rule, or move a long list into `--tensor-type-file`. ## Tensor types are not file types Every override takes a **ggml tensor type**, the same names the tensor table shows: `q4_K`, `q5_K`, `q6_K`, `q8_0`, `f16`, `bf16`, `iq4_xs`. Matching is case-insensitive. Writing `--tensor-type ffn_down=Q5_K_M` fails with "invalid ggml_type", because `Q5_K_M` is a file type (a recipe), not something a single tensor can be. The quantize README's own example comments say "Q5_K_M" while the commands pass `q5_k`. The command is right; the comment is loose. ## A worked build ```bash ./build/bin/llama-quantize --imatrix imatrix.gguf \ --output-tensor-type q8_0 --token-embedding-type q8_0 \ --tensor-type attn_v=q6_k --tensor-type ffn_down=q6_k \ model-bf16.gguf model-Q4_K_M-custom.gguf Q4_K_M ``` Before running it for real: 1. Add `--dry-run` to print every tensor's planned type and size, and the total, without writing a file. 2. Check that each pattern matched what you meant. llama-quantize logs "applying manual override" for every tensor a pattern changes, and the first matching pattern wins. A pattern that matches nothing produces no such line, and a loose one can match more than you intended. 3. If an override puts a tensor at a type on the importance-matrix list, such as `iq2_xs`, the imatrix rule applies to it as well. ## What the label tells users afterwards llama-quantize writes the file type you passed into **`general.file_type`**. Overrides do not change it, so the custom build above still reports `Q4_K_M`: - a Hub filename that follows the convention says `Q4_K_M`; - LM Studio's download list shows the same `Q4_K_M` option name; - Ollama's `/api/tags` reports `quantization_level` as `Q4_K_M` from that key. So a quant label tells users the **recipe the builder started from**. It does not say every tensor's type, whether an importance matrix was used, or what was overridden. Two files labelled `Q4_K_M` from different builders can differ in size and quality. The way to check is the tensor table, plus the `quantize.imatrix.*` keys, which record the imatrix file and dataset when one was used. ## Practical guidance - Override only what you can justify and measure. Each promotion costs size, and `--dry-run` shows how much. - Put the deviation in the filename or model card (for example `-custom`, plus a line listing the overrides), because the label will not show it. - Keep the exact command next to the published file so the build can be reproduced.

  • How do you see which type each tensor actually got?
    Read the tensor table, not the label. `gguf-dump` from the `gguf` Python package lists every tensor with its type, and the Hub's GGUF viewer shows the same for files in a repository. Before building, `llama-quantize --dry-run` prints each tensor's planned type and size.
  • What does --pure do compared with the overrides?
    It goes the other way. `--pure` disables the k-quant mixture, so the quantized tensors all use the file type's base type, for example `Q4_K` throughout a `Q4_K_M` build. One-dimensional tensors such as norms still stay `F32`.

  • --tensor-type ffn_down=Q5_K_M sets that tensor to Q5_K_M
  • A Q4_K_M file stores every weight tensor as Q4_K
  • Overrides change the file's quant label to match
  • Two files labelled Q4_K_M are always byte-for-byte the same build
  • --pure keeps the output tensor at a higher precision