skip to content

GGUF Format and Quantization

GGUF packs weights, tokenizer and chat template into one file; convert_hf_to_gguf.py writes it and llama-quantize with an imatrix shrinks it. Interviewers ask which quant type fits and why.

part ofLocal LLM runtimesread it →
on this pageshow

explore

questions

6

With llama.cpp, how do you turn a fine-tuned Hugging Face checkpoint into a Q4_K_M GGUF that a teammate can run in any local runtime?

level: middleimportance: must knowfreq 13%

answer

  1. two steps, not one
  2. convert at full precision first
  3. --outtype bf16, f16 or auto
  4. llama-quantize input output type
  5. keep the 16-bit GGUF as master

basics

~10 s

Run convert_hf_to_gguf.py on the checkpoint directory with --outtype bf16 (or auto) to get a 16-bit GGUF, then run llama-quantize on that file with the type Q4_K_M. The converter cannot write k-quants itself.

solid answer

~40 s

It is two steps. First, `convert_hf_to_gguf.py` reads the checkpoint directory (config, tokenizer files and safetensors) and writes a GGUF at the precision you pick with `--outtype`: `f32`, `f16`, `bf16`, `q8_0`, `tq1_0`, `tq2_0` or `auto`, the default, which picks the highest-fidelity 16-bit type. No k-quant is on that list. Second, `llama-quantize model-bf16.gguf model-Q4_K_M.gguf Q4_K_M` rewrites the tensors, optionally with `--imatrix`. Before converting, merge any adapter into the base weights and make sure the tokenizer and chat template were saved with the fine-tune, because the converter embeds what it finds. Keep the 16-bit GGUF: it is the source for every other quant you will build, since quantizing an already-quantized file is refused by default. Load the result once in llama-cli or llama-server before you hand it over.

code

bash · 5 lines
bash
python convert_hf_to_gguf.py ./my-finetune \
  --outtype bf16 --outfile my-finetune-bf16.gguf

./build/bin/llama-quantize \
  my-finetune-bf16.gguf my-finetune-Q4_K_M.gguf Q4_K_M

go deeper

for a junior

Recall the two tools and their order: convert_hf_to_gguf.py writes a 16-bit GGUF, llama-quantize turns it into Q4_K_M.

for a middle

Explain the --outtype choices, why no k-quant is among them, what metadata carries through quantization, and what the checkpoint must contain first.

for a senior

Keep a 16-bit master and build every quant from it, verify the template and EOS token before publishing, and know the converter's memory and support escape hatches.

for a principal

Turn the steps into a repeatable build that records the converter commit, outtype and quant type for every published file, so builds can be reproduced and compared.

## Why it takes two steps llama.cpp splits model production into two tools with different jobs: - **`convert_hf_to_gguf.py`** understands Hugging Face checkpoints. It maps each architecture's tensor names onto GGUF names, writes the hyperparameters and tokenizer into metadata, and stores the tensors at a precision close to the source. - **`llama-quantize`** understands only GGUF. It takes a high-precision GGUF and rewrites its tensors into a smaller **file type** such as `Q4_K_M`, following that type's per-tensor recipe and, optionally, an importance matrix. The quantize tool's own README describes the process the same way: convert the original model to GGUF, then quantize the converted file. ## Step 1: convert at full precision The converter takes the checkpoint directory as a positional argument (or a Hub repository id with `--remote`) and writes one GGUF. | `--outtype` | What you get | |---|---| | `auto` (default) | the highest-fidelity 16-bit float type for the source | | `bf16` / `f16` | 16-bit floats; the usual input for llama-quantize | | `f32` | 32-bit floats, twice the size, rarely needed | | `q8_0` | 8-bit, a finished deliverable rather than a quantization source | | `tq1_0` / `tq2_0` | ternary types for models trained that way | When the checkpoint is distributed in 16-bit, `auto` or omitting the flag works well. `--outfile` names the output, and a literal `{ftype}` in the name is replaced with the outtype. Other options that come up: - `--print-supported-models` lists the architectures this converter knows. An unsupported one stops with "Model ... is not supported". - `--use-temp-file` helps when conversion is killed for running out of memory. - `--split-max-size` writes shards directly, for very large models. - Install the Python requirements first. They install transformers 4, and the quantize README notes that newer models need transformers 5, which you can install with `pip install -U transformers`. ## Step 2: quantize ```bash python convert_hf_to_gguf.py ./my-finetune --outtype bf16 --outfile my-finetune-bf16.gguf ./build/bin/llama-quantize my-finetune-bf16.gguf my-finetune-Q4_K_M.gguf Q4_K_M ``` The type argument is case-insensitive, `Q4_K` is an alias for `Q4_K_M`, and an optional trailing number sets the thread count. The output copies every metadata key from the input, so the tokenizer and chat template carry through, and `general.file_type` is set to the type you asked for. `Q4_K_M` does not need an importance matrix, though giving one with `--imatrix` is common practice. `llama-quantize --dry-run` prints the size the result will have without writing it. ## Where the memory and disk go Plan the disk before you start. A run leaves three large files: the checkpoint, the 16-bit GGUF (about the same size as a 16-bit checkpoint) and the quantized output. Each tool has its own memory controls: - The converter evaluates tensors **lazily** by default and writes them as it goes. `--no-lazy` computes everything first and uses more RAM, so keep it for when lazy evaluation breaks. - `--use-temp-file` moves the converter's working data to a temporary file when the process is killed for lack of memory. It cannot be combined with splitting. - llama-quantize works on a tensor in slabs of rows. `--max-buffer-size` (default 8192 rows) can be lowered to quantize models with very large tensors on a machine with little RAM. ## What the fine-tuned checkpoint must contain The converter embeds only what is in the directory, so check it before converting: 1. **Full weights, not just an adapter.** Merge the adapter into the base model with your training stack and save the merged model. 2. **Tokenizer files and the chat template.** The converter reads `chat_template` from `tokenizer_config.json`, or from `chat_template.jinja` / `chat_template.json`. If none of them exist, the GGUF has no template and every runtime has to guess one. 3. **`config.json` with the right architecture**, so the converter picks the right tensor mapping. ## Checks before you hand it over 1. Load it once: `llama-cli -m my-finetune-Q4_K_M.gguf` or llama-server, and hold a short conversation that should end cleanly. 2. Read the metadata with `gguf-dump --no-tensors` and confirm `tokenizer.chat_template` and the EOS id. 3. Name the file with its type, as the Hub convention does (`your-model-Q4_K_M.gguf`), because people and runtimes pick builds by that label. 4. Keep `my-finetune-bf16.gguf`. A later `Q5_K_M`, `Q8_0` or `IQ4_XS` build should come from it, not from the 4-bit file. ## Why the 16-bit master matters `llama-quantize` refuses tensors that are already quantized unless you pass `--allow-requantize`, and its help warns that requantizing "can severely reduce quality compared to quantizing from 16bit or 32bit". Converting straight to `q8_0` and quantizing that down therefore trips the refusal, and forcing past it costs quality. Two steps from a 16-bit file is the path the tools are built for.

  • Why not convert with --outtype q8_0 and quantize that file to Q4_K_M?
    llama-quantize refuses tensors that are already quantized unless you add `--allow-requantize`, and its help warns that this can severely reduce quality compared with quantizing from 16-bit or 32-bit. A `q8_0` GGUF is a fine deliverable on its own, but the source for further quants should be the `bf16` or `f16` conversion.
  • Should the intermediate be f16 or bf16?
    The default `--outtype auto` picks the highest-fidelity 16-bit float type for the source, and the quantize README says that in the usual case of a 16-bit checkpoint, `auto` or omitting the flag works well. Choosing the type the checkpoint already uses, often `bf16`, avoids converting between the two 16-bit formats.
  • What do you check if the converted model never stops generating?
    Look at the metadata before the weights. Run `gguf-dump --no-tensors` and check that `tokenizer.chat_template` exists and that `tokenizer.ggml.eos_token_id` is the token the template ends a turn with. A fine-tune saved without its template or with a changed EOS token produces exactly this symptom.

  • convert_hf_to_gguf.py --outtype q4_k_m does it in one step
  • Quantize the q8_0 GGUF down to Q4_K_M to save a step
  • llama-quantize reads the safetensors directory directly
  • Q4_K_M refuses to build without an importance matrix
  • Ship the adapter folder; the converter merges it automatically
  • The tokenizer and chat template ship as separate files next to the GGUF
open as a page

What does a llama.cpp GGUF file hold besides the weights, and why can a local runtime load it without the checkpoint's config files?

level: juniorimportance: should knowfreq 16%

basics

~20 s

A GGUF file is a header of typed key-value metadata, a table describing every tensor, then the tensor data. The metadata carries the architecture, hyperparameters, tokenizer, chat template and file type, so nothing else needs to travel with it.

open as a page

In llama.cpp 0.5.0, how do you produce an importance matrix with llama-imatrix and use it when quantizing a GGUF with llama-quantize?

level: middleimportance: should knowfreq 8%

basics

~10 s

Run llama-imatrix -m model-bf16.gguf -f calibration.txt, which writes imatrix.gguf, then pass --imatrix imatrix.gguf to llama-quantize. IQ1, IQ2 and IQ3_XXS types refuse to run without one; other types only gain quality from it.

open as a page

llama-quantize refuses with "requantizing from type ... is disabled" when you shrink a downloaded Q4_K_M GGUF to Q2_K — why, and what should you do instead?

level: seniorimportance: should knowfreq 6%

basics

~20 s

llama-quantize refuses already-quantized tensors, because rounding Q4_K_M weights again stacks a second error on the first. Quantize from a 16-bit GGUF instead, converted from the original checkpoint or published as BF16; --allow-requantize is a last resort.

open as a page

How do you ship a very large llama.cpp GGUF as shards with llama-gguf-split, and how do llama.cpp tools load the shards afterwards?

level: middleimportance: nice to knowfreq 3%

basics

~20 s

Run llama-gguf-split --split-max-size 40G model.gguf model, which writes model-00001-of-0000N.gguf and the rest, then point llama.cpp tools at the 00001 shard. The loader finds the others by name in the same directory. --merge rebuilds one file.

open as a page

With llama-quantize, how do you keep chosen tensors at higher precision in a Q4_K_M build, and what does the file's quant label then tell its users?

level: seniorimportance: nice to knowfreq 4%

basics

~20 s

Pass --output-tensor-type and --token-embedding-type for those two tensors and --tensor-type name=type (a regex, repeatable) for others, using ggml tensor types such as q6_K or q8_0. The label stays Q4_K_M, so only the tensor table shows the overrides.

open as a page