With llama.cpp, how do you turn a fine-tuned Hugging Face checkpoint into a Q4_K_M GGUF that a teammate can run in any local runtime?
answer
- two steps, not one
- convert at full precision first
- --outtype bf16, f16 or auto
- llama-quantize input output type
- keep the 16-bit GGUF as master
basics
~10 sRun convert_hf_to_gguf.py on the checkpoint directory with --outtype bf16 (or auto) to get a 16-bit GGUF, then run llama-quantize on that file with the type Q4_K_M. The converter cannot write k-quants itself.
solid answer
~40 sIt is two steps. First, `convert_hf_to_gguf.py` reads the checkpoint directory (config, tokenizer files and safetensors) and writes a GGUF at the precision you pick with `--outtype`: `f32`, `f16`, `bf16`, `q8_0`, `tq1_0`, `tq2_0` or `auto`, the default, which picks the highest-fidelity 16-bit type. No k-quant is on that list. Second, `llama-quantize model-bf16.gguf model-Q4_K_M.gguf Q4_K_M` rewrites the tensors, optionally with `--imatrix`. Before converting, merge any adapter into the base weights and make sure the tokenizer and chat template were saved with the fine-tune, because the converter embeds what it finds. Keep the 16-bit GGUF: it is the source for every other quant you will build, since quantizing an already-quantized file is refused by default. Load the result once in llama-cli or llama-server before you hand it over.
code
bash · 5 linespython convert_hf_to_gguf.py ./my-finetune \
--outtype bf16 --outfile my-finetune-bf16.gguf
./build/bin/llama-quantize \
my-finetune-bf16.gguf my-finetune-Q4_K_M.gguf Q4_K_Mgo deeper
Recall the two tools and their order: convert_hf_to_gguf.py writes a 16-bit GGUF, llama-quantize turns it into Q4_K_M.
Explain the --outtype choices, why no k-quant is among them, what metadata carries through quantization, and what the checkpoint must contain first.
Keep a 16-bit master and build every quant from it, verify the template and EOS token before publishing, and know the converter's memory and support escape hatches.
Turn the steps into a repeatable build that records the converter commit, outtype and quant type for every published file, so builds can be reproduced and compared.
## Why it takes two steps llama.cpp splits model production into two tools with different jobs: - **`convert_hf_to_gguf.py`** understands Hugging Face checkpoints. It maps each architecture's tensor names onto GGUF names, writes the hyperparameters and tokenizer into metadata, and stores the tensors at a precision close to the source. - **`llama-quantize`** understands only GGUF. It takes a high-precision GGUF and rewrites its tensors into a smaller **file type** such as `Q4_K_M`, following that type's per-tensor recipe and, optionally, an importance matrix. The quantize tool's own README describes the process the same way: convert the original model to GGUF, then quantize the converted file. ## Step 1: convert at full precision The converter takes the checkpoint directory as a positional argument (or a Hub repository id with `--remote`) and writes one GGUF. | `--outtype` | What you get | |---|---| | `auto` (default) | the highest-fidelity 16-bit float type for the source | | `bf16` / `f16` | 16-bit floats; the usual input for llama-quantize | | `f32` | 32-bit floats, twice the size, rarely needed | | `q8_0` | 8-bit, a finished deliverable rather than a quantization source | | `tq1_0` / `tq2_0` | ternary types for models trained that way | When the checkpoint is distributed in 16-bit, `auto` or omitting the flag works well. `--outfile` names the output, and a literal `{ftype}` in the name is replaced with the outtype. Other options that come up: - `--print-supported-models` lists the architectures this converter knows. An unsupported one stops with "Model ... is not supported". - `--use-temp-file` helps when conversion is killed for running out of memory. - `--split-max-size` writes shards directly, for very large models. - Install the Python requirements first. They install transformers 4, and the quantize README notes that newer models need transformers 5, which you can install with `pip install -U transformers`. ## Step 2: quantize ```bash python convert_hf_to_gguf.py ./my-finetune --outtype bf16 --outfile my-finetune-bf16.gguf ./build/bin/llama-quantize my-finetune-bf16.gguf my-finetune-Q4_K_M.gguf Q4_K_M ``` The type argument is case-insensitive, `Q4_K` is an alias for `Q4_K_M`, and an optional trailing number sets the thread count. The output copies every metadata key from the input, so the tokenizer and chat template carry through, and `general.file_type` is set to the type you asked for. `Q4_K_M` does not need an importance matrix, though giving one with `--imatrix` is common practice. `llama-quantize --dry-run` prints the size the result will have without writing it. ## Where the memory and disk go Plan the disk before you start. A run leaves three large files: the checkpoint, the 16-bit GGUF (about the same size as a 16-bit checkpoint) and the quantized output. Each tool has its own memory controls: - The converter evaluates tensors **lazily** by default and writes them as it goes. `--no-lazy` computes everything first and uses more RAM, so keep it for when lazy evaluation breaks. - `--use-temp-file` moves the converter's working data to a temporary file when the process is killed for lack of memory. It cannot be combined with splitting. - llama-quantize works on a tensor in slabs of rows. `--max-buffer-size` (default 8192 rows) can be lowered to quantize models with very large tensors on a machine with little RAM. ## What the fine-tuned checkpoint must contain The converter embeds only what is in the directory, so check it before converting: 1. **Full weights, not just an adapter.** Merge the adapter into the base model with your training stack and save the merged model. 2. **Tokenizer files and the chat template.** The converter reads `chat_template` from `tokenizer_config.json`, or from `chat_template.jinja` / `chat_template.json`. If none of them exist, the GGUF has no template and every runtime has to guess one. 3. **`config.json` with the right architecture**, so the converter picks the right tensor mapping. ## Checks before you hand it over 1. Load it once: `llama-cli -m my-finetune-Q4_K_M.gguf` or llama-server, and hold a short conversation that should end cleanly. 2. Read the metadata with `gguf-dump --no-tensors` and confirm `tokenizer.chat_template` and the EOS id. 3. Name the file with its type, as the Hub convention does (`your-model-Q4_K_M.gguf`), because people and runtimes pick builds by that label. 4. Keep `my-finetune-bf16.gguf`. A later `Q5_K_M`, `Q8_0` or `IQ4_XS` build should come from it, not from the 4-bit file. ## Why the 16-bit master matters `llama-quantize` refuses tensors that are already quantized unless you pass `--allow-requantize`, and its help warns that requantizing "can severely reduce quality compared to quantizing from 16bit or 32bit". Converting straight to `q8_0` and quantizing that down therefore trips the refusal, and forcing past it costs quality. Two steps from a 16-bit file is the path the tools are built for.
- Why not convert with --outtype q8_0 and quantize that file to Q4_K_M?llama-quantize refuses tensors that are already quantized unless you add `--allow-requantize`, and its help warns that this can severely reduce quality compared with quantizing from 16-bit or 32-bit. A `q8_0` GGUF is a fine deliverable on its own, but the source for further quants should be the `bf16` or `f16` conversion.
- Should the intermediate be f16 or bf16?The default `--outtype auto` picks the highest-fidelity 16-bit float type for the source, and the quantize README says that in the usual case of a 16-bit checkpoint, `auto` or omitting the flag works well. Choosing the type the checkpoint already uses, often `bf16`, avoids converting between the two 16-bit formats.
- What do you check if the converted model never stops generating?Look at the metadata before the weights. Run `gguf-dump --no-tensors` and check that `tokenizer.chat_template` exists and that `tokenizer.ggml.eos_token_id` is the token the template ends a turn with. A fine-tune saved without its template or with a changed EOS token produces exactly this symptom.
- convert_hf_to_gguf.py --outtype q4_k_m does it in one step
- Quantize the q8_0 GGUF down to Q4_K_M to save a step
- llama-quantize reads the safetensors directory directly
- Q4_K_M refuses to build without an importance matrix
- Ship the adapter folder; the converter merges it automatically
- The tokenizer and chat template ship as separate files next to the GGUF