skip to content

What does a llama.cpp GGUF file hold besides the weights, and why can a local runtime load it without the checkpoint's config files?

level: juniorimportance: should knowfreq 16%

answer

  1. one file, header first
  2. typed key-value pairs before tensors
  3. general.architecture tells the loader what to build
  4. tokenizer.ggml.* and tokenizer.chat_template
  5. Hub viewer or gguf-dump reads it

basics

~20 s

A GGUF file is a header of typed key-value metadata, a table describing every tensor, then the tensor data. The metadata carries the architecture, hyperparameters, tokenizer, chat template and file type, so nothing else needs to travel with it.

solid answer

~40 s

GGUF is llama.cpp's single-file model format: a header of typed key-value pairs, a table that names every tensor with its shape, ggml type and offset, and the aligned tensor data. The keys stand in for the folder of `config.json`, tokenizer files and weights that a Hugging Face checkpoint ships. `general.architecture` tells the loader which graph to build, keys such as `{arch}.context_length` and `{arch}.block_count` give the hyperparameters, `tokenizer.ggml.model`, `tokenizer.ggml.tokens` and the BOS/EOS ids rebuild the tokenizer, `tokenizer.chat_template` carries the Jinja chat template, and `general.file_type` records the file type it was written as, such as `BF16` from the converter or the `Q4_K_M` from llama-quantize that a Hub filename or an Ollama listing shows. When a model loads but chats badly, read the metadata first, with the Hub's GGUF viewer or `gguf-dump` from the `gguf` Python package.

go deeper

for a junior

Recall the three parts of the file: metadata keys, the tensor table, the data. Name the architecture, tokenizer, chat template and file type keys.

for a middle

Explain where each key comes from during conversion, why one file replaces a checkpoint folder, and how to read the metadata with gguf-dump or the Hub viewer.

for a senior

Diagnose a model that loads but chats badly by checking tokenizer.chat_template and the EOS id first, and know the tools that repair metadata without reconverting.

for a principal

Treat the metadata as the contract between whoever builds GGUF files and every runtime that loads them, and set checks that catch a missing template before a file is published.

## What a GGUF file is **GGUF** is the file format of **ggml**, the tensor library under llama.cpp. Every runtime in the llama.cpp family reads it: llama.cpp's own tools, the desktop apps and daemons built on the engine, and the language bindings that embed it. A GGUF file has three parts, in this order: 1. A **header** of typed **key-value metadata**: strings, integers, floats, booleans and arrays, each stored under a dotted key. 2. A **tensor table** listing every tensor by name (such as `blk.0.attn_q.weight`), with its shape, its **ggml type** (`F32`, `BF16`, `Q4_K`, `Q6_K`, `IQ2_XS` and so on) and its offset in the file. 3. The **tensor data**, padded to the alignment recorded in `general.alignment` (32 bytes by default), so a runtime can map it from disk instead of parsing it. A Hugging Face checkpoint, by contrast, is a folder: `config.json` for the hyperparameters, tokenizer files, and safetensors files that hold only tensors. GGUF folds all of that into one file. That is why a teammate can run what you hand them without the original repository. ## The metadata keys a runtime reads | Key | What it tells the runtime | |---|---| | `general.architecture` | which model graph to build, e.g. `llama` or `qwen3`; it also prefixes the hyperparameter keys | | `{arch}.context_length`, `{arch}.block_count`, `{arch}.embedding_length` | the hyperparameters `config.json` used to hold | | `tokenizer.ggml.model`, `tokenizer.ggml.tokens`, `tokenizer.ggml.pre` | the tokenizer type, its vocabulary and its pre-tokenizer | | `tokenizer.ggml.bos_token_id`, `tokenizer.ggml.eos_token_id` | the special tokens that start and end a sequence | | `tokenizer.chat_template` | the Jinja chat template copied from the checkpoint | | `general.file_type` | the llama-quantize file type, such as `Q4_K_M` or `BF16` | | `general.quantization_version` | the version of the quantization formats the tensors use | | `general.name`, `general.basename`, `general.size_label` | descriptive fields for people, not for the loader | A few points matter in practice: - **The tokenizer is inside the file.** A GGUF needs no `tokenizer.json` beside it. - **The chat template is inside the file, if the checkpoint had one.** `convert_hf_to_gguf.py` takes it from `tokenizer_config.json`, or from a `chat_template.jinja` or `chat_template.json` file next to it. llama-server uses the embedded template unless you pass `--chat-template` or `--chat-template-file`, and Ollama picks its template from `tokenizer.chat_template` when it pulls a GGUF from the Hub. - **`general.file_type` names a recipe, not every tensor's type.** A `Q4_K_M` file stores most weights as `Q4_K`, puts some tensors at `Q6_K`, and keeps one-dimensional tensors such as norms in `F32`. The tensor table shows what each tensor really is. ## Why one file is enough - The loader learns the architecture and every hyperparameter from the header, so it needs no code or config from the source repository. - The tokenizer and its special tokens rebuild from the metadata, so text goes in and comes out without a second library. - The quant label every runtime shows comes from `general.file_type`. Ollama's `/api/tags` reports it as `quantization_level`, and Hub filenames repeat it by convention. - Split shards are still GGUF files. The first shard carries the full metadata and `split.count` tells the loader how many shards to find. ## Reading a file you were handed 1. Open the file on the Hub: the GGUF viewer shows the metadata and every tensor's name, shape and type without a download. 2. Locally, run `pip install gguf`, then `gguf-dump your-model.gguf`. It prints the key-value pairs and then the tensor table; `--no-tensors` stops after the metadata. 3. Check that `general.architecture` is one your build supports, that `tokenizer.chat_template` exists, and that `general.file_type` matches the label you were promised. 4. To change a simple value without reconverting, `gguf-set-metadata` edits it in place, and `gguf-new-metadata` writes a copy with keys added, changed or removed. ## When the metadata is the problem The commonest failure is a file converted from a fine-tuned checkpoint whose save dropped the chat template. The weights are fine, but the runtime has no `tokenizer.chat_template`, so it falls back to a generic format and the model answers in the wrong style or never stops. The fix is to restore the template file in the checkpoint and convert again, or to supply the template at run time. A wrong `tokenizer.ggml.eos_token_id` gives the same never-stops symptom. The weights are rarely the problem; the metadata usually is.

  • Where does convert_hf_to_gguf.py find the chat template it embeds?
    It reads `chat_template` from `tokenizer_config.json`, and if that is missing it falls back to a `chat_template.jinja` or `chat_template.json` file in the checkpoint directory. If none exists, the GGUF has no `tokenizer.chat_template`, and every runtime must guess a format or be given one, for example with llama-server's `--chat-template-file`.
  • Can you fix one wrong metadata value without converting again?
    Yes. The `gguf` Python package ships `gguf-set-metadata`, which changes a simple value in place, and `gguf-new-metadata`, which writes a copy with keys added, modified or removed. When you are quantizing anyway, llama-quantize's `--override-kv KEY=TYPE:VALUE` sets a key in the output file.

  • A GGUF still needs tokenizer.json and config.json beside it
  • GGUF is a format only Ollama can load
  • general.file_type means every tensor in the file uses that one type
  • The chat template always comes from the runtime, never from the file
  • Reading a GGUF's metadata requires loading the whole model into memory