How do you ship a very large llama.cpp GGUF as shards with llama-gguf-split, and how do llama.cpp tools load the shards afterwards?
answer
- split at tensor boundaries
- --split-max-size or --split-max-tensors
- -00001-of-0000N.gguf names
- load from the first shard
- --merge from the first shard
basics
~20 sRun llama-gguf-split --split-max-size 40G model.gguf model, which writes model-00001-of-0000N.gguf and the rest, then point llama.cpp tools at the 00001 shard. The loader finds the others by name in the same directory. --merge rebuilds one file.
solid answer
~40 s`llama-gguf-split` splits by default. `--split-max-size` takes a size such as `500M` or `2G`, and otherwise it cuts every `--split-max-tensors` tensors (128 by default). It splits at tensor boundaries. The first shard carries all of the metadata, and every shard records `split.no`, `split.count` and `split.tensors.count`. Output files are named `<prefix>-00001-of-0000N.gguf`. To load, pass the **first** shard to `-m`: llama.cpp derives the other names from it, and loading a later shard fails with "model must be loaded with the first split", so do not rename them. `--merge first-shard.gguf out.gguf` joins them again, and `--dry-run` prints the plan. You can also split at creation: `convert_hf_to_gguf.py` takes `--split-max-size`, and llama-quantize's `--keep-split` keeps an input's shard layout instead of writing a single file.
code
bash · 8 lines./build/bin/llama-gguf-split --split-max-size 40G \
model-Q4_K_M.gguf model-Q4_K_M
# writes model-Q4_K_M-00001-of-0000N.gguf ...
./build/bin/llama-cli -m model-Q4_K_M-00001-of-00004.gguf
./build/bin/llama-gguf-split --merge \
model-Q4_K_M-00001-of-00004.gguf model-Q4_K_M.ggufgo deeper
Recall that big GGUFs ship as -00001-of-0000N shards and that you load the first one.
Explain the size and tensor-count split modes, what metadata each shard carries, and how the loader finds the rest from the first shard's name.
Choose a shard size with headroom for the largest tensor, keep names intact, and know when to use --keep-split or a merge instead of re-splitting.
Set the publishing convention for large builds (shard size, folder layout, naming) so every runtime the team supports can load them without a merge step.
## Why split a GGUF at all A large model at a high-precision type can be hundreds of gigabytes in one file. That is awkward to upload to a file host with a per-file limit, to resume when a download breaks, or to copy onto a filesystem with a file-size cap. GGUF supports **split shards**: several files that together form one model, each of them a valid GGUF. `llama-gguf-split` makes and joins them. ## Splitting ```bash ./build/bin/llama-gguf-split --split-max-size 40G model-Q4_K_M.gguf model-Q4_K_M ``` - **`--split`** is the default operation, so it can be left out. - **`--split-max-size N(M|G)`** caps each shard's size, for example `500M` or `2G`. - **`--split-max-tensors N`** caps the number of tensors per shard. It is the mode used when no size is given, with a default of 128. - **`--no-tensor-first-split`** puts no tensors in the first shard, leaving it metadata only. - **`--dry-run`** prints the split plan and writes nothing. The second argument is an output **prefix**. The tool appends `-00001-of-0000N.gguf` and so on, the same pattern `llama_split_path` in the C API produces. Splits happen between tensors, never inside one, so a shard holding a very large tensor can exceed the size cap. ## Planning before you write Splitting a large file costs a full read and write, so check the plan first. `--dry-run` prints the number of shards and, for each one, its tensor count and total size in megabytes. You can then adjust `--split-max-size` before any disk is used. The converter's `--dry-run` does the same when you split during conversion. ## What each shard contains | Shard | Contents | |---|---| | first (`-00001-of-...`) | the full metadata (architecture, tokenizer, chat template, file type) and, by default, its share of tensors | | every shard | `split.no` (its index), `split.count` (the total) and `split.tensors.count` (tensors across all shards) | | later shards | their tensors and the split keys only | ## Loading shards in llama.cpp 1. Keep all shards in **one directory** with their original names. 2. Pass the **first** shard: `llama-cli -m model-Q4_K_M-00001-of-00004.gguf`, and the same for llama-server and the other tools. 3. The loader reads `split.count`, derives the remaining file names from the first shard's name, and loads them, checking that each shard's `split.no` matches its position. The ways this fails are specific: - Passing a later shard fails with "model must be loaded with the first split". - A renamed shard, or one missing from the directory, fails with "invalid split file name" or "failed to load GGUF split". - A shard count that does not match `split.count` fails with "invalid split count". How Ollama's Modelfile or LM Studio's downloader handles shards belongs to those runtimes. Pulling shards from the Hub with `-hf` belongs to the llama-server material. ## Merging and other tools - **`llama-gguf-split --merge model-00001-of-00004.gguf model.gguf`** joins the shards. You name only the first shard and the output, and the tool finds the rest in the same folder. - **`--delete-splits`** removes shards during the merge to save disk. Its help warns this is unsafe: a failure part-way leaves you with no complete copy. - **`convert_hf_to_gguf.py`** can shard while converting, with `--split-max-size` or `--split-max-tensors`, and its own `--dry-run` prints the plan. - **`llama-quantize`** accepts a sharded input through its first shard. By default it writes one output file; `--keep-split` writes the quantized model in the same shards as the input. ## Choosing a shard size - Pick a size below the hard limit of wherever the files will live, with headroom, since a large tensor can push a shard over. - Keep the shards' `-0000k-of-0000N` names exactly as written. That pattern is how the loader finds them. - Publish all shards of one quant in one folder. The llama-server router's models directory, for example, expects multi-shard models in a subdirectory of their own. - Once loaded, a sharded model is the same tensors as a merged one. Merge only when a tool you depend on cannot load shards.
- You quantize a sharded BF16 model with llama-quantize. Is the output sharded too?Not by default. llama-quantize reads the input through its first shard and writes one output file. Add `--keep-split` to write the quantized model in the same shards as the input, or split the output afterwards with `llama-gguf-split --split-max-size` if you need a different size.
- Why does loading fail after someone renamed the shards to model-part1.gguf and so on?The loader derives every other shard's path from the first shard's name and the `-0000k-of-0000N.gguf` pattern, so other names are rejected as invalid split file names. Rename them back to the pattern, or merge them with `llama-gguf-split --merge` into one file.
- Concatenate the shard files with cat to get a working GGUF
- Any shard can be passed to -m; the loader finds the others
- Shards can be renamed freely as long as they stay in one folder
- Splitting changes the quantization or the model's quality
- Every shard repeats the full tokenizer and chat template metadata