skip to content

llama.cpp

llama.cpp is the ggml C/C++ engine under Ollama and LM Studio, shipping convert_hf_to_gguf.py, llama-quantize and llama-server. Interviewers use it to test local inference below the wrapper.

part ofLocal LLM runtimesread it →
on this pageshow

explore

questions

18

In llama.cpp 0.5.0, what do llama-server's -m/-hf, -ngl and -c flags control, and what happens when you leave them unset?

level: juniorimportance: must knowfreq 14%

answer

  1. where the weights come from
  2. Hub repo plus an optional quant tag
  3. layers kept in VRAM, rest on CPU
  4. zero means the model's own length
  5. an on-by-default memory fitter

basics

~20 s

-m loads a local GGUF, -hf downloads one from Hugging Face (Q4_K_M by default), -ngl caps layers in VRAM and -c sets context. Unset in llama.cpp 0.5.0, -ngl is auto, -c is the trained length, and --fit may shrink both.

solid answer

~40 s

`-m path/to/model.gguf` loads a file you already have; `-hf <user>/<model>[:quant]` downloads one from Hugging Face into llama.cpp's cache, choosing Q4_K_M when no tag is given (or the repo's first file if there is no Q4_K_M), with `-hff` naming an exact file and the token read from `HF_TOKEN`. `-ngl` is the maximum number of layers stored in VRAM: a number, `auto` or `all`. Whatever does not fit runs on the CPU, the usual reason a server is slow. `-c` is the context size, and its default `0` means the length the model was trained with. In llama.cpp 0.5.0 both defaults pass through `--fit`, which is on by default: it shrinks the context, then the layer count, until each device keeps a free margin, and leaves alone any value you set yourself.

code

bash · 8 lines
bash
# local file, both values pinned: --fit will not change them
llama-server -m ./models/your-model-Q4_K_M.gguf -ngl all -c 16384

# from the Hub: Q4_K_M unless the tag names another quant
llama-server -hf ggml-org/gemma-3-4b-it-GGUF:Q4_K_M --port 8080

# same repo later, no network: use the cached copy only
llama-server -hf ggml-org/gemma-3-4b-it-GGUF:Q4_K_M --offline

go deeper

for a junior

Recall the three flags and their defaults: -m or -hf for the weights, -ngl auto for layers in VRAM, -c 0 for the model's trained context.

for a middle

Explain how -hf picks a file, why layers left on the CPU slow generation, and why a large default context costs memory before any request arrives.

for a senior

Show you know --fit adjusts only what you left unset, and that pinning -ngl or -c changes what happens when memory runs short.

for a principal

Argue when to rely on automatic fitting and when to pin values so every machine in a fleet starts the same way.

## What the three flags decide `llama-server` is the HTTP server that ships with llama.cpp. Before it can answer a single request it has to settle three things, and each one has its own flag: - **Which weights to load**: `-m` (`--model`) for a local file, or `-hf` (`--hf-repo`) to fetch one from Hugging Face. - **Where the layers run**: `-ngl` (`--n-gpu-layers`), the maximum number of the model's layers held in VRAM. The rest stay in system RAM and are computed on the CPU. - **How long the conversation can be**: `-c` (`--ctx-size`), the number of tokens of context the server allocates a KV cache for. Every one of these has a default in llama.cpp 0.5.0, and the defaults are smarter than they were in older guides. That is exactly why an interviewer asks about them: a candidate who copies `-ngl 99 -c 2048` from an old blog post gets a server that behaves differently from one started with no flags at all. ## Getting the weights: -m against -hf | Flag | What it takes | What happens | |---|---|---| | `-m` | a path to a `.gguf` file | loads the file as is; nothing is downloaded | | `-hf` | `<user>/<model>[:quant]` | downloads from the Hub into the cache; the quant tag is case-insensitive and defaults to `Q4_K_M`, falling back to the repo's first file if there is no `Q4_K_M` | | `-hff` | a file name inside the repo | overrides the quant tag and picks that exact file | | `-hft` | an access token | defaults to the `HF_TOKEN` environment variable, for gated or private repos | `-hf` also fetches a multimodal projector when the repo has one (turn that off with `--no-mmproj`). Downloads go into llama.cpp's cache, whose location the `LLAMA_CACHE` environment variable controls; `-cl` (`--cache-list`) lists what is cached, and `--offline` forces the server to use the cache and never touch the network. What a label such as `Q4_K_M` means is the quantization material's subject; here it is simply a name the flag matches. ## -ngl: how much of the model lives on the GPU `-ngl` accepts an exact number, `auto` or `all`, and the default is **`auto`**. In practice `auto` starts from every layer on the GPU and lets `--fit` back off if memory runs short. A number is a ceiling: `-ngl 20` keeps at most 20 layers in VRAM and computes the rest on the CPU, which is far slower per token. Two consequences matter: 1. Typing any explicit value, `-ngl 99` and `-ngl all` included, counts as setting it yourself, so `--fit` will not lower it. If the model and its cache then do not fit, you find out at load time. 2. Leaving it unset is usually right on a single GPU, because the server already tries to put everything there. ## -c: the context window, and the default that surprises people `-c` defaults to **`0`, meaning the context the model was trained with**, read from the GGUF's metadata. That can be very large, and the KV cache that backs it costs memory in proportion to its length (why that is so is the KV-cache material's subject). The README's quick-start line uses `-c 2048`, but that is an example, not the default. The server also splits its context into **slots** for concurrent requests; `-np` defaults to auto, which in 0.5.0 means four slots sharing one unified KV buffer, so the context you choose is a pool, not one person's window. ## What --fit does with what you left unset `--fit` is **on by default** in llama.cpp 0.5.0. Before loading, it estimates the memory the model, cache and compute buffers need on each device and, if they do not fit with a margin free (`-fitt`, default 1024 MiB per device), it adjusts **only the arguments you did not set**: - first the context, if `-c` was left out, but never below the floor `-fitc` (default 4096); - then the GPU layer count, if `-ngl` was left at `auto`. So a bare `llama-server -m model.gguf` on a small GPU may start with a shorter context than the model supports, or with some layers on the CPU, and say so in its log. Pinning `-c` and `-ngl` yourself, or passing `--fit off`, takes the decision back. ## Checking what you actually got 1. Read the startup log: the line `initializing, n_slots = ..., n_ctx_slot = ...` shows the slot count and each slot's context. 2. Query `GET /slots` for each slot's `n_ctx`, or `GET /props` for the server's defaults and the model's chat template. 3. Watch tokens per second in a response's `timings` object; a sudden drop after a config change usually means layers moved to the CPU. 4. Run with `-v` to see the fitting messages, such as how far the context was reduced. The server listens on `127.0.0.1:8080` by default, and the built-in web UI is served at the same address, so the first test is simply opening that page.

  • How do you run llama-server on a machine without internet after using -hf?
    Downloads land in llama.cpp's cache, which the `LLAMA_CACHE` environment variable relocates, and `-cl` lists its contents. Restart with the same `-hf` argument plus `--offline`, which forces the cache and blocks network access, or copy the GGUF across and load it with `-m`. A multimodal repo's projector arrives with `-hf` automatically, so copy that file too and pass it with `-mm`.
  • Why can `-ngl 99` copied from an older guide behave differently in llama.cpp 0.5.0?
    `-ngl` already defaults to `auto`, which starts with every layer on the GPU and lets `--fit` back off when memory is short. Any explicit number, 99 included, counts as your choice, so `--fit` stops lowering the layer count: it may still shrink an unset context, but if the model will not fit you find out at load time. Leave `-ngl` unset unless you mean to pin it.

  • llama-server offloads nothing to the GPU unless you pass -ngl.
  • Leaving -c unset gives llama-server a 2048-token context.
  • -hf without a quant tag downloads the full-precision file.
  • -ngl sets how many gigabytes of VRAM llama-server may use.
  • --fit overrides the -c and -ngl values you typed yourself.

Assumes llama.cpp 0.5.0, where -ngl defaults to auto, -c 0 means the trained context and --fit is on by default.

open as a page

With llama.cpp, how do you turn a fine-tuned Hugging Face checkpoint into a Q4_K_M GGUF that a teammate can run in any local runtime?

level: middleimportance: must knowfreq 13%

basics

~10 s

Run convert_hf_to_gguf.py on the checkpoint directory with --outtype bf16 (or auto) to get a 16-bit GGUF, then run llama-quantize on that file with the type Q4_K_M. The converter cannot write k-quants itself.

open as a page

For a Python batch job, when does embedding llama.cpp in-process with llama-cpp-python beat calling a llama-server, and when does it lose?

level: seniorimportance: must knowfreq 12%

basics

~20 s

In-process wins when one job owns the model: no service, port or HTTP hop, plus direct access to tokens, logits, grammars and state. A llama-server wins when many processes or concurrent requests share the model, or a crash must not kill the job.

open as a page

What do llama-cpp-python and node-llama-cpp wrap, and what does a minimal chat call look like in each?

level: juniorimportance: should knowfreq 10%

basics

~10 s

Both embed libllama, the llama.cpp C library declared in include/llama.h, inside your own process. Python builds Llama(model_path=...) and calls create_chat_completion(messages=...); Node calls getLlama(), loadModel(), createContext() and LlamaChatSession.prompt().

open as a page

What does a llama.cpp GGUF file hold besides the weights, and why can a local runtime load it without the checkpoint's config files?

level: juniorimportance: should knowfreq 16%

basics

~20 s

A GGUF file is a header of typed key-value metadata, a table describing every tensor, then the tensor data. The metadata carries the architecture, hyperparameters, tokenizer, chat template and file type, so nothing else needs to travel with it.

open as a page

When building llama.cpp for an NVIDIA, AMD, Intel or Vulkan-capable GPU, which CMake flags select the ggml backend, and how do you confirm it?

level: middleimportance: should knowfreq 6%

basics

~10 s

The backend is compiled in: -DGGML_CUDA=ON for NVIDIA, -DGGML_HIP=ON for AMD ROCm, -DGGML_SYCL=ON for Intel GPUs, -DGGML_VULKAN=ON for Vulkan; Metal is on by default on macOS. Confirm with --list-devices or llama-bench's backend column.

open as a page

In llama.cpp 0.5.0, how do you produce an importance matrix with llama-imatrix and use it when quantizing a GGUF with llama-quantize?

level: middleimportance: should knowfreq 8%

basics

~10 s

Run llama-imatrix -m model-bf16.gguf -f calibration.txt, which writes imatrix.gguf, then pass --imatrix imatrix.gguf to llama-quantize. IQ1, IQ2 and IQ3_XXS types refuse to run without one; other types only gain quality from it.

open as a page

Before sharing one llama-server with a team over the LAN, which flags and default routes do you change, and why?

level: middleimportance: should knowfreq 8%

basics

~10 s

Bind it with --host, since the default is 127.0.0.1, and add --api-key or --api-key-file, because without one there is no authentication. Then narrow --cors-origins, decide on /slots and /metrics, and never enable --tools.

open as a page

Through llama-server, why does one message get a clean reply on /v1/chat/completions but a rambling continuation on /completion?

level: middleimportance: should knowfreq 7%

basics

~20 s

/completion sends your prompt string to the model raw, so it just continues the text. /v1/chat/completions and /v1/messages first render the messages through the GGUF's chat template with Jinja, on by default in llama.cpp 0.5.0.

open as a page

A llama-server for four teammates runs with -np 4 -c 32768 — how much context does each request get, and why?

level: middleimportance: should knowfreq 9%

basics

~20 s

About 8192 tokens each. With an explicit -np, llama-server keeps a separate KV stream per slot and divides -c evenly, so four slots share 32768. Use -c 131072 for 32k each, or -kvu for one shared pool.

open as a page

A GGUF that runs fast under llama-server crawls on the CPU and rejects long prompts through llama-cpp-python's Llama(model_path=...); which defaults explain it, and what do you set?

level: seniorimportance: should knowfreq 8%

basics

~10 s

llama-cpp-python 0.3.36's Llama defaults to n_gpu_layers=0, n_ctx=512 and flash_attn=False, while llama-server 0.5.0 picks GPU layers, the model's own context and flash attention automatically. Set n_gpu_layers=-1, an explicit n_ctx and flash_attn=True.

open as a page

llama-quantize refuses with "requantizing from type ... is disabled" when you shrink a downloaded Q4_K_M GGUF to Q2_K — why, and what should you do instead?

level: seniorimportance: should knowfreq 6%

basics

~20 s

llama-quantize refuses already-quantized tensors, because rounding Q4_K_M weights again stacks a second error on the first. Quantize from a 16-bit GGUF instead, converted from the original checkpoint or published as BF16; --allow-requantize is a last resort.

open as a page

After moving to llama.cpp 0.5.0, llama-server starts with less context than the model supports — what is --fit doing, and how do you take control?

level: seniorimportance: should knowfreq 5%

basics

~20 s

--fit, on by default in llama.cpp 0.5.0, shrinks the unset context, then the unset GPU layer count, until each device keeps a 1024 MiB margin. Pin -c or -ngl, raise the -fitc floor, adjust -fitt, or pass --fit off.

open as a page

A parser reading JSON from llama-server's /v1/chat/completions started failing overnight — which request fields and server settings do you check first?

level: seniorimportance: should knowfreq 6%

basics

~10 s

Check finish_reason first: length means max_tokens or the slot's context cut the JSON off. Then check response_format: json_object without a schema allows any object, and a json_schema type needs its schema nested in json_schema.schema.

open as a page

In llama.cpp 0.5.0, an old launch script's --mlock and --no-mmap flags are rejected; what replaced them, and what does each --load-mode value do?

level: middleimportance: nice to knowfreq 4%

basics

~20 s

llama.cpp 0.5.0 replaced both with -lm / --load-mode: auto (the default, mmap unless a device cannot), none, mmap, mlock, mmap+mlock or dio. --no-mmap becomes none; --mlock becomes mmap+mlock, or mlock when mapping was off too.

open as a page

How do you ship a very large llama.cpp GGUF as shards with llama-gguf-split, and how do llama.cpp tools load the shards afterwards?

level: middleimportance: nice to knowfreq 3%

basics

~20 s

Run llama-gguf-split --split-max-size 40G model.gguf model, which writes model-00001-of-0000N.gguf and the rest, then point llama.cpp tools at the 00001 shard. The loader finds the others by name in the same directory. --merge rebuilds one file.

open as a page

Two llama.cpp builds of the same GGUF differ threefold in llama-bench; how do you read pp512 against tg128 and isolate the cause?

level: seniorimportance: nice to knowfreq 5%

basics

~20 s

pp512 is prompt-processing speed over a 512-token prompt and tg128 is generation speed over 128 tokens, both in tokens per second. Check the backend, ngl and build fields first, then rerun both builds with identical explicit flags, sweeping one parameter.

open as a page

With llama-quantize, how do you keep chosen tensors at higher precision in a Q4_K_M build, and what does the file's quant label then tell its users?

level: seniorimportance: nice to knowfreq 4%

basics

~20 s

Pass --output-tensor-type and --token-embedding-type for those two tensors and --tensor-type name=type (a regex, repeatable) for others, using ggml tensor types such as q6_K or q8_0. The label stays Q4_K_M, so only the tensor table shows the overrides.

open as a page