skip to content

A GGUF that runs fast under llama-server crawls on the CPU and rejects long prompts through llama-cpp-python's Llama(model_path=...); which defaults explain it, and what do you set?

level: seniorimportance: should knowfreq 8%

answer

  1. the wrapper has its own defaults
  2. zero layers on the GPU
  3. a 512-token context
  4. flash attention off, no memory fitting
  5. n_gpu_layers=-1, n_ctx, flash_attn=True

basics

~10 s

llama-cpp-python 0.3.36's Llama defaults to n_gpu_layers=0, n_ctx=512 and flash_attn=False, while llama-server 0.5.0 picks GPU layers, the model's own context and flash attention automatically. Set n_gpu_layers=-1, an explicit n_ctx and flash_attn=True.

solid answer

~40 s

The binding does not inherit llama-server's defaults. In llama-cpp-python 0.3.36, `Llama(...)` defaults to `n_gpu_layers=0`, so every layer runs on the CPU even when the wheel was built with CUDA; `n_ctx=512`, so a longer prompt raises `ValueError: Requested tokens (...) exceed context window of 512` and chat replies are cut short; and `flash_attn=False`. llama-server in llama.cpp 0.5.0 instead defaults to `-ngl auto`, `-c 0` (the model's training context) and `-fa auto`, and `--fit` trims the settings you left unset to fit device memory. Set `n_gpu_layers=-1` for all layers, an explicit `n_ctx` sized to your prompts, and `flash_attn=True`. Then confirm the wheel has a GPU backend with `llama_cpp.llama_supports_gpu_offload()`, since no constructor argument can fix a CPU-only build.

code

python · 19 lines
python
import llama_cpp
from llama_cpp import Llama

# A CPU-only wheel cannot be fixed by constructor arguments.
assert llama_cpp.llama_supports_gpu_offload(), "rebuild with CMAKE_ARGS=-DGGML_CUDA=on"

llm = Llama(
    model_path="./models/your-model.gguf",
    n_gpu_layers=-1,   # default 0: everything on the CPU
    n_ctx=8192,        # default 512; 0 = training context, with no fitting
    flash_attn=True,   # default False
    verbose=True,      # print backend, buffers and chat format at load
)

reply = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Summarize this ticket: ..."}],
    max_tokens=512,
)
print(reply["choices"][0]["message"]["content"])

go deeper

for a junior

Recall that llama-cpp-python's Llama starts with n_gpu_layers=0 and n_ctx=512, and that n_gpu_layers=-1 offloads all layers.

for a middle

Explain each symptom from its default: CPU speed from zero offloaded layers, the context-window ValueError and truncated replies from n_ctx=512, and missing flash attention.

for a senior

Separate a CPU-only build from a misconfigured constructor, explain why n_ctx=0 is risky without llama-server's --fit, and make every setting explicit when moving a workload between the two.

for a principal

Set a team convention for embedded inference: pinned binding versions, explicit load parameters checked into code, and a benchmark that compares the binding with the server before a rollout.

## Same file, two sets of defaults llama-server and llama-cpp-python both run llama.cpp, but they are different programs with **different defaults**, and the gap is large in 2026. llama.cpp's own server grew automatic settings; the Python binding kept conservative ones. A GGUF that performs well under `llama-server -m your-model.gguf` can therefore be slow and fragile when you load the same file with `Llama(model_path="your-model.gguf")`. | Setting | llama-cpp-python 0.3.36 `Llama(...)` | llama-server (llama.cpp 0.5.0) | |---|---|---| | GPU layers | `n_gpu_layers=0` (none) | `-ngl auto` | | Context length | `n_ctx=512` | `-c 0`: the model's training context | | Flash attention | `flash_attn=False` | `-fa auto` | | Fitting to memory | none | `--fit on`: adjusts unset settings to device memory | | Memory mapping | `use_mmap=True`, `use_mlock=False` | `--load-mode auto` | ## The symptoms, mapped to defaults - **Slow, CPU-bound generation on a GPU box.** `n_gpu_layers=0` keeps every layer in system memory. The GPU backend may be compiled in and still hold no weights. The load log shows the model buffer in CPU memory. - **`ValueError: Requested tokens (N) exceed context window of 512`.** The prompt alone is longer than `n_ctx`. Retrieval-style prompts with a few documents pass 512 tokens quickly. - **Replies that stop mid-sentence.** `create_chat_completion` trims the generation budget to whatever is left of `n_ctx` after the prompt, so a 450-token prompt leaves room for only about 60 tokens of answer. - **Very short plain completions.** `create_completion()` and calling the object default to `max_tokens=16`, a separate default worth knowing when you move from chat to completion. - **Slower or hungrier attention than the server.** Flash attention stays off unless you ask for it, while llama-server turns it on automatically where the backend supports it. ## What to set 1. `n_gpu_layers=-1` offloads every layer (the constructor turns `-1` into "all"). Use a smaller number only when the model does not fit. 2. `n_ctx` sized to the longest prompt plus answer you expect, such as 8192. `n_ctx=0` means "use the model's training context", the same rule as the server's `-c 0`, but the binding has **no `--fit` equivalent**: on a model trained for a very long context it allocates a KV cache for all of it and may run out of memory, where llama-server would have reduced the context to fit. 3. `flash_attn=True` where the backend supports it. 4. `verbose=True` while tuning, so the load log shows the backend, the buffers per device and the chat format chosen. ## Rule out the build before the arguments Arguments cannot add a backend that was not compiled. llama-cpp-python compiles llama.cpp at `pip install` time; on Linux or Windows a plain `pip install llama-cpp-python` builds a CPU-only library, while on macOS the package's own build turns Metal on. Check: - `llama_cpp.llama_supports_gpu_offload()` returns `False` on a CPU-only build. - If so, reinstall with the backend switch, for example `CMAKE_ARGS="-DGGML_CUDA=on" pip install --upgrade --force-reinstall --no-cache-dir llama-cpp-python`, so pip does not reuse its cached CPU wheel. ## Checking what you actually got After construction, read the values back instead of trusting the call site: - `llm.n_ctx()` returns the context the binding really created, which matters when you passed `0` and the model decided. - `llm.metadata` holds the GGUF's metadata keys, including `tokenizer.chat_template` and the trained context length. - With `verbose=True`, the load log names each backend device and the size of the model buffer placed on it, so a CPU-only placement is visible before the first request. - A short timed call on a fixed prompt, run once under the binding and once under `llama-server`, tells you whether the two now behave alike. If the binding still trails the server with every setting matched, compare the llama.cpp builds themselves: the binding compiled its own copy, which can be older than the server binary on the same machine. ## Why the defaults differ The binding's constructor is a thin layer over `llama.h`'s parameter structs with keyword defaults of its own, and changing them would silently change the behaviour of code already written against it. llama-server's automatic `-ngl`, `-c 0` and `--fit` exist because the server is meant to start sensibly with nothing but a model path. Neither is wrong; the mistake is assuming one inherits the other. When you move a workload between them, write every setting down explicitly on both sides, and compare them with the same prompt before blaming the model or the hardware.

  • Why not just pass n_ctx=0 to match llama-server's -c 0?
    `n_ctx=0` does read the training context from the model, like `-c 0`, but llama-cpp-python has no equivalent of llama-server's `--fit`, which lowers settings you left unset to fit device memory. On a model trained for a very long context the binding tries to allocate the whole KV cache and can fail to load, so an explicit `n_ctx` is safer.
  • After setting n_gpu_layers=-1, generation is still CPU-speed. What next?
    Check the build: `llama_cpp.llama_supports_gpu_offload()` returning `False` means the wheel was compiled without a GPU backend. Reinstall with `CMAKE_ARGS` set to the right `-DGGML_*` switch and `--force-reinstall --no-cache-dir`, then confirm in the `verbose=True` load log that the model buffer sits on the GPU device.
  • Why do llama-cpp-python chat replies stop early with a small n_ctx even when max_tokens is large?
    The binding caps generation at what remains of `n_ctx` after the prompt: it trims `max_tokens` to `n_ctx` minus the prompt length. With the default 512, a long prompt leaves only a few dozen tokens for the answer, so raising `n_ctx` is the fix, not raising `max_tokens`.

  • Assumes Llama(...) offloads to the GPU automatically like llama-server.
  • Believes llama-cpp-python's default context is the model's training context.
  • Raises max_tokens to fix replies that stop because n_ctx is full.
  • Sets n_ctx=0 on a long-context model expecting it to fit memory.
  • Tunes constructor arguments before checking the wheel has a GPU backend.

Assumes llama-cpp-python 0.3.36's Llama defaults and llama-server from llama.cpp 0.5.0, whose -ngl auto, -c 0, -fa auto and --fit on defaults the binding does not share.