skip to content

After moving to llama.cpp 0.5.0, llama-server starts with less context than the model supports — what is --fit doing, and how do you take control?

level: seniorimportance: should knowfreq 5%

answer

  1. on by default, adjusts only what is unset
  2. context first, then GPU layers
  3. a free margin per device
  4. a context floor it will not cross
  5. explicit -c 0 versus omitting it

basics

~20 s

--fit, on by default in llama.cpp 0.5.0, shrinks the unset context, then the unset GPU layer count, until each device keeps a 1024 MiB margin. Pin -c or -ngl, raise the -fitc floor, adjust -fitt, or pass --fit off.

solid answer

~40 s

In llama.cpp 0.5.0 `--fit` is on by default. Before loading, it estimates what the weights, KV cache and compute buffers need on each device and, if that leaves less than the `-fitt` margin free (1024 MiB per device by default), it adjusts **only the arguments you did not set**. It lowers the context first, if `-c` was left out, but never below `-fitc` (default 4096); then the GPU layer count, if `-ngl` was left at auto. Leaving `-c` out is the trigger, so the server starts shorter than the model's trained length. To take control, pin `-c` (it then gives up layers instead), pin `-ngl`, raise `-fitc` to the window your app needs, change `-fitt` when another process shares the GPU, or pass `--fit off` and own any out-of-memory failure yourself.

code

bash · 12 lines
bash
# see what --fit decided (its messages appear with -v)
llama-server -m ./models/your-model-Q4_K_M.gguf -v 2>&1 | grep -E "context size|n_ctx_slot"

# keep at least 16k of context; let --fit drop GPU layers instead
llama-server -m ./models/your-model-Q4_K_M.gguf -fitc 16384

# GPU shared with another process: keep 3 GiB free
llama-server -m ./models/your-model-Q4_K_M.gguf -fitt 3072

# full control: pin both values, or switch fitting off
llama-server -m ./models/your-model-Q4_K_M.gguf -c 32768 -ngl 40
llama-server -m ./models/your-model-Q4_K_M.gguf --fit off

go deeper

for a junior

Recall that --fit is on by default in llama.cpp 0.5.0 and may shorten the context or move layers to fit memory.

for a middle

Explain the order: unset context first, down to -fitc, then unset GPU layers, keeping the -fitt margin free on each device.

for a senior

Diagnose a silently shorter context after an upgrade, and choose between pinning -c, raising -fitc, pinning -ngl and turning fitting off.

for a principal

Set a policy for which values are pinned per machine class, so a fleet's servers do not each pick a different context on their own.

## What --fit is `--fit` (`-fit`) is a llama.cpp startup step that is **on by default** in llama.cpp 0.5.0. Its flag text is precise: "whether to adjust unset arguments to fit in device memory". Before the model loads, it measures what the weights, the KV cache and the compute buffers would need on each GPU at the requested settings. If the total leaves less free memory than a target margin, it changes the settings **you did not specify** until the load fits. Anything you typed, it leaves alone. That design is why the symptom in the question appears after an upgrade: older guides assumed nothing was adjusted, so an unchanged command line on a smaller GPU now starts with a shorter context instead of failing to load. ## The order it works in 1. **Start from the defaults.** With `-c` unset the context starts at the model's trained length, multiplied by the number of sequences when several slots are in use. 2. **Check the margin.** `-fitt` (`--fit-target`) is the memory to keep free on each device, 1024 MiB by default; a single value applies to every device, or you can give a comma-separated list. 3. **Reduce the context first**, but only if `-c` was left unset, and never below `-fitc` (`--fit-ctx`, default 4096). On one device, if a shorter context is enough, it stops there. 4. **Then reduce GPU layers**, but only if `-ngl` is still at `auto`. Across several GPUs it can also choose the tensor split, again only if you did not set `-ts`. If an argument it would need to change was set by you, it logs `failed to fit params to free device memory` with the reason and loads with your values, so pinning `-ngl` on a GPU that is too small turns a graceful downgrade into a load-time failure. ## The -c 0 subtlety Leaving `-c` out and typing `-c 0` both mean "the model's trained context", but `--fit` treats them differently. An explicit `-c 0` is read as a request for the **full** context, so `--fit` will not shrink it and moves straight to the layer count. Omitting `-c` marks the context as adjustable. The first keeps the window and may put layers on the CPU; the second keeps the layers and may cut the window. ## Taking control | Goal | Flag | Effect on --fit | |---|---|---| | a fixed window | `-c 32768` | context untouched; layers may drop if `-ngl` is unset | | a minimum window | `-fitc 16384` | context may shrink, but not below 16384; layers give way first | | everything on the GPU | `-ngl all` | layers untouched; an unset context may shrink | | room for another process | `-fitt 3072` | keeps 3 GiB free per device before fitting | | no automatic changes | `--fit off` | settings used exactly as given; out-of-memory is yours | `--fit` does not support `-sm tensor`, and it does not choose the slot count; `-np` is decided before it runs. ## Choosing what to give up - **Context loss** shows up as `request (N tokens) exceeds the available context size` errors and replies cut short with `finish_reason` `length`. Long-document and retrieval workloads cannot tolerate it, so pin `-c` or raise `-fitc`. - **Layer loss** shows up as lower tokens per second in a response's `timings`, because layers left on the CPU are computed far more slowly. Short-prompt, latency-sensitive workloads prefer `-ngl all` and a smaller window. - **A shared GPU** is the case for a larger `-fitt`, so a desktop session or a second process does not push the server into an out-of-memory error later. ## Common misreadings - **"The model's context is broken."** The GGUF still reports its full trained length in `/v1/models`; only the allocated window shrank. - **"`-fitc` sets the context."** It is a floor for automatic reduction, not a size. - **"`-ngl all` is free."** It keeps every layer on the GPU only by letting an unset context shrink instead; if even the `-fitc` floor does not fit, the fit fails and the load may run out of memory. - **"`--fit off` is the safe choice."** It is the literal one: `-ngl` auto then means every layer and an unset `-c` means the trained length, whether or not they fit, so a model that loaded with automatic trimming may now fail. ## Seeing what it decided - Run with `-v`: the fitting step logs messages such as `context size reduced from ... to ...`. - The startup line `initializing, n_slots = ..., n_ctx_slot = ...` shows the window each slot ended up with. - `GET /slots` reports each slot's `n_ctx`; `GET /props` shows the default generation settings.

  • Why does typing `-c 0` behave differently from leaving -c out of a llama-server command?
    Omitting `-c` lets `--fit` treat the context as adjustable: it starts from the trained length and may cut it down to `-fitc`. Typing `-c 0` is read as asking for the full trained context, so `--fit` does not reduce it and goes to the GPU layer count instead. You keep the window but may end up with layers on the CPU.
  • How do you decide whether llama-server should give up context or GPU layers?
    Ask what the workload fails on. A chat or retrieval app whose prompts exceed the slot window gets `exceeds the available context size` errors, so protect the context with `-c` or `-fitc` and accept some layers on the CPU. A short-prompt classifier wants speed, so pin `-ngl all` and let the context shrink. Compare tokens per second in the responses' `timings` either way.

  • --fit only runs when you pass --fit on.
  • --fit lowers -c even when you typed a value.
  • -fitc sets the context size llama-server will use.
  • Typing -c 0 and leaving -c out mean the same thing to --fit.
  • --fit decides how many slots llama-server creates.

Assumes llama.cpp 0.5.0, where llama-server's --fit is on by default with -fitt 1024 MiB and -fitc 4096.