skip to content

A llama-server for four teammates runs with -np 4 -c 32768 — how much context does each request get, and why?

level: middleimportance: should knowfreq 9%

answer

  1. the total is divided, not copied
  2. explicit slot count, no shared buffer
  3. -kvu default depends on -np
  4. the /slots route shows n_ctx
  5. error: exceeds the available context size

basics

~20 s

About 8192 tokens each. With an explicit -np, llama-server keeps a separate KV stream per slot and divides -c evenly, so four slots share 32768. Use -c 131072 for 32k each, or -kvu for one shared pool.

solid answer

~40 s

Each slot gets about **8192 tokens**. `-np` sets the number of slots, the concurrent requests the server decodes together, and `-c` is the total context behind them. When you set `-np` yourself the unified KV buffer is off (`-kvu` defaults on only when the slot count is auto), so the server divides `-c` evenly across the slots. A prompt longer than its slot fails with `exceeds the available context size`, and a reply that fills the slot stops with `finish_reason` `length`, because context shift is off by default. To give each person 32k, raise `-c` to 131072 and pay four times the cache memory, or turn on `-kvu` so the slots share one pool, capped per slot with `--kv-unified-per-slot`. `GET /slots` shows the `n_ctx` each slot actually got.

code

bash · 9 lines
bash
# four slots, each with its own 32k window (pool = 4 x 32768)
llama-server -m ./models/your-model-Q4_K_M.gguf -np 4 -c 131072

# or one shared 64k pool, no single slot above 32k
llama-server -m ./models/your-model-Q4_K_M.gguf -np 4 -kvu -c 65536 \
  --kv-unified-per-slot 32768

# what each slot actually got
curl -s http://127.0.0.1:8080/slots | jq '.[] | {id, n_ctx, is_processing}'

go deeper

for a junior

Remember that -np is the number of concurrent requests and -c is the total context those requests share.

for a middle

Explain why explicit -np turns the unified buffer off, how -c is divided per slot, and what /slots and the startup log show.

for a senior

Diagnose truncated replies and context-size errors as a slot-sizing problem, and choose between per-slot windows, a shared pool and cheaper cache types.

for a principal

Weigh guaranteed per-user windows against shared capacity for a small team, and set the policy for when a second GPU or a smaller model is cheaper.

## Slots are the unit of concurrency A **slot** is llama-server's name for one request being processed. `-np` (`--parallel`) sets how many slots exist; each slot holds one conversation's tokens in the **KV cache**, the memory that stores what the model has already read. The weights are loaded once and shared by every slot, so adding slots costs cache memory, not another copy of the model. Continuous batching is on by default, so busy slots are decoded together; how that scheduling works is the serving material's subject, not this question's. When all slots are busy, the next request waits until one frees up. With `--metrics` enabled, the `llamacpp:requests_deferred` gauge on `/metrics` counts the waiting requests. ## What -c means once there are several slots `-c` is the size of the whole context pool, not one user's window. What each slot can use depends on whether the KV cache is **unified**: | Setup | KV layout | Context per slot | |---|---|---| | `-np 4 -c 32768` (explicit slots) | one stream per slot | about 32768 / 4 = 8192 | | `-np 4 -c 32768 -kvu` | one shared buffer | up to 32768, minus what the other slots hold | | `-np 4 -kvu -c 65536 --kv-unified-per-slot 32768` | one shared buffer | at most 32768, from a 65536 pool | | `-np` left unset (auto) | one shared buffer | auto resolves to 4 slots with `-kvu` on | In llama.cpp 0.5.0, `-np` defaults to `-1` (auto), which the server resolves to **four slots with the unified buffer on**. `-kvu` (`--kv-unified`) is documented as "enabled if number of slots is auto", so the moment you type `-np 4` you also switch back to per-slot streams, unless you add `-kvu` yourself. That is the trap in the question: `-np 4 -c 32768` looks like "four users, 32k each" and is really "four users, 8k each". Whatever the layout, a slot never gets more than the model's trained context: the server caps it there and logs a warning if you asked for more. ## The symptoms of a slot that is too small 1. A long prompt is refused outright with `request (N tokens) exceeds the available context size (M tokens), try increasing it`. 2. A reply that runs into the end of the slot stops early. `--context-shift` is disabled by default, so instead of discarding old tokens the server ends generation, marks the result truncated and returns `finish_reason` `length`. 3. Nothing crashes, so the first person to notice is usually the one whose summary stops mid-sentence. ## Choosing between per-slot and unified - **Per-slot streams** give every user a guaranteed window, at the price of reserving memory for slots that sit idle. - **A unified buffer** lets one long conversation borrow what idle slots are not using, but the slots compete for the same pool. `--kv-unified-per-slot N` caps any single slot at N tokens, and when it is set without `-c` the pool is sized to `-np` times N. - **Cache types** are the other lever: `-ctk` and `-ctv` default to `f16`, and lower-precision types such as `q8_0` store the same context in roughly half the memory. Whether the quality cost is acceptable belongs to the KV-cache material; here it is one more flag that decides how much `-c` your GPU can hold. On a single 24 GB GPU, the honest sequence is to decide the window each person needs, multiply by the slot count, and check that the result fits next to the weights. If it does not, either accept fewer slots, a smaller window, a shared pool or a cheaper cache type, and say which. ## Mistakes that cost a team its context - **Copying `-np` from a guide without `-c`.** With `-c` left out and `--fit` on, the server sizes the pool for every slot from the model's trained length and shrinks it only if it does not fit, so the window each person gets depends on the GPU, not on the guide. - **Raising `-np` to "serve more people".** With per-slot streams and a fixed `-c`, each extra slot makes every window smaller; a fifth person who would have waited a few seconds now shrinks everyone's context for as long as the server runs. - **Reading `n_ctx_train` from `/v1/models` as the live window.** That is the model's trained length; the slot's real limit is the `n_ctx` in `/slots`. - **Forgetting the router copies flags.** In router mode every model instance inherits the router's `-np` and `-c`, so the sizing has to be right for the largest model it loads. ## Checking what the server decided - The startup log prints `initializing, n_slots = 4, n_ctx_slot = 8192, kv_unified = 'false'` or similar, which answers the question directly. - `GET /slots` (on by default; `--no-slots` turns it off) lists every slot with its `n_ctx` and `is_processing`. With `?fail_on_no_slot=1` it returns 503 when no slot is free, which suits a load balancer's health check. - `GET /props` reports `total_slots` and the default generation settings.

  • The team now wants a second model on the same box: what does llama-server's router mode change?
    Started with no `-m`, llama-server becomes a router. It lists models from its cache, a `--models-dir` folder or a `--models-preset` INI file, loads an instance per model on demand, and routes each request by its `model` field. `--models-max` (default 4, 0 = unlimited) caps how many are loaded at once, and instances inherit the router's flags, so `-np` and `-c` apply to each model.
  • What happens to a fifth simultaneous request on a four-slot llama-server?
    It waits. Each slot runs one request at a time, so with all four busy the next request is deferred until a slot frees up, and the `llamacpp:requests_deferred` gauge on `/metrics` (with `--metrics`) shows it. Raising `-np` adds slots, but with a fixed `-c` and per-slot streams it also shrinks every slot's share.

Explicit slots are a car park with painted bays: -c is the length of the kerb and -np the number of bays, so four bays on a 32-metre kerb are eight metres each. The unified buffer removes the paint, so one long vehicle can use the whole kerb while the others are away.

  • -c sets the context each slot gets, whatever -np says.
  • Four slots load four copies of the model weights into VRAM.
  • A request arriving while every slot is busy is rejected with an error.
  • Leaving -np unset gives llama-server a single slot.
  • With -kvu, -np 4 -c 32768 reserves 32768 tokens for each slot.

Assumes llama.cpp 0.5.0, where -np defaults to auto (four slots) and -kvu is on only when the slot count is auto.