skip to content

After raising OLLAMA_NUM_PARALLEL from 1 to 4 on a shared Ollama box, `ollama ps` shows a 14B model at 41%/59% CPU/GPU — why, and what do you change?

level: seniorimportance: should knowfreq 9%

answer

  1. the weights did not change
  2. one context per parallel slot
  3. num_ctx times OLLAMA_NUM_PARALLEL
  4. fewer layers fit in VRAM
  5. shrink context or quantize cache

basics

~10 s

Ollama loads a model with num_ctx times OLLAMA_NUM_PARALLEL tokens of context, so four slots roughly quadruple context memory and push layers off the GPU. Lower num_ctx, set OLLAMA_KV_CACHE_TYPE=q8_0, or reduce parallelism.

solid answer

~40 s

`OLLAMA_NUM_PARALLEL` gives each loaded model that many request slots, and Ollama allocates one context per slot: the model loads with `num_ctx × OLLAMA_NUM_PARALLEL` tokens of context. On a 24 GiB GPU, Ollama 0.35.0 defaults to a 32k context, so four slots reserve room for 128k tokens. The weights did not grow, but with `num_gpu` unset Ollama fits as many layers as remain room for and leaves the rest in system RAM — hence the split. Size the context to what clients send (`OLLAMA_CONTEXT_LENGTH=8192` makes four slots cost what one 32k slot did), switch the cache to `q8_0`, drop to two slots, or stop other loaded models. Forcing `num_gpu` higher does not create VRAM. Then confirm `100% GPU` in `ollama ps`.

code

bash · 12 lines
bash
# Same drop-in that `systemctl edit ollama.service` writes
sudo mkdir -p /etc/systemd/system/ollama.service.d
sudo tee /etc/systemd/system/ollama.service.d/override.conf <<'EOF'
[Service]
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_CONTEXT_LENGTH=8192"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"
EOF
sudo systemctl daemon-reload && sudo systemctl restart ollama

# Preload once, then read PROCESSOR and CONTEXT
ollama run your-model "" && ollama ps

go deeper

for a junior

Remember that OLLAMA_NUM_PARALLEL lets a model serve several requests at once and that each slot needs its own context memory.

for a middle

Explain that the model loads with num_ctx times the slot count, and how that leaves fewer layers on the GPU when num_gpu is automatic.

for a senior

Work the scenario end to end: compute the total context, pick between a smaller num_ctx, a q8_0 cache and fewer slots, and verify 100% GPU under concurrent load.

for a principal

Decide what a shared box promises its users — concurrency, context length or model size — since one GPU cannot maximise all three, and write that choice into the server's configuration.

## The scenario A team shares one Ollama server with a single 24 GiB GPU. A 14B model ran at `100% GPU` while one person used it. To let four people work at once, someone set `OLLAMA_NUM_PARALLEL=4` and restarted the server. Now `ollama ps` shows the model at `41%/59% CPU/GPU` and generation has slowed to a few tokens per second. Nothing about the model changed — so where did the memory go? ## Why parallelism multiplies the context `OLLAMA_NUM_PARALLEL` (default **1**) is the maximum number of requests **each loaded model** processes at the same time. Ollama implements it by loading the model with one context per parallel slot: the model's runner is started with a total context of `num_ctx × OLLAMA_NUM_PARALLEL`. The FAQ's own example is a 2K context with 4 parallel requests becoming an 8K context with the extra memory that implies. On a 24 GiB GPU, Ollama 0.35.0's VRAM-based default context is **32768** tokens. With four slots the model is loaded with room for **131072** tokens of context. The weights are the same size as before; what grew is the memory reserved for the context. Why that cache grows linearly with tokens is foundation material — here it is enough that four slots cost roughly four times the context memory of one. ## Why that becomes a CPU/GPU split When `num_gpu` is left unset (its internal default is -1, meaning "decide automatically"), Ollama places as many of the model's layers on the GPU as fit after the context is accounted for, and leaves the rest in system RAM. More context memory means fewer layers fit, so some layers drop to the CPU — and `ollama ps` reports the share of the loaded model outside VRAM. Because every token passes through every layer, the CPU-side layers set the speed. ## What to change, in order 1. **Size the context to the workload.** If the team's prompts fit in 8k, set `OLLAMA_CONTEXT_LENGTH=8192` on the server. Four slots of 8k cost the same as one slot of 32k, and the model returns to `100% GPU`. 2. **Quantize the context cache.** `OLLAMA_KV_CACHE_TYPE=q8_0` stores the cache in roughly half the memory of the default `f16`. It needs flash attention to be in effect and applies to every model on the server. 3. **Lower the parallelism.** Two slots may serve four people well enough, because a request that finds every slot busy waits in the queue rather than failing. 4. **Free VRAM held elsewhere.** Every other row in `ollama ps` is holding memory too; `ollama stop` unloads one, and `OLLAMA_MAX_LOADED_MODELS` caps how many stay loaded. 5. **Use a smaller tag** if the context really must stay large. Do not reach for `num_gpu` to force all layers onto the GPU: it is a **layer count**, not a VRAM allocation, and asking for more layers than fit risks an out-of-memory failure at load rather than a faster model. `num_gpu` is useful the other way — `0` forces CPU only, and a smaller number deliberately leaves room for something else. ## Edge cases worth knowing | Situation | What Ollama does | |---|---| | Embedding models | always loaded with one slot, whatever `OLLAMA_NUM_PARALLEL` says | | Some architectures | forced back to one slot, with a warning in the server log | | All slots busy | further requests queue, up to `OLLAMA_MAX_QUEUE` (default 512); beyond that the server answers 503 | | Two GPUs, model fits on one | loaded on a single GPU; spread across all only when it does not fit (or when `OLLAMA_SCHED_SPREAD` is set) | ## How to verify After each change, restart the server, send one request and run `ollama ps`. The CONTEXT column shows the context per request; PROCESSOR should read `100% GPU`. Then load-test with four concurrent clients and watch that the reading holds. Two conclusions to avoid while doing this: - **"The GPU driver broke."** A split that appears right after a parallelism or context change is a memory budget problem; a GPU Ollama failed to detect shows up as `100% CPU`, not a partial split. - **"The model got bigger."** The weights are unchanged; SIZE grew because the context reserved for four slots grew. Comparing SIZE before and after the change makes this visible in one glance. The server log is the other witness: it records the VRAM-based default context at startup and warns when an architecture is forced back to one slot.

  • What happens to the fifth concurrent user when OLLAMA_NUM_PARALLEL is 4?
    The request waits in Ollama's queue until a slot frees up. The queue holds up to `OLLAMA_MAX_QUEUE` requests, 512 by default; beyond that the server rejects new requests with a 503 saying it is busy. Lower parallelism therefore trades latency for fit, not failures, until the queue fills.
  • The box gains a second 24 GiB GPU. Does Ollama spread the 14B model across both?
    Not if it fits on one. Ollama loads a model on a single GPU when it fits entirely there, which avoids traffic between cards, and spreads it across all GPUs only when it does not fit. `OLLAMA_SCHED_SPREAD` forces spreading every time; usually the better use of the second card is a second model or more slots.
  • You set OLLAMA_NUM_PARALLEL=4, yet an embedding model still serves one request at a time. Why?
    Ollama always loads embedding models with a single slot, regardless of the variable, and also forces some architectures back to one slot with a warning in the server log. Concurrency for those comes from queueing or from batching inputs in one `/api/embed` call, not from more slots.

  • OLLAMA_NUM_PARALLEL shares one context among all parallel requests
  • Parallel slots only cost CPU threads, not memory
  • Setting num_gpu to the layer count forces the model to fit
  • OLLAMA_NUM_PARALLEL applies to the whole server, not per model
  • A fifth request beyond four slots is rejected immediately

Assumes Ollama 0.35.0: the 32k figure is its VRAM-based default context for a 24 GiB GPU; parallel slots multiplying context memory holds across recent releases.