skip to content

Bindings and Compute Backends

libllama's C API is embedded through llama-cpp-python or node-llama-cpp, and each build targets ggml backends such as CUDA, Metal or Vulkan. Interviewers ask when in-process inference beats a server.

part ofLocal LLM runtimesread it →
on this pageshow

explore

questions

6

For a Python batch job, when does embedding llama.cpp in-process with llama-cpp-python beat calling a llama-server, and when does it lose?

level: seniorimportance: must knowfreq 12%

answer

  1. who else needs the model
  2. one process, no port, no hop
  3. one Llama, one sequence at a time
  4. a copy per worker process
  5. crash isolation and upgrade cadence

basics

~20 s

In-process wins when one job owns the model: no service, port or HTTP hop, plus direct access to tokens, logits, grammars and state. A llama-server wins when many processes or concurrent requests share the model, or a crash must not kill the job.

solid answer

~50 s

Embedding with `llama-cpp-python` puts libllama inside the job: the model loads once at start-up, there is no port to bind or secure, no second service to supervise, and you can reach things an HTTP route may not expose: `tokenize()`, logits processors, a `LlamaGrammar`, `save_state()` and `load_state()`, embeddings with `embedding=True`. A fixed instruction prefix is also reused from the KV cache between calls. It loses when the model must be shared. One `Llama` object generates one sequence at a time, and the binding's own optional server serializes requests behind a lock, while llama-server runs parallel slots with continuous batching. Every worker process that builds its own `Llama` loads its own copy into memory. A native crash takes the job down with it. And the job is tied to the binding's bundled llama.cpp and the backend it was compiled with.

code

python · 17 lines
python
import json
from llama_cpp import Llama

INSTRUCTION = "Classify the ticket as billing, bug or other. Reply with one word."

llm = Llama(model_path="./models/your-model.gguf",
            n_gpu_layers=-1, n_ctx=4096, verbose=False)

with open("tickets.jsonl") as src, open("labels.jsonl", "w") as out:
    for line in src:                       # one sequence at a time
        ticket = json.loads(line)
        reply = llm.create_chat_completion(
            messages=[{"role": "system", "content": INSTRUCTION},
                      {"role": "user", "content": ticket["text"]}],
            max_tokens=4, temperature=0)
        label = reply["choices"][0]["message"]["content"].strip()
        out.write(json.dumps({"id": ticket["id"], "label": label}) + "\n")

go deeper

for a junior

Recall that llama-cpp-python runs the model inside your Python process while llama-server is a separate program you call over HTTP, and that both run the same engine.

for a middle

Explain what embedding gives you, no port, no hop and direct access to tokens, logits and state, and why one Llama object handles one sequence at a time.

for a senior

Decide from consumers, concurrency, crash impact and upgrade cadence, and spot the multi-process trap where every worker holds its own copy of the model.

for a principal

Choose a standard shape for a team's local inference, embedded per job or one shared server, weighing isolation, GPU memory, upgrade ownership and the engine features each team actually needs.

## Two ways to run the same engine A Python job that needs a local model can either **embed** llama.cpp, loading libllama into its own process through `llama-cpp-python`, or **call a server**, sending HTTP requests to a `llama-server` process started separately. Both run the same engine on the same GGUF. The choice is about ownership: who else needs the model, how many requests run at once, and what happens when something fails. ## When in-process wins - **One owner, one lifetime.** A nightly job that loads a model, processes a file of rows and exits has no use for a long-lived service. `Llama(model_path=...)` at the top and the process exit at the bottom is the whole deployment. - **Nothing to expose.** No port to bind, no API key to manage, no health checks, no second unit file. On an air-gapped customer machine one Python environment is easier to ship and audit than a service plus a client. - **No hop.** Each call is a function call: no JSON encoding, no HTTP, no connection pool. - **Engine access an HTTP route may not give you.** `llm.tokenize()` and `detokenize()` for exact token counts, `logits_processor` hooks to bias generation, a `grammar` built with `LlamaGrammar.from_json_schema()`, `save_state()` and `load_state()` to snapshot the KV cache, and embeddings when the model is loaded with `embedding=True`. - **Prefix reuse for free.** Between calls the `Llama` object keeps its KV cache and reuses the longest matching token prefix, so a batch whose prompts share a long fixed instruction evaluates that instruction once; `verbose=True` logs each prefix-match hit. ## When a server wins - **Concurrency.** A `Llama` object holds one sequence: calls run one after another. The binding's optional OpenAI-compatible server wraps that single object in a lock, so it serializes too. `llama-server` runs several **slots** and batches them continuously; how that scheduling works belongs to the server and batching material, but the effect is far higher throughput under concurrent load. - **Sharing across processes.** A `multiprocessing` pool in which every worker constructs its own `Llama` holds one copy of the model per worker, in VRAM when layers are offloaded. A single server loads the model once and every worker, service or language calls it. - **Fault isolation.** libllama is native code. An abort inside it, which some GPU out-of-memory failures cause, ends the Python process and the job's in-memory progress with it. A server can fail and be restarted by its supervisor while the job retries. - **Upgrade cadence.** The binding compiles its **own** llama.cpp at install time and fixes the GPU backend then. Support for a new model architecture reaches the binding only through a new `llama-cpp-python` release, whereas a `llama-server` binary can be swapped independently of the job's Python environment. - **Operations.** A server has its own logs, monitoring routes and lifecycle, which a team can watch without touching the job. ## A decision table | Situation | Lean towards | |---|---| | One batch job, one model, sequential rows | in-process | | Needs logits, grammars, token counts or KV state snapshots | in-process | | Many worker processes or several services share the model | server | | Several requests must run at the same time | server | | A crash must not lose the job's progress | server, or checkpoint the job | | Must pick up new model architectures quickly | server | | Air-gapped install with the fewest moving parts | in-process | ## How to answer the trade-off 1. Count the **consumers**: one job, or many processes and services? 2. Count the **concurrent requests**: sequential rows, or parallel traffic? 3. Ask what the job needs from the engine beyond text in, text out. 4. Ask who upgrades the engine, and how fast new models must run. A strong answer also names the middle ground. A sequential batch can scale out with several processes only if the GPU has room for several copies; otherwise, one server with parallel slots is the efficient shape. In Node, `node-llama-cpp` can create a context with several `sequences` and batch them inside one process, so "in-process means one request at a time" is a property of the Python `Llama` class, not of embedding itself. The costs of the feature in money and latency are a separate discussion; this one is about where the engine lives.

  • The batch job is too slow, so you start eight worker processes, each with its own Llama. What goes wrong?
    Each worker loads its own copy of the model, so offloaded layers are duplicated in VRAM eight times and the job may not fit at all. The GPU also ends up time-sliced between separate contexts. One llama-server with several parallel slots, called by the eight workers, loads the model once and batches their requests.
  • Why does llama-cpp-python's own OpenAI-compatible server not solve concurrency?
    It is built on the same single `Llama` object and guards it with a lock, so requests are served one after another. It gives you an HTTP interface and process separation, but not llama-server's parallel slots or continuous batching.
  • Which engine features push you towards embedding even for a long-running service?
    Needs that an HTTP route may not cover: custom `logits_processor` hooks, saving and restoring KV state with `save_state()` and `load_state()`, exact tokenization in the same process, or a grammar assembled per call with `LlamaGrammar`. If those matter more than sharing, embed and accept one model copy per process.

  • Assumes one Llama object can serve concurrent requests in parallel.
  • Scales a llama-cpp-python job with worker processes without counting model copies.
  • Thinks llama-cpp-python's bundled server batches requests like llama-server.
  • Assumes every native failure inside libllama surfaces as a catchable Python exception.
  • Believes embedding always beats a server because it skips HTTP.
  • Assumes the binding runs the same llama.cpp version as the system's llama-server.
open as a page

What do llama-cpp-python and node-llama-cpp wrap, and what does a minimal chat call look like in each?

level: juniorimportance: should knowfreq 10%

basics

~10 s

Both embed libllama, the llama.cpp C library declared in include/llama.h, inside your own process. Python builds Llama(model_path=...) and calls create_chat_completion(messages=...); Node calls getLlama(), loadModel(), createContext() and LlamaChatSession.prompt().

open as a page

When building llama.cpp for an NVIDIA, AMD, Intel or Vulkan-capable GPU, which CMake flags select the ggml backend, and how do you confirm it?

level: middleimportance: should knowfreq 6%

basics

~10 s

The backend is compiled in: -DGGML_CUDA=ON for NVIDIA, -DGGML_HIP=ON for AMD ROCm, -DGGML_SYCL=ON for Intel GPUs, -DGGML_VULKAN=ON for Vulkan; Metal is on by default on macOS. Confirm with --list-devices or llama-bench's backend column.

open as a page

A GGUF that runs fast under llama-server crawls on the CPU and rejects long prompts through llama-cpp-python's Llama(model_path=...); which defaults explain it, and what do you set?

level: seniorimportance: should knowfreq 8%

basics

~10 s

llama-cpp-python 0.3.36's Llama defaults to n_gpu_layers=0, n_ctx=512 and flash_attn=False, while llama-server 0.5.0 picks GPU layers, the model's own context and flash attention automatically. Set n_gpu_layers=-1, an explicit n_ctx and flash_attn=True.

open as a page

In llama.cpp 0.5.0, an old launch script's --mlock and --no-mmap flags are rejected; what replaced them, and what does each --load-mode value do?

level: middleimportance: nice to knowfreq 4%

basics

~20 s

llama.cpp 0.5.0 replaced both with -lm / --load-mode: auto (the default, mmap unless a device cannot), none, mmap, mlock, mmap+mlock or dio. --no-mmap becomes none; --mlock becomes mmap+mlock, or mlock when mapping was off too.

open as a page

Two llama.cpp builds of the same GGUF differ threefold in llama-bench; how do you read pp512 against tg128 and isolate the cause?

level: seniorimportance: nice to knowfreq 5%

basics

~20 s

pp512 is prompt-processing speed over a 512-token prompt and tg128 is generation speed over 128 tokens, both in tokens per second. Check the backend, ngl and build fields first, then rerun both builds with identical explicit flags, sweeping one parameter.

open as a page