skip to content

For a Python batch job, when does embedding llama.cpp in-process with llama-cpp-python beat calling a llama-server, and when does it lose?

level: seniorimportance: must knowfreq 12%

answer

  1. who else needs the model
  2. one process, no port, no hop
  3. one Llama, one sequence at a time
  4. a copy per worker process
  5. crash isolation and upgrade cadence

basics

~20 s

In-process wins when one job owns the model: no service, port or HTTP hop, plus direct access to tokens, logits, grammars and state. A llama-server wins when many processes or concurrent requests share the model, or a crash must not kill the job.

solid answer

~50 s

Embedding with `llama-cpp-python` puts libllama inside the job: the model loads once at start-up, there is no port to bind or secure, no second service to supervise, and you can reach things an HTTP route may not expose: `tokenize()`, logits processors, a `LlamaGrammar`, `save_state()` and `load_state()`, embeddings with `embedding=True`. A fixed instruction prefix is also reused from the KV cache between calls. It loses when the model must be shared. One `Llama` object generates one sequence at a time, and the binding's own optional server serializes requests behind a lock, while llama-server runs parallel slots with continuous batching. Every worker process that builds its own `Llama` loads its own copy into memory. A native crash takes the job down with it. And the job is tied to the binding's bundled llama.cpp and the backend it was compiled with.

code

python · 17 lines
python
import json
from llama_cpp import Llama

INSTRUCTION = "Classify the ticket as billing, bug or other. Reply with one word."

llm = Llama(model_path="./models/your-model.gguf",
            n_gpu_layers=-1, n_ctx=4096, verbose=False)

with open("tickets.jsonl") as src, open("labels.jsonl", "w") as out:
    for line in src:                       # one sequence at a time
        ticket = json.loads(line)
        reply = llm.create_chat_completion(
            messages=[{"role": "system", "content": INSTRUCTION},
                      {"role": "user", "content": ticket["text"]}],
            max_tokens=4, temperature=0)
        label = reply["choices"][0]["message"]["content"].strip()
        out.write(json.dumps({"id": ticket["id"], "label": label}) + "\n")

go deeper

for a junior

Recall that llama-cpp-python runs the model inside your Python process while llama-server is a separate program you call over HTTP, and that both run the same engine.

for a middle

Explain what embedding gives you, no port, no hop and direct access to tokens, logits and state, and why one Llama object handles one sequence at a time.

for a senior

Decide from consumers, concurrency, crash impact and upgrade cadence, and spot the multi-process trap where every worker holds its own copy of the model.

for a principal

Choose a standard shape for a team's local inference, embedded per job or one shared server, weighing isolation, GPU memory, upgrade ownership and the engine features each team actually needs.

## Two ways to run the same engine A Python job that needs a local model can either **embed** llama.cpp, loading libllama into its own process through `llama-cpp-python`, or **call a server**, sending HTTP requests to a `llama-server` process started separately. Both run the same engine on the same GGUF. The choice is about ownership: who else needs the model, how many requests run at once, and what happens when something fails. ## When in-process wins - **One owner, one lifetime.** A nightly job that loads a model, processes a file of rows and exits has no use for a long-lived service. `Llama(model_path=...)` at the top and the process exit at the bottom is the whole deployment. - **Nothing to expose.** No port to bind, no API key to manage, no health checks, no second unit file. On an air-gapped customer machine one Python environment is easier to ship and audit than a service plus a client. - **No hop.** Each call is a function call: no JSON encoding, no HTTP, no connection pool. - **Engine access an HTTP route may not give you.** `llm.tokenize()` and `detokenize()` for exact token counts, `logits_processor` hooks to bias generation, a `grammar` built with `LlamaGrammar.from_json_schema()`, `save_state()` and `load_state()` to snapshot the KV cache, and embeddings when the model is loaded with `embedding=True`. - **Prefix reuse for free.** Between calls the `Llama` object keeps its KV cache and reuses the longest matching token prefix, so a batch whose prompts share a long fixed instruction evaluates that instruction once; `verbose=True` logs each prefix-match hit. ## When a server wins - **Concurrency.** A `Llama` object holds one sequence: calls run one after another. The binding's optional OpenAI-compatible server wraps that single object in a lock, so it serializes too. `llama-server` runs several **slots** and batches them continuously; how that scheduling works belongs to the server and batching material, but the effect is far higher throughput under concurrent load. - **Sharing across processes.** A `multiprocessing` pool in which every worker constructs its own `Llama` holds one copy of the model per worker, in VRAM when layers are offloaded. A single server loads the model once and every worker, service or language calls it. - **Fault isolation.** libllama is native code. An abort inside it, which some GPU out-of-memory failures cause, ends the Python process and the job's in-memory progress with it. A server can fail and be restarted by its supervisor while the job retries. - **Upgrade cadence.** The binding compiles its **own** llama.cpp at install time and fixes the GPU backend then. Support for a new model architecture reaches the binding only through a new `llama-cpp-python` release, whereas a `llama-server` binary can be swapped independently of the job's Python environment. - **Operations.** A server has its own logs, monitoring routes and lifecycle, which a team can watch without touching the job. ## A decision table | Situation | Lean towards | |---|---| | One batch job, one model, sequential rows | in-process | | Needs logits, grammars, token counts or KV state snapshots | in-process | | Many worker processes or several services share the model | server | | Several requests must run at the same time | server | | A crash must not lose the job's progress | server, or checkpoint the job | | Must pick up new model architectures quickly | server | | Air-gapped install with the fewest moving parts | in-process | ## How to answer the trade-off 1. Count the **consumers**: one job, or many processes and services? 2. Count the **concurrent requests**: sequential rows, or parallel traffic? 3. Ask what the job needs from the engine beyond text in, text out. 4. Ask who upgrades the engine, and how fast new models must run. A strong answer also names the middle ground. A sequential batch can scale out with several processes only if the GPU has room for several copies; otherwise, one server with parallel slots is the efficient shape. In Node, `node-llama-cpp` can create a context with several `sequences` and batch them inside one process, so "in-process means one request at a time" is a property of the Python `Llama` class, not of embedding itself. The costs of the feature in money and latency are a separate discussion; this one is about where the engine lives.

  • The batch job is too slow, so you start eight worker processes, each with its own Llama. What goes wrong?
    Each worker loads its own copy of the model, so offloaded layers are duplicated in VRAM eight times and the job may not fit at all. The GPU also ends up time-sliced between separate contexts. One llama-server with several parallel slots, called by the eight workers, loads the model once and batches their requests.
  • Why does llama-cpp-python's own OpenAI-compatible server not solve concurrency?
    It is built on the same single `Llama` object and guards it with a lock, so requests are served one after another. It gives you an HTTP interface and process separation, but not llama-server's parallel slots or continuous batching.
  • Which engine features push you towards embedding even for a long-running service?
    Needs that an HTTP route may not cover: custom `logits_processor` hooks, saving and restoring KV state with `save_state()` and `load_state()`, exact tokenization in the same process, or a grammar assembled per call with `LlamaGrammar`. If those matter more than sharing, embed and accept one model copy per process.

  • Assumes one Llama object can serve concurrent requests in parallel.
  • Scales a llama-cpp-python job with worker processes without counting model copies.
  • Thinks llama-cpp-python's bundled server batches requests like llama-server.
  • Assumes every native failure inside libllama surfaces as a catchable Python exception.
  • Believes embedding always beats a server because it skips HTTP.
  • Assumes the binding runs the same llama.cpp version as the system's llama-server.