What do llama-cpp-python and node-llama-cpp wrap, and what does a minimal chat call look like in each?
answer
- one C library under both
- include/llama.h, the libllama API
- model in your process, no port
- Llama(...) then create_chat_completion
- getLlama, loadModel, LlamaChatSession
basics
~10 sBoth embed libllama, the llama.cpp C library declared in include/llama.h, inside your own process. Python builds Llama(model_path=...) and calls create_chat_completion(messages=...); Node calls getLlama(), loadModel(), createContext() and LlamaChatSession.prompt().
solid answer
~40 sllama.cpp's engine is a C/C++ library, libllama, whose public API lives in `include/llama.h`: load a model, create a context, tokenize, call `llama_decode`, sample a token. `llama-cpp-python` binds that header through `ctypes` and adds a high-level `Llama` class: `Llama(model_path=...)` or `Llama.from_pretrained(repo_id=..., filename=...)`, then `create_chat_completion(messages=[...])`, which returns an OpenAI-shaped dict. `node-llama-cpp` is a TypeScript binding: `getLlama()` picks a compute backend, `llama.loadModel({modelPath})` loads the GGUF, `model.createContext()` allocates the context, and a `LlamaChatSession` built on `context.getSequence()` answers `session.prompt(text)`. Neither needs a server or a port: the weights live in your process. Both format the turns for you, from the chat template in the GGUF in Python's case and from an automatically resolved chat wrapper in Node's.
code
typescript · 16 linesimport {getLlama, LlamaChatSession} from "node-llama-cpp";
const llama = await getLlama(); // detects Metal, CUDA or Vulkan
console.log("GPU type:", llama.gpu);
const model = await llama.loadModel({
modelPath: "./models/your-model.gguf"
});
const context = await model.createContext();
const session = new LlamaChatSession({
contextSequence: context.getSequence()
});
console.log(await session.prompt("Summarize GGUF in one sentence."));
await model.dispose(); // also disposes the contextgo deeper
Recall that both packages load a GGUF straight into your process through libllama, and be able to write the three or four calls that produce one chat reply in Python or TypeScript.
Explain the model and context split from llama.h, how each binding obtains its llama.cpp build, and where chat formatting comes from when you pass no override.
Show you know the binding pins its own llama.cpp and its own defaults, so behaviour can diverge from llama-server on the same file, and how you would check which build and template are in use.
Weigh standardizing on a binding against a shared server for a team: who owns upgrades for new architectures, how many copies of a model end up in memory, and which language ecosystems need first-class support.
## What a binding is **llama.cpp** is both a set of command-line tools (`llama-server`, `llama-cli`, `llama-bench`, `llama-quantize`) and the library they are built on. That library is **libllama**, and its public C API is the single header `include/llama.h`. A **binding** is a package that loads libllama into another language's process and calls that API, so an application can run a GGUF model without starting any separate program. The two bindings people meet most are separate projects with their own release numbers: - **llama-cpp-python** (0.3.36 at the time of writing), for Python. - **node-llama-cpp** (3.22.1 at the time of writing), for Node.js and TypeScript. Each one compiles or ships **its own copy** of llama.cpp. The llama.cpp inside your Python environment is not the `llama-server` binary on your `PATH`, and the two can be months apart. ## The C API underneath Whatever language you call from, the work goes through the same sequence of `llama.h` calls: 1. `llama_backend_init()` once per program. 2. `llama_model_load_from_file(path, llama_model_default_params())` reads the GGUF and returns a `llama_model`. 3. `llama_init_from_model(model, llama_context_default_params())` creates a `llama_context`, which owns the KV cache and the context length. 4. `llama_model_get_vocab()` and `llama_tokenize()` turn text into tokens. 5. `llama_batch_get_one()` and `llama_decode()` run the model over those tokens. 6. A sampler chain from `llama_sampler_chain_init()` picks the next token with `llama_sampler_sample()`, and `llama_token_to_piece()` turns it back into text; the loop repeats until `llama_vocab_is_eog()` says the turn ended. 7. `llama_free(ctx)` and `llama_model_free(model)` release everything. The split between a **model** (the weights, loaded once) and a **context** (per-conversation state) is the design every binding inherits. ## llama-cpp-python - `pip install llama-cpp-python` **builds llama.cpp from source** during installation; a prebuilt CPU-only wheel is also published on a separate package index. - The **high-level API** is the `Llama` class. `Llama(model_path="./models/your-model.gguf")` loads a local file; `Llama.from_pretrained(repo_id=..., filename="*Q8_0.gguf")` downloads a matching file first and needs the `huggingface-hub` package. - `create_chat_completion(messages=[...])` returns a dict shaped like an OpenAI chat completion; `create_completion()` (or calling the object) does plain text completion. - Chat formatting follows a fixed precedence: an explicit `chat_handler`, else an explicit `chat_format`, else the `tokenizer.chat_template` stored in the GGUF, else a `llama-2` fallback. `verbose=True` prints which one was chosen. - The **low-level API**, the `llama_cpp` module, is a direct `ctypes` mirror of `llama.h`, for when you need a call the `Llama` class does not expose. - An optional extra, `llama-cpp-python[server]`, adds an OpenAI-compatible HTTP server built on the same class. ## node-llama-cpp - `npm install node-llama-cpp` brings **prebuilt binaries** for macOS, Linux and Windows. When none fits the platform it downloads a llama.cpp release and builds it with CMake; `NODE_LLAMA_CPP_SKIP_DOWNLOAD=true` turns that fallback off. - `getLlama()` detects the available compute backend (Metal on Apple Silicon, CUDA or Vulkan when support is found) and returns a `Llama` object; `llama.gpu` tells you which one it picked. - `llama.loadModel({modelPath})` returns a model; `model.createContext()` returns a context; `context.getSequence()` hands out one sequence of that context. - `new LlamaChatSession({contextSequence})` keeps the chat history and formats it with a **chat wrapper** chosen by `resolveChatWrapper()`; `session.prompt(text)` returns the reply as a string. - Every object has `.dispose()`, and disposing a model disposes its contexts too. ## Side by side | | llama-cpp-python | node-llama-cpp | |---|---|---| | Gets llama.cpp by | compiling it at `pip install` | prebuilt binaries, CMake fallback | | Load a model | `Llama(model_path=...)` | `llama.loadModel({modelPath})` | | Chat | `create_chat_completion(messages=...)` | `LlamaChatSession` + `prompt()` | | Chat formatting | GGUF template unless overridden | resolved chat wrapper unless overridden | | Raw C API | `llama_cpp` ctypes module | its own lower-level classes | ## Why it matters in an interview The point is not the method names but the shape: the model runs **in-process**, nothing listens on a port, and the binding decides defaults (context size, GPU offload) that may differ from what `llama-server` would pick for the same file. Knowing that a binding is "libllama plus a friendlier object model" explains why a model that works in one place behaves differently in the other, and why upgrading the binding is how you pick up support for a new model architecture.
- Which llama.cpp version does a llama-cpp-python install actually run?The one it compiled at install time, from the llama.cpp source it vendors, not the `llama-server` on your machine. A new model architecture therefore needs a newer `llama-cpp-python` release, and rebuilding means reinstalling with `--upgrade --force-reinstall --no-cache-dir` so pip does not reuse an old wheel.
- How does llama-cpp-python decide how to format chat messages for a model?It takes an explicit `chat_handler` first, then an explicit `chat_format` such as `chatml`, then the `tokenizer.chat_template` stored in the GGUF metadata, and falls back to the `llama-2` format when none exists. Passing `verbose=True` prints the format it selected, which is the first check when replies look garbled.
- In node-llama-cpp, why is a chat session built on a context sequence rather than on the model?The model holds only the weights; a context holds the KV cache and can be split into several sequences. A `LlamaChatSession` owns one sequence's history, so several sessions can share one loaded model, and a context created with `sequences: 2` lets two sessions generate in parallel.
- Says llama-cpp-python talks to a running llama-server over HTTP.
- Assumes the binding uses the llama.cpp version installed on the system.
- Thinks node-llama-cpp always compiles llama.cpp from source on install.
- Believes create_chat_completion needs a chat_format for every GGUF model.
- Confuses the libllama C API with llama-server's HTTP routes.