skip to content

HTTP Routes & Compat Gaps

Ollama's native /api/chat, /api/generate and /api/embed sit beside OpenAI-compatible /v1 routes and an Anthropic /v1/messages route. Interviewers ask what breaks when an SDK is repointed at localhost.

part ofLocal LLM runtimesread it →
on this pageshow

explore

questions

5

With a local Ollama server, how do you repoint an OpenAI Python SDK app, and what must the client still supply?

level: juniorimportance: must knowfreq 24%

answer

  1. base URL, not new code
  2. the /v1 path suffix
  3. a key nobody checks
  4. the model name is Ollama's
  5. ollama cp for hard-coded names

basics

~10 s

Set the client's base_url to http://localhost:11434/v1/, pass any placeholder api_key (the SDK requires one, Ollama ignores it), and name a model already pulled into Ollama, such as gpt-oss:20b.

solid answer

~40 s

Ollama serves an OpenAI-compatible subset under `/v1` on the same port as its native API, so only the client construction changes: `OpenAI(base_url='http://localhost:11434/v1/', api_key='ollama')`. The key is required by the SDK but ignored by the local server, which has no authentication. The `model` must be an Ollama model name that is already on the machine (`ollama pull gpt-oss:20b`); otherwise the request returns a 404 "model not found" error. Code that hard-codes a hosted name such as `gpt-3.5-turbo` can be served by `ollama cp llama3.2 gpt-3.5-turbo`. Chat completions, streaming, JSON mode, tools, base64 vision, embeddings and stateless responses all work through the same client, but each route honours only a listed subset of request fields.

code

python · 12 lines
python
from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1/",
    api_key="ollama",  # required by the SDK, ignored by Ollama
)

reply = client.chat.completions.create(
    model="gpt-oss:20b",  # pulled first: ollama pull gpt-oss:20b
    messages=[{"role": "user", "content": "Say this is a test"}],
)
print(reply.choices[0].message.content)

go deeper

for a junior

Recall the three client settings: base_url with the /v1 suffix, a placeholder api_key the server ignores, and an Ollama model name you have already pulled.

for a middle

Explain why the key is ignored (the local API has no authentication) and how ollama cp lets code with a hard-coded hosted model name run unchanged.

for a senior

Show that a working connection is not full compatibility: name a request field the /v1 routes drop, and say when you would move a client to the native /api routes.

for a principal

Weigh keeping the OpenAI SDK for portability across hosted and local backends against the Ollama-only controls a native client gains, and decide where that seam sits in the codebase.

## What the compatibility layer is Ollama runs one HTTP server, on `127.0.0.1:11434` by default, that answers two families of routes. The **native API** lives under `/api` (`/api/chat`, `/api/generate`, `/api/embed`) and has its own request and response shapes. Beside it, Ollama serves an **OpenAI-compatible subset** under `/v1`: `/v1/chat/completions`, `/v1/completions`, `/v1/models`, `/v1/embeddings` and `/v1/responses`. An application written against the official OpenAI SDK can therefore talk to a local model without being rewritten: the SDK builds the same JSON it would send to the hosted service, and Ollama translates it into a native chat or generate call. "Compatible" means the request and response **shapes** match. It does not mean every field is honoured; those gaps are a separate question. The first job is simply to get a working connection. ## The three things the client supplies 1. **`base_url`** — `http://localhost:11434/v1/`. The `/v1` suffix matters: the SDK appends paths such as `chat/completions` to whatever base it is given, and only the `/v1` prefix leads to the compatibility handlers. Without it the request lands on a path the server does not have and comes back 404. 2. **`api_key`** — any placeholder, conventionally `"ollama"`. The SDK refuses to construct a client without a key, but the local server has no authentication and ignores the value. The docs annotate it as "required but ignored". 3. **`model`** — an **Ollama model name** in `model:tag` form, such as `gpt-oss:20b` or `llama3.2`, that is already on the machine. Ollama does not fetch a model because a request names it; an unknown name comes back as a 404 "model not found" error. Run `ollama pull <name>` beforehand. Nothing else in the calling code needs to change: `client.chat.completions.create(...)`, `client.embeddings.create(...)` and `client.responses.create(...)` keep their signatures. ## Names the code cannot change Some tools hard-code a hosted model name. Rather than patching them, give a local model that name: `ollama cp llama3.2 gpt-3.5-turbo` copies an existing local model to the new name, and requests for `gpt-3.5-turbo` are then served by it. `GET /v1/models` lists what is installed, with `owned_by` set to the Ollama namespace (`"library"` by default) and `created` set to when the model was last modified, which helps a tool that validates its configured model at start-up. ## Checking the connection before blaming the SDK Two quick requests separate a client bug from a server problem: - `curl http://localhost:11434/` answers "Ollama is running" when the daemon is up and reachable on that address. - `curl http://localhost:11434/v1/models` returns the installed models in OpenAI's list shape; if the name the code sends is absent, pull it or alias it. If both succeed and the SDK still fails, the fault is in the client's base URL or model string, not in Ollama. ## What works through the same client | SDK call | Route | Notes | |---|---|---| | `chat.completions.create` | `/v1/chat/completions` | streaming, JSON mode, tools, vision with base64 images | | `completions.create` | `/v1/completions` | `prompt` must be a string | | `embeddings.create` | `/v1/embeddings` | a string or a list of strings | | `models.list` | `/v1/models` | installed models only | | `responses.create` | `/v1/responses` | stateless only (added in v0.13.3) | Streaming over `/v1` uses the event-stream framing the SDK already parses, so `stream=True` needs no special handling in the client. ## First-connection mistakes - **Dropping `/v1`.** The native routes live at `/api/...`; the bare host has no `chat/completions` path, so the SDK gets a 404. - **Blaming the key.** A placeholder key is never why a local request fails; look at the URL and the model name instead. - **Sending the hosted model name** without an alias. No local model has that name, so the request returns 404. - **Running the client on another machine.** The default bind is loopback only; reaching it from elsewhere means setting `OLLAMA_HOST`, and because the local API checks no credentials, anything that can reach the port can use it. - **Confusing local with cloud.** `https://ollama.com/v1` is Ollama's hosted API: there the key is real (`OLLAMA_API_KEY`, sent as a bearer token) and prompts leave the machine. ## Where compatibility stops Once the connection works, the next questions are which request fields the `/v1` routes ignore and what only the native API can do. Two examples: the OpenAI request has no field for the model's context size, and several request fields are dropped without an error. Treat the compatibility page's field checklist, not the hosted API's reference, as the contract for a local server, and test behaviour rather than status codes when a field matters to you.

  • A vendor tool hard-codes model 'gpt-3.5-turbo' — how do you serve it from Ollama without patching the tool?
    Copy a local model to that name with `ollama cp llama3.2 gpt-3.5-turbo`. Requests naming `gpt-3.5-turbo` on `/v1/chat/completions` are then answered by the copied local model. The tool still receives an OpenAI-shaped response; it simply never reaches OpenAI.
  • The same client works on your laptop but cannot connect from a teammate's machine — why?
    Ollama binds `127.0.0.1:11434` by default, so only local processes can connect. Setting `OLLAMA_HOST` (for example `0.0.0.0:11434`) makes it listen on the network, but the local API has no authentication, so put access control in front before widening the bind.

  • The api_key must be a valid OpenAI key or Ollama rejects the request.
  • The base_url is http://localhost:11434 with no /v1 suffix.
  • Naming a model in the request makes Ollama download it automatically.
  • Ollama maps hosted names like gpt-4o to a similar local model by itself.
  • The local server checks the incoming key against OLLAMA_API_KEY.
open as a page

In Ollama, what changes when an indexer moves from /api/embeddings to /api/embed, and what does /api/embed do with over-long inputs?

level: middleimportance: should knowfreq 8%

basics

~20 s

/api/embed takes input as a string or list and returns one L2-normalized vector per item under embeddings; /api/embeddings took one prompt and returned a raw vector. Over-long inputs are silently cut unless truncate is false.

open as a page

When does a client need Ollama's native /api/chat rather than /v1/chat/completions, and how do you read its streamed response?

level: middleimportance: should knowfreq 12%

basics

~20 s

Use /api/chat when a request needs Ollama's own controls directly: per-request options such as num_ctx, think levels, or a JSON schema in format. It streams by default as newline-delimited JSON; the last object has done: true and the timings.

open as a page

Against Ollama 0.35's /v1/chat/completions, an agent sending tool_choice 'required' and n=3 gets one choice and prose replies — why, and what else differs silently?

level: seniorimportance: should knowfreq 11%

basics

~10 s

Ollama 0.35's /v1/chat/completions does not support tool_choice, n, logit_bias or user and drops them without an error. It also sends temperature and top_p of 1.0 when omitted and has no context-size field.

open as a page

Pointing an Anthropic-SDK coding tool at a local Ollama 0.35 server, what do you configure, and which Messages API features stop working?

level: middleimportance: nice to knowfreq 5%

basics

~10 s

Set the base URL to http://localhost:11434 with a placeholder token and a local model name. Ollama's /v1/messages handles messages, streaming, tools and thinking, but not tool_choice, count_tokens, prompt caching, batches, PDFs or image URLs.

open as a page