LLM Providers & Models

Every entry in the messages array carries a role. system holds standing instructions the model should follow throughout, user holds what the human said, and assistant holds the model's own earlier replies that you replay as context.

A POST to /v1/embeddings sends a model plus input (a string or an array of strings) and returns a data array of vectors, each carrying its embedding and the index of the input it came from, plus a usage object counting input tokens.

OpenAI bills per token, quoted per million tokens, and output tokens cost several times more than input tokens. The API is stateless, so every turn resends the whole conversation as input — the input bill grows with each turn.

previous_response_id points at a stored earlier response, so the server replays that whole conversation as the prefix of the new request. You send only the new turn, but you are still billed for every replayed input token.

Upload the files, add them to a vector store, and wait for indexing to finish, then reference that store's id from the file_search tool on your request. OpenAI handles chunking, embedding, and ranking; you pay for storage per day.

The Messages API takes the system prompt as a top-level system parameter on the request body, alongside model, max_tokens and messages. It is not a message inside the messages array; that array carries only user and assistant turns.

Haiku, Sonnet and Opus are Anthropic's three Claude size tiers, ordered smallest to largest. Haiku is the fastest and cheapest, Opus is the most capable and most expensive, and Sonnet sits between them. All three are called through the same Messages API.

Send a top-level tools array on the Messages request. Each entry needs a name, a description that tells Claude when to call it, and an input_schema: a JSON Schema object describing the parameters.

Set the user message content to an array of blocks and add an image block next to your text block. The image block carries a source object that is either base64 (with media_type and data) or a url. Image blocks belong in user turns.

max_tokens is a hard ceiling on the tokens Claude generates in that one response, including any thinking tokens. It is required on every Messages API request, and hitting it truncates the output mid-stream with stop_reason "max_tokens".

Gemini returns an ordinary response whose candidate content holds a FunctionCall part: a function name plus an args object of already-parsed arguments. The model never runs your code — your application executes the call and sends the result back.

Create a client with genai.Client(), which picks up the GEMINI_API_KEY environment variable, then call client.models.generate_content(model=..., contents=...). You get back a GenerateContentResponse whose .text property joins the text parts of the first candidate.

Gemini takes media as a Part inside contents. Small files ride along as inline_data (raw bytes plus a mime_type); larger or reused files are uploaded to the Files API first and referenced as file_data with a file_uri and mime_type.

Each safetySettings entry pairs a harm category (harassment, hate speech, sexually explicit, dangerous content, civic integrity) with a block threshold. The threshold sets how confident the filter must be before Gemini refuses to return content in that category.

task_type tells Gemini's embedding model what the text is for — a stored passage, a search query, a classification input — and the model places the same text differently in vector space for each. Index and query sides must use the matching pair.

A Cohere v2/chat call needs a model ID and a messages array of role/content objects, authenticated with a bearer API key. The response carries an assistant message whose content is a list of typed blocks, plus finish_reason and usage.

Send texts plus a model and an input_type to POST /v2/embed. The response groups vectors by type: a float request returns them under embeddings.float, one vector per input text, in the order you sent them.

Send a top-level documents array alongside messages in the v2 chat call. Each entry is a plain string or an object with an id and a data map. The reply's message.citations then link answer spans to those documents.

A Cohere v2/rerank call sends a model name, one query string, and a documents list of plain strings, plus an optional top_n. It returns a results array ordered by relevance, each entry carrying the document's original index and a relevance_score.

Command A is Cohere's current flagship chat model with a 256K-token context, above the earlier Command R+ and Command R at 128K and the small Command R7B. IDs carry a date suffix; undated aliases follow the newest snapshot, so production should pin the dated ID.

Mistral's POST /v1/chat/completions needs only two body fields: model (for example mistral-large-latest) and messages, an ordered array of role/content objects. Authentication is a bearer API key; max_tokens, temperature and the rest are optional.

POST /v1/embeddings on api.mistral.ai with a bearer API key, model "mistral-embed" and an "input" array of strings. The response carries a data array of 1024-dimension float vectors, each tagged with the index of the string it came from, plus a token usage block.

Mistral sets choices[0].finish_reason to "tool_calls" and fills choices[0].message.tool_calls. Each entry has an id, a function.name, and function.arguments — a JSON-encoded string you must parse before running your code.

Set stream true on Mistral's chat completions request and the reply arrives as chat.completion.chunk objects over server-sent events. Concatenate choices[0].delta.content across chunks in arrival order; the last chunk carries finish_reason, and the stream closes with data: [DONE].

Use "any". Mistral's documented tool_choice values are auto (the default when tools are present), any (the model must call a tool), and none (tools stay declared but unused); "required" is OpenAI's spelling for the any mode.

xAI exposes an OpenAI-compatible chat completions API. Keep the OpenAI SDK, point base_url at https://api.x.ai/v1, authenticate with your xAI key (conventionally XAI_API_KEY) as the bearer token, and pass a Grok model id such as grok-4.

Send a search_parameters object on the chat completions request. Its mode field turns search on, off, or leaves the decision to the model; sources selects web, x, news or rss. The response carries a citations array, and each source used is billed on top of tokens.

On xAI's lighter Grok reasoning models you set reasoning_effort to "low" or "high" to trade depth against latency and cost. The trace comes back in the message as reasoning_content, and thinking tokens are billed as completion tokens with a reasoning_tokens breakdown in usage.

xAI documents presence_penalty, frequency_penalty, stop and reasoning_effort as unsupported on the grok-4 family: sending them fails the request rather than being ignored. Compatibility is per model, so a shared parameter struct must be filtered before dispatch.

Use the OpenAI-compatible endpoint when portability across vendors is the priority and you mostly need plain chat. Reach for xai-sdk when xAI-specific capabilities — live search, deferred sampling, typed helpers — are central enough that untyped passthrough fields become a liability.

Point the OpenAI client at base_url https://api.deepseek.com, pass your DeepSeek key as api_key, and use a DeepSeek model id such as deepseek-chat. The request shape, streaming and response objects stay unchanged.

deepseek-chat selects DeepSeek's general-purpose V3 chat line; deepseek-reasoner selects the R1-style reasoning line that thinks before answering. Both are moving aliases pointing at whatever generation is current, not frozen model versions.

deepseek-reasoner puts its chain of thought in message.reasoning_content and the user-facing answer in message.content. Both come back in one ordinary chat-completion response. Show users content; treat reasoning_content as diagnostic text you log, collapse, or discard.

DeepSeek's JSON output mode constrains the model to emit syntactically valid JSON, nothing more. It does not validate against a schema you supply, so field names, types and required keys remain your responsibility to prompt for and then verify.

DeepSeek splits a request's input tokens into two billed buckets: prompt_cache_hit_tokens, served from its context cache at a steeply discounted input rate, and prompt_cache_miss_tokens, billed at the full input rate. The two always sum to prompt_tokens.

apply_chat_template() turns a list of {role, content} messages into the exact prompt string the checkpoint was fine-tuned on, using the Jinja template shipped alongside that model's tokenizer — so the special tokens come from the model, not from your code.

Llama weights are downloadable and usable commercially, but the Llama Community License imposes field-of-use limits through an acceptable-use policy and singles out very large operators for extra permission. Those restrictions fail the Open Source Definition, so the accurate term is open-weight.

Weights cost bytes-per-parameter times parameter count: about 140 GB for 70B at FP16, about 70 GB at 8-bit, and roughly 35-40 GB at 4-bit once block scales are counted. KV cache and activations are extra and are not shrunk by weight quantization.

Ollama wraps llama.cpp for local, single-user use: one command pulls a quantized model and runs it on a laptop, CPU included. vLLM is a GPU server built for concurrent production traffic, using continuous batching and a paged KV cache.

Llama 2 wraps each user turn in [INST] ... [/INST] plain-text markers and nests the system prompt inside the first one with <<SYS>>. Llama 3 drops both, using real special tokens: a role header per turn, closed by <|eot_id|>.

Gemma 3 ships at 1B, 4B, 12B and 27B parameters, each in a pretrained and an instruction-tuned variant. The 4B, 12B and 27B checkpoints accept images alongside text; the 1B is text-only and has a much shorter context window.

Gemma weights ship under Google's own Gemma Terms of Use plus a prohibited-use policy, not an OSI-approved open-source licence. Commercial use is free with no user or revenue threshold, but you accept use restrictions and must pass those terms to anyone you distribute a derivative to.

Gemma is trained on an exact turn format — start_of_turn markers with user and model roles, closed by end_of_turn, and a single leading BOS token. Serving prompts that differ from the training format, or that duplicate BOS, puts the model off-distribution and quality drops silently.

At bfloat16 you need roughly 2 GB per billion parameters, so 24 GB holds 4B or 12B but not 27B's ~54 GB. Google's quantisation-aware-trained int4 checkpoints shrink 27B to roughly 15 GB, which fits — at the cost of some quality and nearly all your KV-cache headroom.

Decide by how you deliver the model. If you serve it yourself, Gemma's terms cost you almost nothing. If you ship weights inside a product, its use restrictions and pass-along duties follow every copy, while an Apache-2.0 model imposes only attribution and carries a patent grant.

Point the OpenAI client at Model Studio's compatible-mode base URL — https://dashscope-intl.aliyuncs.com/compatible-mode/v1 for Singapore, https://dashscope.aliyuncs.com/compatible-mode/v1 for Beijing — and pass your DASHSCOPE_API_KEY as api_key. Model ids are Qwen names such as qwen-plus.

Qwen ships task-specific lines beside its general chat models: Qwen3-Coder for code and agentic coding, Qwen3-VL for images and video, Qwen3-Embedding and Qwen3-Reranker for retrieval, and the Omni/Audio line for speech input.

vllm serve Qwen/Qwen3-8B pulls the weights from Hugging Face and starts an OpenAI-shaped HTTP server on port 8000. Point any OpenAI client at http://localhost:8000/v1, pass any non-empty API key, and set model to the served name.

Qwen3 is the generation, 30B the total parameter count, A3B the roughly 3B parameters active per token in a Mixture-of-Experts layout, Instruct the post-trained chat checkpoint (versus Base or Thinking), and 2507 the July 2025 refresh of that checkpoint.

Qwen uses ChatML: every message is rendered as <|im_start|>role, a newline, the content, then <|im_end|>. Generation starts from a trailing <|im_start|>assistant header, and <|im_end|> is the stop token. Wrong formatting means the model rambles or never stops.

Anthropic publishes each Claude Opus release as a dated snapshot ID such as claude-opus-4-1-20250805, plus a floating alias such as claude-opus-4-1 that resolves to the newest snapshot in that line. Production should pin the dated snapshot.

The same Opus release carries a different identifier on each platform: claude-opus-4-1-20250805 on Anthropic's own API, anthropic.claude-opus-4-1-20250805-v1:0 on Amazon Bedrock, and claude-opus-4-1@20250805 on Google Vertex AI.

Treat it as a dependency upgrade with a hard deadline: inventory every place the retired Opus snapshot ID appears, run a golden-set eval against the successor snapshot, fix prompt and parsing regressions, canary the new pin, and finish before the shutdown date, because retired IDs return errors rather than falling back.

Current Claude Sonnet models are called by undated alias strings such as claude-sonnet-5 and claude-sonnet-4-6. The ID is complete as written; appending a -YYYYMMDD snapshot suffix produces an unknown model and the request fails.

claude-sonnet-5 removed temperature, top_p and top_k — sending any of them returns a 400. The fixed thinking budget (budget_tokens) is gone too. Depth and token spend are set with output_config.effort, alongside adaptive thinking.

On claude-sonnet-5 the thinking parameter's display field defaults to omitted, so thinking blocks stream with empty text. The previous generation defaulted to summarized. Set thinking to {"type": "adaptive", "display": "summarized"} to get readable reasoning back.

gpt-image models always return base64-encoded bytes in the response's data[].b64_json field. There is no hosted URL to fetch and no response_format choice, so the caller decodes the string and stores or serves the bytes itself.

The endpoint path stays the same, but the parameter vocabulary changes: new size values, a low/medium/high quality scale instead of standard/hd, no style and no response_format, no revised_prompt in the response, and new background, output_format and moderation options.

The mask is an optional PNG with an alpha channel, uploaded alongside the image and matching its dimensions. Fully transparent pixels mark the region the model may repaint; opaque pixels are preserved. Omit it and the whole image is re-rendered.

A blocked image request returns HTTP 400 with a moderation_blocked error, not a rate-limit or server error. It is not retryable: surface a clear refusal, invite the user to rephrase, log the attempt for abuse review, and never loop on backoff.

gpt-image calls are billed in tokens, not at a flat per-image price. Text prompt tokens, any input image tokens, and image output tokens are counted separately; output tokens scale with the requested size and quality tier.

Re-encode to mono 16 kHz compressed audio first, since Whisper resamples to 16 kHz anyway. If the file is still over the limit, split it at silence boundaries, transcribe each chunk, then add each chunk's start offset to its timestamps before merging.

Transcription writes down what was said in the language it was spoken in. Translation always outputs English, whatever the source language — there is no target-language parameter, because the model was only trained to translate into English.

It supplies prior text as decoder context, biasing spellings, jargon and punctuation style toward what the prompt contains. It is not an instruction field — Whisper does not follow commands in it — and only roughly the last 224 tokens are used.

Send timestamp_granularities: ["word"] on the transcription request and set response_format to verbose_json; any other response format rejects the granularity option. The response then carries a words array with a start and end time per word, alongside the usual segment list.

This is Whisper's known hallucination on non-speech: with no acoustic evidence the decoder falls back on language priors and loops or emits training-set boilerplate. Strip silence with voice-activity detection before transcribing, and drop segments whose no_speech_prob is high or compression_ratio suggests repetition.

OpenRouter runs on prepaid credits: you top up one balance and each call deducts the chosen provider's own per-token price. GET /api/v1/models publishes every model's pricing object, quoted in US dollars per single token.

Set the OpenAI client's base URL to https://openrouter.ai/api/v1 and send your OpenRouter key as a bearer token. Models are addressed by a vendor-prefixed slug such as anthropic/claude-sonnet-4.5, so one key reaches every vendor.

Send "usage": {"include": true} in the chat completion request body. The response's usage object then carries cost in credits alongside token counts from the serving model's own tokenizer, with cost_details breaking out any upstream charge.

OpenRouter treats models as an ordered fallback list: it tries the first entry and, if that model errors or is unavailable, retries the next one inside the same request. You are billed for whichever model actually answered.

provider.order lists upstream providers to try in priority order for the chosen model. allow_fallbacks, true by default, decides what happens when none of them can serve: leave it on and OpenRouter tries other providers, set it false and the request fails instead.

GitHub Models is a hosted catalog and playground of models you can call with your GitHub token. It gives free, rate-limited inference for prototyping, so you can compare models from several publishers before opening any paid provider account.

Send a GitHub token as an HTTP bearer credential. Locally that is a fine-grained personal access token carrying the Models read permission; inside GitHub Actions it is the workflow's built-in GITHUB_TOKEN, after the job declares the models: read permission.

Store prompts as .prompt.yml files in the repository, grant the job models: read so the workflow token can call inference, install the gh-models CLI extension, and run gh models eval on those files so prompt changes are reviewed and scored like code.

Two client styles work: the Azure AI Inference SDK, and any OpenAI-compatible client pointed at the GitHub Models base URL. With the OpenAI SDK you override base_url, pass the GitHub token as api_key, and use publisher-qualified model ids such as openai/gpt-4o-mini.

Treat 429 as quota, not a bug: GitHub Models throttles per account on requests per minute and per day, tokens per request and concurrency, with tighter limits on larger models. Back off using the wait the error states, and move sustained load to a paid endpoint.