skip to content

Workers AI Inference

Workers AI runs catalogue models through env.AI.run, from text generation to embeddings and speech. Interviewers ask how streaming, neuron pricing and model choice shape a feature.

on this pageshow

explore

questions

6

In Workers AI, how does a Worker call a model through the AI binding with env.AI.run(), and what comes back?

level: juniorimportance: must knowfreq 45%

answer

  1. a binding, not a token
  2. catalogue id as the first argument
  3. inputs shaped by the model's task
  4. response field for text generation

basics

~20 s

A Worker declares an ai binding in its Wrangler file and calls env.AI.run(model, inputs) with an @cf/ catalogue id; the promise resolves to that task's output, such as { response } for text generation or { shape, data } for embeddings.

solid answer

~40 s

The Wrangler file declares `"ai": { "binding": "AI" }`, which puts a Workers AI client on `env.AI`; the binding carries the account's identity, so the Worker holds no API token. `env.AI.run(model, inputs, options?)` takes a catalogue id such as `@cf/meta/llama-3.3-70b-instruct-fp8-fast`, an inputs object shaped by the model's task (`messages` or `prompt` for text generation, `text` for embeddings) and an optional third object for how the call is made, such as `gateway`. Without `stream: true`, a text-generation call resolves to an object whose `response` field holds the text, with `usage` token counts beside it; an embeddings call resolves to `{ shape, data }`. Code outside Cloudflare reaches the same models through the REST endpoint `/accounts/{account_id}/ai/run/{model}` with an API token, and gets the output wrapped in `result`.

code

javascript · 13 lines
javascript
export default {
  async fetch(request, env) {
    const { ticket } = await request.json();
    const result = await env.AI.run("@cf/meta/llama-3.3-70b-instruct-fp8-fast", {
      messages: [
        { role: "system", content: "Summarise this support ticket in three sentences." },
        { role: "user", content: ticket },
      ],
      max_tokens: 200,
    });
    return Response.json({ summary: result.response, usage: result.usage });
  },
};

go deeper

for a junior

Recall the binding entry, the env.AI.run(model, inputs) call with an @cf/ id, and that text generation's text sits in the response field.

for a middle

Explain the three arguments: model id, task-shaped inputs and call options, and how the result's shape changes with stream, returnRawResponse or queueRequest.

for a senior

Show you keep the REST token on servers, read usage to estimate neurons, and treat a misspelt or retired model id as an error to catch, not a fallback.

for a principal

Frame the binding as the default path for Workers and the REST API as the escape hatch, and decide where model ids live so a catalogue change is one edit.

## What the AI binding is **Workers AI** runs models from Cloudflare's catalogue on Cloudflare's GPUs. A Worker reaches it through a **binding**: a Wrangler file entry that places a client object on the `env` parameter of the Worker's handlers. ```json { "name": "support-tickets", "main": "src/index.js", "compatibility_date": "2026-10-10", "ai": { "binding": "AI" } } ``` The binding name is yours to pick; `AI` is the convention, giving `env.AI`. Because the binding carries the account's identity, the Worker stores **no API token** for Workers AI, and there is nothing to leak from its code or its secrets. Two side effects are worth knowing: - `ai` is one of the Wrangler keys a named environment does not inherit, so a staging environment declares it again; - there is no local simulator for this binding, so calls made during local development still reach Cloudflare's service and count toward the account's limits. ## Calling `env.AI.run()` `env.AI.run(model, inputs, options)` returns a promise and takes up to three arguments: 1. **model**: a catalogue id string such as `@cf/meta/llama-3.3-70b-instruct-fp8-fast` or `@cf/baai/bge-base-en-v1.5`. The id is exact. A misspelt or retired id does not fall back to another model: the error table lists `5007` (No such model) and `3042` (Invalid model ID). 2. **inputs**: an object whose shape the model's **task** decides. Text generation takes `prompt` or a `messages` array, plus parameters such as `max_tokens`, `temperature` and `stream`. Text embeddings take `text`, a string or an array of strings. Speech recognition takes `audio`. 3. **options** (optional): how the call is made rather than what the model sees: `gateway` to send the call through an AI Gateway, `extraHeaders`, `returnRawResponse`, `queueRequest` for the asynchronous Batch API, and `rejectIfBusy` to fail fast when capacity is short. Each model's page, or `wrangler ai models schema <model>`, prints the exact input and output schema, which is the reference when a call returns a validation error. ## What comes back, by task | Call | Example model | Resolves to | |---|---|---| | Text generation | `@cf/meta/llama-3.3-70b-instruct-fp8-fast` | `{ response, usage }`, plus `tool_calls` when tools were offered | | Text generation with `stream: true` | the same | a `ReadableStream` of server-sent events | | Text embeddings | `@cf/baai/bge-base-en-v1.5` | `{ shape, data, pooling }` | | Any call with `returnRawResponse: true` | any | a `Response` object | | A batch with `queueRequest: true` | models with batch support | an acknowledgement carrying a `request_id` to poll | `usage` reports `prompt_tokens`, `completion_tokens` and `total_tokens`, which is the raw material for estimating the call's cost in neurons. ## Calling the same models from outside a Worker Code that does not run on Workers (a nightly job, a backend elsewhere) uses the **REST API**: - `POST https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/run/{model}` with `Authorization: Bearer <token>` and the same inputs as a JSON body; - the token is a Cloudflare API token; one made without the dashboard's Workers AI template needs both `Workers AI - Read` and `Workers AI - Edit`; - the model's output arrives inside the API envelope, `{ "result": { ... }, "success": true, "errors": [], "messages": [] }`, so the text is at `result.response`; - a Chat Completions-compatible surface also exists under `/ai/v1/` (for example `/ai/v1/chat/completions`), for clients built around that request shape. The token belongs on a server. A browser that needs a summary calls your Worker, and the Worker calls the binding. ## Mistakes interviewers listen for - Storing an API token as a Worker secret to call the binding: the binding needs none. - Treating a text-generation result as a string: the text is in `response`. - Putting call options inside the inputs: `gateway` and `rejectIfBusy` go in the third argument, while `stream` and `max_tokens` are model inputs. - Expecting the REST API to return the bare model output: it is wrapped in `result`. ## Handling failures in the ticket Worker The promise from `env.AI.run()` rejects when Workers AI cannot serve the call, so the summary endpoint wraps it in `try`/`catch` and maps the failure to a status the caller understands. The error table is the guide to which failures are worth a retry: | Internal code | HTTP | Meaning | Sensible reaction | |---|---|---|---| | `5007` | 400 | no such model or task | a bug: fix the model id | | `3036` | 429 | daily free neuron allocation used up | a budget event: alert, do not retry | | `3040` | 429 | capacity temporarily exceeded | retry with backoff | | `3007` | 408 | request timeout | retry once | Logging the internal code beside the HTTP status keeps the two 429s apart, which matters because one needs a retry and the other needs a plan or budget change.

  • Why does a Worker not need an API token to call Workers AI, while a script elsewhere does?
    The `ai` binding is attached to the Worker by the platform and carries the account's identity, so `env.AI.run()` is authorised by the binding itself. A script outside Cloudflare has no binding, so it authenticates to the REST API with a Cloudflare API token that holds Workers AI permissions. Keeping that token server-side matters: in a browser it would let anyone spend the account's neurons.
  • How do you find out exactly which inputs a Workers AI model accepts?
    Read the model's page in the catalogue or run `wrangler ai models schema <model>`, which prints its input and output JSON schema. `wrangler ai models list` with `--task` or `--search` narrows the catalogue first. The schema shows, for instance, whether a text model takes `prompt`, `messages` or both, and its `max_tokens` default.
  • What does passing a gateway option to env.AI.run() change?
    A third argument such as `{ gateway: { id: "support" } }` sends the same Workers AI call through the named AI Gateway, which adds its logging, analytics and optional caching; `env.AI.aiGatewayLogId` then holds the log entry's id. The model, the inputs and the shape of the result do not change.

  • The Worker must keep a Workers AI API token as a secret to call env.AI.run().
  • A text-generation call to env.AI.run() resolves to a plain string.
  • An unknown model id quietly falls back to a default catalogue model.
  • The REST API returns the bare model output with no result wrapper.
  • Options such as gateway belong inside the inputs object.
open as a page

In Workers AI, how do you embed support tickets with a @cf/baai/bge model, and what about its output must stay fixed?

level: middleimportance: should knowfreq 30%

basics

~20 s

Call env.AI.run("@cf/baai/bge-base-en-v1.5", { text: [...] }) with up to 100 texts; it returns { shape, data } with one 768-number vector per text. The model and its pooling must stay fixed, because vectors from another model or pooling cannot be compared.

open as a page

In Workers AI, how do you stream a text-generation model's draft reply to the browser, and what does stream: true change?

level: middleimportance: should knowfreq 35%

basics

~20 s

Set stream: true in the inputs of env.AI.run(); it then resolves to a ReadableStream of server-sent events, which the Worker returns as a Response with content-type text/event-stream so the browser shows the draft as it is generated.

open as a page

On the Workers Free plan, a Workers AI ticket summariser fails with 429 errors every afternoon and recovers next day; what limit is it hitting, and how do you fix it?

level: seniorimportance: should knowfreq 28%

basics

~20 s

It is spending the daily free allocation of 10,000 neurons: Workers AI then returns 429 with internal code 3036 until the reset at 00:00 UTC. Move to Workers Paid, which bills usage beyond 10,000 at $0.011 per 1,000 neurons, or cut neurons per call.

open as a page

As the lead of a support-ticket product, how do you decide whether summaries and draft replies run on Workers AI or on an external model provider behind AI Gateway?

level: principalimportance: should knowfreq 20%

basics

~20 s

Decide per workload: high-volume summaries and embeddings usually fit Workers AI's catalogue models, billed in neurons and called through a binding, while draft replies go wherever an evaluation on real tickets says quality justifies the cost, with every call routed through one AI Gateway.

open as a page

In Workers AI, how do you run a LoRA adapter trained on your support tone, and what does Cloudflare require of the adapter?

level: middleimportance: nice to knowfreq 12%

basics

~20 s

Upload the adapter with wrangler ai finetune create against a LoRA-capable base model, then call env.AI.run() on that base with lora set to the finetune's name or id. The adapter is two exactly named files, under 300 MB, rank up to 32.

open as a page