skip to content

Inference Providers & Endpoints

Inference Providers route a Hub model id to a partner through one OpenAI-compatible router and one HF token; Inference Endpoints are dedicated deployments. Interviewers probe cost and cold starts.

on this pageshow

questions

6

What is the difference between Hugging Face Inference Providers and Inference Endpoints, and how is each one billed?

level: juniorimportance: must knowfreq 14%

answer

  1. shared capacity versus your own
  2. one token, many partner providers
  3. per request at the provider's rate
  4. instance hourly rate, charged per minute
  5. paused or scaled to zero stops the meter

basics

~20 s

Inference Providers route a Hub model id to a partner that already serves it, billed per request at that provider's own rate. Inference Endpoints deploy the model on a dedicated instance you choose, billed for the time it runs.

solid answer

~40 s

**Inference Providers** is serverless: you send a Hub model id with one Hugging Face token and the router forwards it to a partner (or Hugging Face's own `hf-inference`) that already serves the model. You pay per request at the provider's rate with no Hugging Face markup, so chat usage is effectively priced per token, and monthly credits apply. **Inference Endpoints** is a dedicated deployment of a Hub repo on an instance you pick, with your choice of engine, access level and autoscaling. It bills the instance's hourly rate, charged by the minute while the endpoint is initializing or running; paused or scaled-to-zero endpoints cost nothing. Providers fit prototypes and spiky traffic on models a partner serves; Endpoints fit a private fine-tune, a pinned revision, custom code, a private network path or a steady load.

go deeper

for a junior

Recall the one-line split: Providers send your request to a partner that already runs the model; Endpoints run the model on an instance that is yours. Then say how each is charged.

for a middle

Explain what each bill measures: the provider's per-request rate passed through with no markup, against an instance's hourly rate charged per minute while initializing or running.

for a senior

Show the triggers for moving off Providers: a model no partner serves, a pinned revision, custom handling, a private network path, or traffic steady enough that a busy replica beats per-token pricing.

for a principal

Frame the choice as shared versus owned capacity: who carries cold starts, who controls the model revision and region, and when to re-measure token volume before committing to dedicated hardware.

## Two hosted products on one Hub Hugging Face offers two ways to run a Hub model without operating the serving stack yourself, and they answer different questions. **Inference Providers** answers "can I call this model right now?": it is a serverless routing layer in front of partner inference companies and Hugging Face's own `hf-inference` service. **Inference Endpoints** answers "can I run this exact model as my own service?": it is a managed, dedicated deployment of a Hub repository on hardware you pick. The Endpoints FAQ draws the same line: Providers is for exploring and evaluating models through one consistent API, Endpoints is for deploying your models on managed infrastructure. Both start from a **Hub model id** such as `openai/gpt-oss-120b`, and both authenticate with a Hugging Face **user access token**, which is why they are easy to confuse in conversation. ## Inference Providers: someone else's warm capacity - **No deployment step.** You send a request naming a Hub model id, and Hugging Face's router forwards it to a partner that already serves that model. If no partner serves it, Providers cannot run it; the Hub marks served models with an inference status of `warm`, and `hf models ls --warm` lists them. - **One token for every partner.** A fine-grained token with the "Make calls to Inference Providers" permission covers all of them, so you need no account with the partner itself. - **Shared, serverless capacity.** You never see an instance, a replica count or a scaling setting; the partner owns all of that. - **Three client paths.** An OpenAI-compatible chat endpoint at `https://router.huggingface.co/v1`, `InferenceClient` in `huggingface_hub`, and `InferenceClient` in the JavaScript package `@huggingface/inference`. - **Data handling.** Hugging Face states it does not store request bodies or responses when routing, and keeps logs for up to 30 days. ## Inference Endpoints: a deployment you own - **You pick the hardware.** Cloud (AWS, Azure or Google Cloud), region, accelerator (CPU, GPU or AWS Inferentia) and an instance such as one NVIDIA L4 or eight H200s. - **You pick the engine.** vLLM, SGLang, TGI, TEI, llama.cpp, the Inference Toolkit with an optional custom `handler.py`, or your own container. - **You pick who may call it** (Private, Authenticated or Public) and how it scales, including scale-to-zero after an idle period. - **It serves any repository you can access,** including a private fine-tune, optionally pinned to a commit revision in the advanced settings. ## How the two are billed | | Inference Providers | Inference Endpoints | |---|---|---| | Unit | each request, at the serving provider's own rate | instance uptime, at the instance's hourly rate | | Granularity | per request; chat models are priced per token | shown per hour, charged per minute | | When you pay | only when you send traffic | while the endpoint is initializing or running | | Idle cost | none | none once paused or scaled to zero; full rate while a replica is up | | Credits | monthly credits apply to routed requests | PRO, Team and Enterprise compute credits can be spent here too | Three details make the table precise: 1. **No markup on Providers.** Hugging Face passes the provider's rate through unchanged. The `:cheapest` routing policy ranks providers by price per output token, which is how chat usage is metered. 2. **`hf-inference` is metered by compute time.** Hugging Face's own provider, formerly the serverless Inference API, charges compute time multiplied by the hardware price: a 10-second request on a GPU costing $0.00012 per second is billed $0.0012. 3. **Endpoints bill the replica, not the request.** An endpoint that keeps one minimum replica bills every minute of the day whether it serves one request or a million. Pausing or scaling to zero stops the meter, at the price of a cold start on the next request. ## Choosing between them Start on Providers when a partner already serves the model and the goal is to ship a feature without running a GPU: prototypes, evaluations, spiky or low-volume traffic. Move to an Endpoint when you need something Providers cannot give: - a model no partner serves, such as your own fine-tune in a private repo; - a pinned revision, custom pre- or post-processing, or a specific engine configuration; - a private network path (AWS PrivateLink) or a chosen region; - capacity that is yours alone rather than shared with every other caller. A steady workload is the other trigger. Once traffic keeps a replica busy most of the day, a fixed hourly rate can undercut per-token pricing; working out exactly where that line falls is a cost-per-token calculation, not a product feature, and it is the reason to measure your real token volume before migrating.

  • Can Inference Providers run any model on the Hub?
    No. Providers only reach models that at least one partner, or `hf-inference`, has mapped and serves. `model_info(repo_id, expand="inference")` returns `warm` for a served model and `None` otherwise, and `hf models ls --warm` lists every served model. A private fine-tune that no partner serves needs an Inference Endpoint instead.
  • Does an Inference Endpoint cost anything while it is scaled to zero?
    No compute charge: billing covers the time the endpoint is initializing or running, charged per minute, and a paused or scaled-to-zero endpoint costs nothing. The trade-off is a cold start of possibly several minutes when the next request arrives, during which the proxy answers 503 unless the caller asks it to wait.

Inference Providers is a taxi: you pay per ride and never own the car. An Endpoint is a leased car: you pay for every hour it is yours, driven or parked, unless you hand the keys back by pausing or scaling to zero.

  • Inference Providers reserves a GPU for my account while I use it.
  • Inference Endpoints are billed per token, like the serverless router.
  • Hugging Face adds its own markup on top of each provider's price.
  • Any model on the Hub can be called through Inference Providers.
  • An endpoint with one minimum replica only bills for the minutes it serves requests.
open as a page

How do you point existing OpenAI SDK chat code at Hugging Face Inference Providers, and how do you control which provider serves it?

level: middleimportance: must knowfreq 11%

basics

~20 s

Set the OpenAI client's base URL to https://router.huggingface.co/v1 and its API key to a Hugging Face token, and pass the Hub model id. A suffix picks the provider: none means :fastest, or use :cheapest, :preferred or a provider name.

open as a page

With Hugging Face's InferenceClient, how does the key you pass decide whether a request goes through Hugging Face's router and who bills it?

level: middleimportance: should knowfreq 6%

basics

~20 s

A Hugging Face token (hf_ prefix) sends the call through Hugging Face's router and bills your Hugging Face account at the provider's rate. A provider's own key makes InferenceClient call that provider directly, and the provider bills you.

open as a page

A nightly job calls a Hugging Face Inference Endpoint that scales to zero, and its first requests fail with HTTP 503 — what is happening and how do you fix it?

level: seniorimportance: should knowfreq 8%

basics

~20 s

The endpoint scaled to zero after its idle window, so the first request starts a cold replica and the proxy returns 503 until it is ready. Send X-Scale-Up-Timeout so the proxy waits, resume explicitly, or keep a replica up.

open as a page

When does a Hugging Face Inference Endpoint need a custom handler.py, and what must that file implement?

level: middleimportance: nice to knowfreq 5%

basics

~20 s

Use a handler.py when the model needs custom pre- or post-processing or a task the built-in serving lacks. It defines class EndpointHandler: init(path) loads the model once, and call(data) receives the request body with an inputs key and returns JSON-serializable output.

open as a page

A script created a Hugging Face Inference Endpoint with huggingface_hub's create_inference_endpoint and no type argument — who can call it, and how do you restrict it?

level: seniorimportance: nice to knowfreq 4%

basics

~20 s

In huggingface_hub 2.1, type defaults to authenticated, so anyone with a Hugging Face account can call it with their own token. Recreate it with type private, the dashboard's default, adding AWS PrivateLink if traffic must stay off the internet.

open as a page