skip to content

When does a Hugging Face Inference Endpoint need a custom handler.py, and what must that file implement?

level: middleimportance: nice to knowfreq 5%

answer

  1. logic the stock server lacks
  2. a file in the model repository
  3. class EndpointHandler
  4. __init__ gets a path, __call__ gets the body
  5. requirements.txt beside it

basics

~20 s

Use a handler.py when the model needs custom pre- or post-processing or a task the built-in serving lacks. It defines class EndpointHandler: init(path) loads the model once, and call(data) receives the request body with an inputs key and returns JSON-serializable output.

solid answer

~40 s

A custom handler fits a Transformers, Sentence Transformers or Diffusers model that needs logic the stock server does not have: extra request fields, business rules, chaining steps, or a non-standard task. You add `handler.py` to the model repository with a class `EndpointHandler`: `__init__(self, path="")` runs once at startup with the local path of the repository files, so load the model there; `__call__(self, data)` runs per request with the JSON body as a dict that always contains `inputs`, and returns a list or dict that is serialized back. Extra packages go in `requirements.txt`. The Inference Toolkit engine finds a valid `handler.py` and uses it whatever Task you selected. It is a fallback, not as production-ready as vLLM or SGLang for chat LLMs, TEI for embeddings, or llama.cpp for GGUF files.

code

python · 14 lines
python
from typing import Any, Dict, List
from transformers import pipeline


class EndpointHandler:
    def __init__(self, path: str = ""):
        # runs once at startup; path is the local copy of the repository
        self.classifier = pipeline("text-classification", model=path)

    def __call__(self, data: Dict[str, Any]) -> List[Dict[str, Any]]:
        inputs = data.pop("inputs", data)        # the body always carries "inputs"
        min_score = data.pop("min_score", 0.0)    # an extra field your clients may send
        predictions = self.classifier(inputs)
        return [p for p in predictions if p["score"] >= min_score]

go deeper

for a junior

Recall the file and class names, handler.py and EndpointHandler, and that it lives in the model repository with a requirements.txt for extra packages.

for a middle

Explain the two methods: init gets the local repository path and loads once, call gets the body dict with inputs and returns JSON-serializable output.

for a senior

Decide between a handler on the Inference Toolkit, a dedicated engine such as vLLM, SGLang, TEI or llama.cpp, and a custom container, and pin the revision so handler pushes are deliberate.

for a principal

Weigh where custom logic should live: beside the model in a handler, in a custom container, or in the calling service, given batching, ownership and how changes are reviewed and rolled out.

## Where a custom handler fits among the engines Creating an Inference Endpoint includes choosing the **inference engine** that runs inside the container. The Endpoints docs offer: | Engine | Use it for | |---|---| | **vLLM** or **SGLang** | chat and other generative LLMs; the docs' recommendation for new LLM endpoints | | **TGI** | existing endpoints only: maintenance mode since late 2025, with a migration guide to vLLM | | **TEI** (Text Embeddings Inference) | embedding models | | **llama.cpp** | models in GGUF format | | **Inference Toolkit** | Transformers, Sentence Transformers and Diffusers models the engines above do not cover, and any **custom handler** | | **custom container** | anything else, with your own server and image | The Inference Toolkit wraps a model from those libraries in a light web server. The docs call it good for testing and demos but not as production-ready as the dedicated engines. Its most distinctive feature is the **custom handler**: a `handler.py` that replaces the default request handling with your own Python. Reach for one when: - the request carries **extra fields** that change behaviour (a date, a threshold, a tenant id); - you need **pre-processing** (cleaning, chunking, fetching a file) or **post-processing** (filtering labels, reshaping output) next to the model; - the model's task is **not one the toolkit serves** as-is, or several models must run in one call. If you instead need a different web framework, system packages or full control of the server, a custom container is the better tool; for a standard chat LLM, a custom handler gives up the batching a dedicated engine provides. ## The contract The file is `handler.py` in the model repository on the Hub, and it must define a class named **`EndpointHandler`** with two methods: 1. **`__init__(self, path="")`** runs once when the endpoint starts. `path` is a string with the local path of the repository's files, so this is where you load the model, tokenizer or pipeline. Anything slow belongs here, not in the request path. 2. **`__call__(self, data)`** runs on every request. `data` is the request body as a Python dict, and it **always contains the key `inputs`**; any other keys are fields your clients send. The return value, a list or a dict, is serialized and sent back as the response. Supporting files follow the same repository: - **`requirements.txt`** lists extra Python packages to install in the container. - The **Task** shown in the dashboard becomes **Custom**: when the endpoint is created, the service checks for a valid `handler.py` and uses it for serving whatever Task you selected. - Pushing to the repository needs a token with **write** access to it. ## Developing it safely 1. **Test locally first.** Import the class, construct it with `path="."` inside a clone of the repository, and call it with sample payloads, including the fields you made optional. 2. **Read `inputs` defensively.** The documented example uses `data.pop("inputs", data)` and `data.pop("field", None)` so missing optional fields do not raise. 3. **Keep heavy work in `__init__`.** Loading a model per request multiplies latency and memory. 4. **Pin what the endpoint deploys.** The endpoint's advanced settings accept a commit revision, so every replica, including ones started later by scaling, downloads the same reviewed `handler.py` rather than whatever is newest on the branch. ## How a request reaches the handler Clients call the endpoint URL with an HTTP `POST`, a JSON body and, unless the endpoint is Public, an `Authorization: Bearer` header carrying a Hugging Face token. The proxy checks the token against the endpoint's access level, then the toolkit parses the body and passes it to `__call__` as a dict. For the handler shown with this question, a body such as ```json {"inputs": "The refund never arrived", "min_score": 0.8} ``` reaches `__call__` as a dict: `inputs` is the text to classify and `min_score` is the extra field the handler defined. Whatever `__call__` returns becomes the JSON response, so the response schema is entirely yours to document for callers. ## What the handler does not change The handler controls only the inside of a request. Instance type, access level, autoscaling and scale-to-zero are still endpoint settings, and the in-process `pipeline()` you call from `__init__` behaves exactly as it does in a notebook, with its own task names and defaults.

  • Your handler imports a package the Endpoint container does not have. How do you add it?
    List it in a `requirements.txt` in the same model repository as `handler.py`. The Endpoints docs require custom handler dependencies to be listed there, and the documented workflow installs the same file locally so the handler can be tested before pushing.
  • Why not wrap a chat LLM in a custom handler that calls `generate` directly?
    Because the Inference Toolkit is a light fallback server: the Endpoints FAQ notes it might not batch requests, while vLLM and SGLang are built to serve many concurrent generations. Use one of those engines for the LLM and keep custom logic in the calling service, or in a custom container if it must live beside the model.

  • The handler can be any module name as long as it defines a predict function.
  • __init__ receives the request payload and should load the model on demand.
  • The handler only runs if the endpoint's Task is set to Custom by hand.
  • Extra packages are installed with pip calls inside __call__.
  • A custom handler is the best way to serve a production chat LLM.