skip to content

Scoring Server & Deployments

mlflow models serve exposes /invocations for dataframe_split, instances or inputs payloads, build-docker images it, and mlflow deployments targets cloud endpoints. Interviewers probe portability.

on this pageshow

explore

questions

6

What request body does MLflow's scoring server accept at POST /invocations, and what does it send back?

level: juniorimportance: must knowfreq 28%

answer

  1. the header picks the parser
  2. a JSON object, never a bare list
  3. exactly one wrapper key
  4. split orientation keeps column order
  5. the reply is wrapped too

basics

~10 s

MLflow's scoring server takes a JSON object with exactly one of dataframe_split, dataframe_records, instances or inputs (plus optional params), or a text/csv or Parquet body, and replies with JSON shaped {"predictions": ...}.

solid answer

~40 s

`POST /invocations` picks its parser from the `Content-Type` header: `application/json`, `text/csv` or `application/vnd.apache.parquet`. A JSON body must be an object holding exactly one of `dataframe_split` (pandas split orientation: `data` plus optional `columns` and `index`), `dataframe_records` (a list of row objects), `instances` or `inputs` (tensor-style input), and it may add a `params` object for a model whose signature declares params. CSV and Parquet bodies become a pandas DataFrame directly and carry no params. With a signature, the server casts the parsed input to the declared types before `predict` runs. The reply is JSON with the output under `predictions`. In MLflow 3.16 this FastAPI app is the only pyfunc serving backend; MLServer support was removed in 3.13.

code

json · 9 lines
json
{
  "dataframe_split": {
    "columns": ["living_area_m2", "bedrooms", "district", "year_built"],
    "data": [
      [84.0, 2, "D04", 1998],
      [132.5, 4, "D11", 2015]
    ]
  }
}

go deeper

for a junior

Recall the four JSON wrapper keys, that exactly one is allowed, the three content types, and that the answer sits under predictions.

for a middle

Explain how the Content-Type selects the parser, why dataframe_split preserves column order, and how a signature casts and checks the parsed input before predict runs.

for a senior

Show where a client goes wrong silently: params ignored on a model without a params schema, a 1.x body rejected after an upgrade, and reading the reply without checking the status code.

for a principal

Weigh JSON against CSV or Parquet for large batch scoring through the HTTP server: payload size and parse cost against losing params, and when batch scoring should not go over HTTP at all.

## What answers at `/invocations` `mlflow models serve -m <model-uri>` loads a model through its **`python_function` (pyfunc)** flavor and starts a small **FastAPI** application under **uvicorn**. The app defines four routes: - `POST /invocations` — scoring. - `GET /ping` and `GET /health` — the same handler, 200 once the model is loaded. - `GET /version` — the MLflow version of the serving process. Since **MLflow 3.13** this built-in app is the only pyfunc serving backend: the deprecated `enable_mlserver` option was removed, so a 2.x setup that served through MLServer gets the built-in server. For a property-valuation estimator, the internal web form sends one listing per request and the nightly batch sends thousands, but both go through the same route and the same rules. ## The Content-Type picks the parser | `Content-Type` | Body | Parsed into | |---|---|---| | `application/json` | a JSON object with one wrapper key | a DataFrame, an array or a dict, depending on the key | | `text/csv` | pandas-style CSV with a header row | a pandas DataFrame | | `application/vnd.apache.parquet` | Parquet bytes | a pandas DataFrame | The only parameter the header may carry is `charset`, and only UTF-8 is accepted, so `application/json; charset=utf-8` is fine while `charset=latin-1` is not. Any other media type is refused with **HTTP 415**. One docs page still shows `application/csv`; the 3.16 source recognises only `text/csv`. The choice is not cosmetic: JSON keeps per-request `params` and nested values, while CSV and Parquet are plain tables whose column types come from the signature or from pandas' own inference. ## The four JSON wrapper keys A JSON body must be an **object** (a top-level list is rejected) and must hold **exactly one** of these keys: - **`dataframe_split`** — pandas `split` orientation: a required `data` list of rows, an optional `columns` list and an optional `index`. The docs recommend it over records because it keeps column order. - **`dataframe_records`** — a list of `{column: value}` objects; the docs warn it is not guaranteed to preserve column ordering. - **`instances`** — tensor-style input in the TF Serving request convention, cast to NumPy arrays. - **`inputs`** — the same convention under another key. A model whose signature was built from Python type hints must receive `inputs`. The two tensor keys are not quite interchangeable for a tabular model: the server converts an `instances` list of row objects into a column dictionary, and when schema enforcement then fails it logs a hint to send `inputs` instead to avoid that conversion. No wrapper key, or two of them, is a **400** whose message lists the allowed fields. That includes the 1.x shape `{"columns": [...], "data": [...]}` at the top level; the 2.0 protocol change moved those fields inside `dataframe_split`. A separate path accepts unwrapped chat, completion and embedding bodies for LLM models; that GenAI surface is outside this answer. ## `params` and the signature Next to the wrapper key, a JSON body may carry **`params`**, an object of inference-time settings passed to `predict(..., params=...)`. They take effect only if the model's signature declares a params schema; for a model logged without one, the server logs a warning and **ignores** them. CSV and Parquet bodies have nowhere to put params. If the model was logged with a **signature**, the server parses the payload against its input schema: CSV columns are read with the declared pandas dtypes, base64 strings become bytes for a binary column, and ISO-8601 strings become datetimes. pyfunc then enforces the schema before the model runs. What the signature records belongs to model packaging; what matters at the server is that a missing required column becomes a 400 instead of a wrong price. ## What comes back A successful call with one of these wrapped or tabular bodies returns **200** with `{"predictions": ...}`: - a NumPy array becomes nested JSON lists; - a pandas DataFrame or Series becomes a list of row objects (records orientation); - anything else is serialised as it is. Errors come back as JSON with `error_code` and `message` fields, so a client should check the status code before reading `predictions`. ## Writing the request by hand 1. Send `Content-Type: application/json`. 2. Wrap the rows in `dataframe_split`, with `columns` named as the signature names them. 3. Put inference settings in `params`, never as extra top-level keys. 4. Read the result from `predictions`, not from the top level of the reply. For the nightly batch, a CSV or Parquet body avoids building a large JSON document, at the cost of losing `params`.

  • Why do the docs steer you to `dataframe_split` rather than `dataframe_records` for tabular models?
    `dataframe_records` is a list of row objects, and the docs warn it is not guaranteed to preserve column ordering. `dataframe_split` sends an explicit `columns` list alongside `data`, so the DataFrame the server builds has the columns in the order you sent. It is also more compact, because column names are not repeated on every row.
  • A client sends `{"dataframe_split": {...}, "params": {"currency": "EUR"}}` to a model logged without a params schema. What happens to `currency`?
    The rows are scored and `currency` has no effect. When a model's signature declares no params schema, MLflow logs a warning that the provided params are being ignored and calls the model with empty params. Nothing in the response says so, which is why the model needs a params schema in its signature before a client relies on params.

  • A JSON list of row objects at the top level is a valid body
  • Sending both dataframe_split and inputs lets the server pick one
  • CSV bodies must be sent as application/csv
  • Predictions come back as a bare JSON array with no wrapper
  • MLflow 3 can still serve through MLServer with enable_mlserver
  • Params on a model without a params schema raise an error

Assumes MLflow 3.16; since 3.13 the built-in FastAPI scoring server is the only pyfunc serving backend because enable_mlserver was removed.

open as a page

What does `mlflow models build-docker --model-uri` put into the image, and how does the container it builds serve requests?

level: middleimportance: should knowfreq 12%

basics

~20 s

mlflow models build-docker --model-uri copies the model into /opt/ml/model and installs its dependencies while the image builds; the container then runs nginx on port 8080 in front of uvicorn workers, one per CPU unless MLFLOW_MODELS_WORKERS is set.

open as a page

When MLflow's scoring server answers POST /invocations with 415, 400 or 504, what does each status tell you to fix?

level: middleimportance: should knowfreq 11%

basics

~20 s

From MLflow's scoring server, 415 means the Content-Type header was refused, 400 means the body, the signature check or the model's own predict call failed, and 504 means scoring outlasted the request timeout, 60 seconds by default.

open as a page

How do you use the `mlflow models predict` CLI to check that a request file scores the way MLflow's scoring server will score it?

level: middleimportance: should knowfreq 8%

basics

~20 s

Run mlflow models predict -m <uri> -i request.json, or -i listings.csv -t csv: it parses the file with the scoring server's own JSON or CSV code inside the model's environment and writes the {"predictions": ...} JSON the server would return.

open as a page

On a fresh VM, why does `mlflow models serve` fail on a missing pyenv or take minutes to start, and what does `--env-manager` change?

level: seniorimportance: should knowfreq 6%

basics

~20 s

mlflow models serve defaults to --env-manager virtualenv: before starting, it builds an environment for the model's recorded Python version and requirements, using pyenv for the Python. uv builds it faster, conda uses conda, and local skips building entirely.

open as a page

How does MLflow's `mlflow deployments` CLI send one logged model to different managed endpoints, and what does its `-t` target select?

level: middleimportance: nice to knowfreq 6%

basics

~20 s

mlflow deployments is a plug-in front end: -t names a target URI whose scheme selects an installed deployment plug-in, and create, update, predict and the other subcommands call that plug-in's client with the same model URI plus target-specific -C settings.

open as a page