skip to content

Experiment Tracking

Runs record params, metrics and artifacts through log_param, log_metric and mlflow.autolog, grouped in experiments on a tracking server. Interviewers ask how you would prove one run beat another.

on this pageshow

questions

6

In MLflow, when do you record a value with log_param rather than log_metric, and what happens if a param is logged twice?

level: juniorimportance: must knowfreq 36%

answer

  1. inputs versus results
  2. written once per run
  3. step and history on metrics
  4. same key, different value refused
  5. tags stay overwritable

basics

~20 s

log_param records a run's configuration once per key; logging that key again with a different value raises "Changing param values is not allowed". log_metric records numeric results with a step and keeps every value, so anything that changes is a metric.

solid answer

~40 s

A param is an input to the run — `n_estimators`, `learning_rate`, the feature window — stored as a string and written once. Logging the same key again with the **same** value is accepted; a **different** value fails with an `MlflowException`, "Changing param values is not allowed". A metric is a numeric output: `mlflow.log_metric("val_rmse", v, step=i)` appends a point (step defaults to 0, timestamp to now), the whole series is kept, and the runs table and `mlflow.search_runs` show the latest point — highest step, then most recent timestamp. So a per-round loss, or a learning rate a scheduler changes, is a metric; a text label that changes, such as a status, is a tag set with `mlflow.set_tag`, which overwrites. `log_params` and `log_metrics` take dicts and batch the calls.

code

python · 16 lines
python
import mlflow

mlflow.set_experiment("store-demand")

with mlflow.start_run(run_name="store-demand-weekly"):
    mlflow.log_params({"max_depth": 6, "learning_rate": 0.05})
    mlflow.log_param("max_depth", 6)  # same value: accepted

    best = float("inf")
    for step, val_rmse in enumerate([12.4, 11.9, 11.7, 11.8]):
        mlflow.log_metric("val_rmse", val_rmse, step=step)  # history kept
        best = min(best, val_rmse)
    mlflow.log_metric("best_val_rmse", best)
    mlflow.set_tag("status", "evaluated")  # tags can be overwritten

    # mlflow.log_param("max_depth", 8)  # would raise: changing param values is not allowed

go deeper

for a junior

Recall the split: params are a run's inputs, written once; metrics are numeric outputs with a step and a kept history. Name the error a changed param raises.

for a middle

Explain what the store does on a repeated param key, how step and timestamp decide which metric point counts as latest, and why search treats params as strings.

for a senior

Show how the rules bite in real scripts: retries that recompute params, schedulers that change values mid-run, rankings that read the last point instead of the best, labels that belong in tags.

for a principal

Weigh which values a team standardises as params, metrics or tags so that runs from many projects stay searchable, and where the write-once rule saves a comparison from silent edits.

## Two kinds of record in an MLflow run An MLflow **run** is one execution of training code — one `python train.py`, or one trial of a sweep. While it is open, the code records small pieces of metadata in a **backend store** (a database) and files in an **artifact store**. Two of the metadata kinds look alike and behave very differently: - a **param** — a configuration value the run was started with; - a **metric** — a numeric value the run produced, possibly many times. Picking the wrong one is the most common first mistake with the tracking API, and MLflow enforces the difference rather than leaving it to convention. ## Params: configuration, written once `mlflow.log_param(key, value)` and its batch form `mlflow.log_params({...})` record inputs such as `max_depth`, `learning_rate` or the name of the feature set a store-demand forecaster was trained on. - The value is **string-ified** before it is stored, so `0.05` becomes `"0.05"`. Built-in backend stores accept param values up to 6,000 characters and keys up to 250. - A key is **written once per run**. Logging the same key again with an **identical** value is a no-op; logging it with a **different** value raises an `MlflowException`: "Changing param values is not allowed. Param with key='…' was already logged with value='…'". `log_params` checks the whole batch and lists every conflicting key. - Because params are strings, `mlflow.search_runs` filters on them with string comparators only — `=`, `!=`, `LIKE`, `ILIKE`, and `IS NULL` / `IS NOT NULL`. A range filter such as `params.learning_rate > 0.01` is not one of them. The write-once rule is what lets anyone trust that a run's param list describes the configuration it actually ran with. Why a run record needs that guarantee is a design question in its own right; the tool simply refuses the overwrite. ## Metrics: numbers with a step and a history `mlflow.log_metric(key, value, step=None, timestamp=None, ...)` and `mlflow.log_metrics({...}, step=...)` record numeric results. - `step` defaults to **0**; `timestamp` defaults to the current time in milliseconds. - Every call **appends** a point. Nothing is overwritten, so the whole curve of a boosting run's validation loss is kept. - `MlflowClient().get_metric_history(run_id, key)` returns every point. The runs table, `run.data.metrics` and `search_runs` show only the **latest** point per key: the one with the highest step, then the most recent timestamp. - Values are floats. A string label is not a metric. "Latest" is not "best": if validation error bottoms out at round 140 and rises to round 199, the run's `metrics.val_rmse` is the round-199 value unless you also log the score you intend to rank on. ## Choosing between param, metric and tag | What you are recording | Call | Can it change within a run? | |---|---|---| | A hyperparameter fixed for the run | `log_param` / `log_params` | No — a different value raises | | A loss or score per round or epoch | `log_metric(..., step=i)` | Yes — every point kept | | A value a scheduler changes during training | `log_metric(..., step=i)` | Yes | | A text label such as a status or owner | `set_tag` / `set_tags` | Yes — the last write wins | | A file: a plot, a CSV of predictions | `log_artifact` / `log_artifacts` | Files are added to the artifact store, not the database | Tags are the escape hatch for labels that legitimately change: the backend store merges a tag on write, so the last value wins. ## A typical training script 1. Open the run with `with mlflow.start_run(run_name="store-demand-weekly"):` so it ends cleanly even on an exception. 2. Log the configuration once with `mlflow.log_params(config)`. 3. Inside the training loop, log per-round values with `mlflow.log_metric("val_rmse", v, step=i)`. 4. After training, log the number you will rank runs on — for example `best_val_rmse` — as its own metric. 5. Attach files such as a residuals plot with `mlflow.log_artifact("residuals.png")`. ## Traps worth naming - Logging a per-epoch value as a param (`loss_epoch_3`) turns one curve into hundreds of keys and loses the step axis. - Recomputing a param on retry — a path containing a timestamp, a seed drawn at call time — makes the second attempt fail with the changing-param error. - Treating `metrics.val_rmse` in the runs table as the run's best value rather than its last one.

  • How do you get a metric's whole curve when the MLflow runs table shows only one number?
    Call `MlflowClient().get_metric_history(run_id, "val_rmse")`, which returns every logged point with its step, timestamp and value. The runs table, `run.data.metrics` and `search_runs` keep only the latest point per key — highest step, then most recent timestamp — so the full series comes from the history call or the run page's chart.
  • A retry wrapper re-runs the param-logging code inside the same MLflow run. Will it fail?
    Only if a value differs. The store compares each key with what is already recorded: an identical value is skipped, a different one raises "Changing param values is not allowed". A retry that recomputes a param nondeterministically — a timestamped output path, a seed drawn at call time — fails on the second attempt, so compute such values once before logging.
  • Why does a filter like params.learning_rate > 0.01 not work in mlflow.search_runs?
    Params are stored as strings, and the search grammar only allows string comparators on them: `=`, `!=`, `LIKE`, `ILIKE`, plus `IS NULL` and `IS NOT NULL`. Numeric comparators apply to metrics. If a team needs range queries on a configuration value, it records that value as a metric as well as a param.

A param is the recipe card written before the oven goes on — once baking starts you cannot amend it, only start another bake. A metric is the oven thermometer's log: a reading every minute, all of them kept, with the latest one shown on the display.

  • log_param overwrites the earlier value, like assigning to a dictionary key
  • log_metric keeps only the last value, so there is no curve to look at
  • Logging a param a second time with the same value raises an error
  • Params are stored as numbers, so params.learning_rate > 0.01 works in search_runs
  • A learning rate changed by a scheduler should be re-logged with log_param each epoch
open as a page

In MLflow, how do you log a nightly sweep's gradient-boosting trials as nested runs and then find the best trial from code?

level: middleimportance: must knowfreq 22%

basics

~10 s

Open a parent with mlflow.start_run, open each trial inside it with mlflow.start_run(nested=True), then call mlflow.search_runs filtered on tags.mlflow.parentRunId and ordered by the validation metric. Search ranks each run by that metric's latest logged value.

open as a page

With mlflow.autolog() enabled for a scikit-learn regressor, what does MLflow record on its own, and what must you still log yourself?

level: middleimportance: should knowfreq 15%

basics

~20 s

On fit(), scikit-learn autologging records the estimator's get_params, scores computed on the training data, class-name tags and the fitted model with a signature. A held-out score your own code computes, and settings outside the estimator, you log yourself.

open as a page

Why do a scheduled job's MLflow runs land in a local mlflow.db instead of the team's shared tracking server, and how do you fix it?

level: middleimportance: should knowfreq 13%

basics

~20 s

The MLflow client uses mlflow.set_tracking_uri, else MLFLOW_TRACKING_URI, else a default — sqlite:///mlflow.db in the working directory since MLflow 3.7. A scheduler that doesn't pass the variable gets that default; set it in the job's environment or code.

open as a page

When you run mlflow server for a team, how do you separate the backend store from the artifact store, and when should clients write artifacts through the server rather than straight to the bucket?

level: seniorimportance: should knowfreq 8%

basics

~20 s

Point --backend-store-uri at PostgreSQL or MySQL for metadata and --artifacts-destination at a bucket for files. Artifact serving is on by default, so clients upload through the server with its credentials; --no-serve-artifacts with --default-artifact-root makes clients write the bucket directly.

open as a page

What breaks when an MLflow 2 training script that logs a model with artifact_path= and loads it from the run's artifacts runs on MLflow 3?

level: seniorimportance: nice to knowfreq 6%

basics

~20 s

In MLflow 3 a logged model is a LoggedModel with its own model_id, stored outside the run's artifacts. artifact_path= still works but warns to use name=, and loading via mlflow.get_artifact_uri("model") fails; load from ModelInfo.model_uri or models:/<model_id>.

open as a page