skip to content

Pipelines & Experiments

dvc.yaml declares stages with deps, outs and params so dvc repro reruns only what changed, and dvc exp run layers parameter sweeps on that graph. Interviewers check runs are reproducible, not logged.

on this pageshow

explore

questions

6

In a DVC pipeline, what do `dvc.yaml` and `dvc.lock` each record, and why do both get committed to Git?

level: juniorimportance: must knowfreq 24%

answer

  1. declared intent against recorded state
  2. cmd, deps, outs, params per stage
  3. hashes and param values per stage
  4. written by dvc repro, not by hand
  5. commit, then dvc checkout restores results

basics

~20 s

dvc.yaml declares each stage's command, dependencies, outputs and tracked parameters; dvc.lock records what the last run actually used, with content hashes of dependencies and outputs and the parameter values. Committing both ties a commit to exact pipeline results.

solid answer

~50 s

`dvc.yaml` is the pipeline you write: a `stages` map where each stage has a required `cmd` and optional `deps`, `outs`, `params`, `metrics` and `plots`. Stages connect when one stage's output is another's dependency, and that graph, not the order in the file, sets execution order. `dvc.lock` is what `dvc repro` or `dvc exp run` writes after a stage runs: the command, every dependency and output with its `md5` hash and size, and the value of each tracked param key. On the next `dvc repro`, DVC compares the workspace against the lock and runs only the stages whose command, dependencies, params or outputs no longer match. Cached outputs live in the DVC cache, not in Git; committing both files with the code means a teammate can check out that commit and run `dvc checkout` to get the matching results instead of retraining.

code

yaml · 19 lines
yaml
stages:
  train:
    cmd: python src/train.py data/features models/langid.pkl
    deps:
      - src/train.py
      - data/features
    params:
      - train.lr
    outs:
      - models/langid.pkl
  evaluate:
    cmd: python src/evaluate.py models/langid.pkl data/features
    deps:
      - src/evaluate.py
      - models/langid.pkl
      - data/features
    metrics:
      - eval/metrics.json:
          cache: false

go deeper

for a junior

Know the split: dvc.yaml is what you declare, dvc.lock is what DVC recorded after running. Name the stage fields and say that only cmd is required.

for a middle

Explain how dvc repro compares commands, dependency hashes and param values against dvc.lock, and why that comparison needs the lock committed together with the code.

for a senior

Show how a committed lock makes a pull request reviewable and lets a teammate restore results with dvc checkout instead of retraining, and how metrics with cache: false change what Git holds.

for a principal

Treat dvc.yaml and dvc.lock as the team's contract for what a commit means: decide what must be declared, what stays in Git, and what a review should check in the lock.

## Two files, two jobs A **DVC pipeline** is a set of **stages**: shell commands that read some files and write others. DVC keeps two YAML files side by side for it, and an interviewer asking about them wants to hear that one is *intent* and the other is *record*. | | `dvc.yaml` | `dvc.lock` | |---|---|---| | Who writes it | You, by hand or with `dvc stage add` | `dvc repro` and `dvc exp run` | | What it holds | Stage definitions: `cmd`, `deps`, `outs`, `params`, `metrics`, `plots` | The last run of each stage: command, hashes and sizes of dependencies and outputs, param values | | Edit by hand? | Yes, it is the source | No; DVC's docs say to avoid editing it | | In Git? | Yes | Yes | ## What a stage declares in dvc.yaml Only `cmd` is required. The rest tells DVC what to watch and what to keep: - **`cmd`** is one shell command or a list run in order; the first failure halts the stage. - **`deps`** lists the files or directories the command reads, usually including the script itself. - **`params`** lists keys in a **parameters file** (`params.yaml` by default; YAML, JSON, TOML and Python files are supported). Dot-separated names such as `featurize.ngram_max` are paths into that file. - **`outs`** lists what the command writes. DVC caches these and deletes them before the command runs, unless an output sets `persist: true`. - **`metrics`** and **`plots`** are outputs that are also metrics or plot sources; small ones are often marked `cache: false` so they stay in Git. - Optional fields include `wdir`, `frozen`, `always_changed` and `desc`. ```yaml stages: featurize: cmd: python src/featurize.py data/prepared data/features deps: - src/featurize.py - data/prepared params: - featurize.ngram_min - featurize.ngram_max outs: - data/features train: cmd: python src/train.py data/features models/langid.pkl deps: - src/train.py - data/features params: - train.lr outs: - models/langid.pkl ``` Because `data/features` is an output of `featurize` and a dependency of `train`, `featurize` always runs first. `dvc dag` draws this graph. ## What dvc.lock records After a stage runs, DVC writes its entry in `dvc.lock` (the file starts with `schema: '2.0'`). Each entry repeats the `cmd`, lists every dependency and output by `path` with a content hash (`md5` for local files) and `size`, and lists every tracked parameter with its **value** under the params file's name. Templated values are written resolved, and `foreach` or `matrix` stages are expanded into individual stages. The lock is how DVC knows what the last run consumed and produced. ## How dvc repro uses the pair 1. DVC reads the pipeline's `dvc.yaml` files and builds the dependency graph. 2. For each stage, in graph order, it compares the current command, the hashes of the dependencies and the values of the listed params with the lock, and checks that the recorded outputs exist and are in the cache. 3. If nothing differs, it prints `Stage 'featurize' didn't change, skipping`. 4. If something differs, it removes the old outputs, runs the command, caches the new outputs and rewrites that stage's entry (`Updating lock file 'dvc.lock'`). DVC also keeps a **run cache**: if a stage was run before with exactly the same command, dependencies and params, it can restore those outputs from the cache instead of running the command again. ## Why both belong in Git - `dvc.yaml` alone says how to build the results, not which results were built, so everyone would have to run the whole pipeline again. - `dvc.lock` connects a Git commit to specific cached bytes. Commit it with the code and `params.yaml`, and `git checkout <rev>` followed by `dvc checkout` puts that revision's features and model back into the workspace from the cache. Getting the bytes into the cache from shared storage is the remotes material's subject. - Reviewers see real changes. A pull request that changes `train.lr` shows the new value in `params.yaml` and the new hashes in `dvc.lock`, so it is clear the model was rebuilt with that value. ## Mistakes interviewers listen for - Putting `dvc.lock` in `.gitignore`. The pipeline still runs, but the commit no longer records what was produced. - Thinking DVC runs stages in the order they appear in the file. - Thinking `dvc.yaml` stores hashes, or that committing the lock puts the model into Git. Git holds the small text files; the model's bytes stay in the DVC cache.

  • Can you add a stage without writing the YAML by hand?
    Yes. `dvc stage add -n featurize -d src/featurize.py -d data/prepared -p featurize.ngram_min,featurize.ngram_max -o data/features python src/featurize.py data/prepared data/features` writes the entry into `dvc.yaml` and checks the arguments and graph first. It does not run the stage unless you pass `--run`, and templating is not available through it, so many teams edit the file directly.
  • What does `dvc dag` show, and which flags matter?
    It prints the stage graph built from every `dvc.yaml` in the project. Give it a stage name to see that stage's ancestors, or add `--full` for the whole pipeline it belongs to. `--outs` draws the graph of files instead of stages, and `--mermaid`, `--md` or `--dot` export it for a README or a pull-request comment.
  • Why mark a metrics file `cache: false`, and what does that cost?
    A small JSON metrics file kept in Git can be read straight from any commit, so `git diff` and `dvc metrics diff` work without the DVC cache. It is still listed with its hash in `dvc.lock`. The cost: if any output of a stage has `cache: false`, the run cache is turned off for that stage.

  • dvc.lock is optional and can safely go in .gitignore
  • dvc repro runs stages in the order they appear in dvc.yaml
  • dvc.yaml stores the hashes of the files each stage produced
  • Committing dvc.lock puts the trained model itself into Git
  • dvc repro reruns every stage each time it is called
open as a page

Where does DVC keep an experiment created by `dvc exp run`, and how do you keep the winning one and share it with a teammate?

level: middleimportance: should knowfreq 10%

basics

~20 s

Each DVC experiment is a hidden commit under a custom Git ref in .git/refs/exps, based on the HEAD it ran from. Keep the winner with dvc exp apply or dvc exp branch; share it with dvc exp push.

open as a page

In DVC, after `featurize.ngram_max` changes in `params.yaml`, why does `dvc repro` re-run featurize, train and evaluate but skip prepare?

level: middleimportance: should knowfreq 13%

basics

~20 s

DVC tracks params per key, so only featurize, which lists featurize.ngram_max, sees a value that differs from dvc.lock. Its new features change train's dependency hash, the new model changes evaluate's, and prepare's recorded hashes still match.

open as a page

In a DVC pipeline, after fixing a bug in `src/textnorm.py`, a helper the featurize script imports, `dvc repro` says featurize didn't change. Why, and what do you fix?

level: seniorimportance: should knowfreq 8%

basics

~10 s

DVC hashes only the paths listed in a stage's deps and never follows the script's imports, so with textnorm.py undeclared nothing featurize depends on changed. Add it, or the src directory, to deps.

open as a page

In a DVC project, how do `dvc params diff`, `dvc metrics diff` and `dvc plots diff` compare a pull request's branch with `main`?

level: middleimportance: nice to knowfreq 6%

basics

~20 s

All three take Git revisions: none means workspace against HEAD, one means that revision against the workspace, two compare each other. params diff lists old and new values, metrics diff adds numeric changes, plots diff overlays the versions.

open as a page

In DVC 3.x, how do you run an overnight grid over learning rate and n-gram range with `dvc exp run --queue` and `dvc queue`, and what can break it?

level: seniorimportance: nice to knowfreq 4%

basics

~20 s

Queue the grid with dvc exp run --queue and sweep overrides in -S, then run it with dvc queue start. Each run executes in a copy under .dvc/tmp/exps, so untracked inputs and parallel jobs are the usual breakers.

open as a page