skip to content

DVC

DVC versions data and models beside Git with small .dvc pointers, pushes the bytes to remote storage and reruns only stale pipeline stages. Interviewers ask which data built a given model.

on this pageshow

explore

questions

17

In DVC, after `dvc add data/camera-traps`, what do you commit to Git, and where do the 1.4 million photos actually live?

level: juniorimportance: must knowfreq 30%

answer

  1. small YAML placeholder
  2. Git sees the pointer, not bytes
  3. an entry appended to .gitignore
  4. cache addressed by MD5 of contents
  5. directory hash ends in .dir

basics

~20 s

You commit data/camera-traps.dvc and data/.gitignore. The pointer records the directory's MD5 hash, size, file count and path; the photos move into DVC's content-addressed cache under .dvc/cache and are linked or copied back into the workspace.

solid answer

~40 s

`dvc add data/camera-traps` hashes every file, moves the contents into the project cache (in DVC 3, `.dvc/cache/files/md5/<first two hex characters>/<remaining 30>`) and links or copies them back into the workspace. It writes one pointer, `data/camera-traps.dvc`, whose `outs` entry holds `md5` (the directory's hash, ending in `.dir`), `size`, `nfiles`, `hash: md5` and `path`, and it appends `/camera-traps` to `data/.gitignore` so Git never stages the photos. You then `git add data/camera-traps.dvc data/.gitignore` and commit. The `.dir` object in the cache is a JSON list mapping each relative path to its file's hash, so one small pointer stands for all 1.4 million files. A plain `dvc add` uploads nothing: getting the bytes to another machine is a separate step.

code

bash · 4 lines
bash
dvc add data/camera-traps data/labels.csv
git add data/camera-traps.dvc data/labels.csv.dvc data/.gitignore
git commit -m "Track camera-trap photos and labels with DVC"
cat data/camera-traps.dvc

go deeper

for a junior

Recall the two files you commit after dvc add, the pointer and the ignore entry, and say plainly that the photos never enter Git.

for a middle

Explain the pointer's fields, how the cache path is built from the hash, and how a .dir listing lets one pointer stand for a whole directory.

for a senior

Show you know where the bytes really are at each moment: only in the local cache until pushed, which is why a committed pointer alone can leave a clone with nothing to check out.

for a principal

Weigh one pointer per large directory against finer-grained pointers: fewer Git changes and simpler history versus coarser change tracking and larger listings to rewrite on every edit.

## What `dvc add` does DVC (Data Version Control) is a command-line tool that versions large files beside a Git repository without putting their bytes into Git. Its everyday entry point is `dvc add`. Running `dvc add data/camera-traps` on a lab's two-terabyte folder of camera-trap JPEGs does six things: 1. **Hashes the contents.** Every file gets an MD5 hash of its bytes; for a directory, DVC also builds a listing of every file inside. 2. **Moves the contents into the cache.** The **cache** is DVC's local object store, `.dvc/cache` by default. In DVC 3 the objects sit under `.dvc/cache/files/md5/`. 3. **Puts the files back in the workspace** as links or copies, according to the `cache.type` setting (default `reflink,copy`). 4. **Writes a pointer file**, `data/camera-traps.dvc`, next to the target. 5. **Appends the target to `.gitignore`** in the same directory (`/camera-traps` in `data/.gitignore`), unless the project was initialised with `dvc init --no-scm`. 6. **Prints the `git add` command** you should run next. With `core.autostage` set to `true`, DVC stages the files itself; it still never commits. ## The pointer file A `.dvc` file is a few lines of YAML. For the photo directory it looks like this: ```yaml outs: - md5: 8d2b6f0c1a9e47d3b5c2e7f40a6d91c3.dir size: 2000000000000 nfiles: 1400000 hash: md5 path: camera-traps ``` | Field | What it records | |---|---| | `md5` | The content hash; for a directory, the hash of its `.dir` listing plus the `.dir` suffix | | `size` | Total bytes (the sum of all files for a directory) | | `nfiles` | Number of files inside, for a directory | | `hash` | The hash algorithm; DVC 3 writes `md5`, the only one supported | | `path` | The tracked path, relative to the `.dvc` file | Optional fields such as `desc`, `remote`, `push` and `cache` can be added, but the fields above are what `dvc add` writes for a local directory; a single file gets no `nfiles`, and an executable one also gets `isexec`. The pointer is the only thing Git needs: checking out an old commit brings back an old pointer, and the pointer names exactly which bytes belong in the workspace. ## How the cache is laid out The cache is **content-addressed**: an object's location is derived from its hash, never from its name. - A file whose hash is `ec1d2935f811b77cc49b031b999cbf17` is stored at `.dvc/cache/files/md5/ec/1d2935f811b77cc49b031b999cbf17` — the first two hex characters name a subdirectory and the remaining thirty name the file. - File names and directory trees are flattened away. Two photos with different names but identical bytes are stored **once**. - A tracked directory gets one extra object with a `.dir` suffix: a JSON array of `{"md5": ..., "relpath": ...}` entries, one per file. That is how DVC knows which cached objects make up `camera-traps/`. - `dvc init` lists the cache directory in `.gitignore`, so the cache itself is never committed. Because a directory is added as one target, the lab gets one pointer for 1.4 million files, while each photo is still cached individually. Commands that use tracked data can still target a single file inside the directory. ## What Git sees, and what it does not - Git versions `data/camera-traps.dvc` and `data/.gitignore`. The history of the pointer *is* the history of the dataset. - Git never sees a photo. A fresh clone has the pointer but an empty cache, so it has no photos until they are brought in from remote storage — a separate step with its own commands. - `dvc add` with no extra flags transfers nothing off the machine. Committing the pointer to a Git server backs up the pointer, not the data. - `dvc add --no-commit` writes the pointer but skips caching; `dvc commit` finishes the job later. ## Checking the result A quick way to confirm what happened, and a habit worth showing in an interview: - `cat data/camera-traps.dvc` shows the five fields above — a few hundred bytes for two terabytes. - `git status` lists only the new pointer and the changed `data/.gitignore`; the photo directory does not appear because Git now ignores it. - `dvc status` prints `Data and pipelines are up to date.`, meaning the workspace matches the pointer. - Listing `.dvc/cache/files/md5/` shows two-character directories, not photo names: the cache keeps no trace of the original file names outside the `.dir` listing. ## Mistakes interviewers listen for - Treating the `.dvc` file as the data, or as a compressed copy of it. It is a few hundred bytes of metadata. - Committing the photo directory to Git as well, which defeats the point and is what the `.gitignore` entry prevents. - Expecting one `.dvc` file per photo. A directory target produces a single pointer backed by a `.dir` listing. - Assuming `dvc add` makes the data available to colleagues. Until the bytes are pushed somewhere, they exist only in this machine's cache.

  • Why does DVC write one `.dvc` file for the directory instead of one per photo?
    Adding a directory creates a single pointer whose `md5` names a `.dir` object in the cache: a JSON list of every file's relative path and hash. Git's view stays one small file however many photos there are, while each photo is still cached individually, so commands can target one file inside the tracked directory. The DVC docs recommend directory targets for datasets with large file counts.
  • Two camera-trap JPEGs have different names but identical bytes. How many copies does the cache hold?
    One. The cache path comes from the hash of the contents only, so both entries in the `.dir` listing point at the same object. The workspace still shows two files.
  • What changes with `dvc add --no-commit`?
    The `.dvc` file is still written, but the contents are not stored in the cache. It suits data you are still exploring and do not want cached yet; a later `dvc commit` stores the contents and completes the operation.

A .dvc file is a cloakroom claim ticket: Git keeps the ticket, the cache keeps the coat. Handing someone the ticket does not hand them the coat, and a ticket whose coat was never checked in redeems nothing.

  • The .dvc file contains a compressed copy of the data.
  • dvc add commits the photos to Git automatically.
  • dvc add uploads the data to remote storage by itself.
  • Every photo in the directory gets its own .dvc file.
  • Pushing the .dvc file to a Git server backs up the photos.

Assumes DVC 3.x, checked against 3.67; since 3.0 cached objects live under .dvc/cache/files/md5/ and each .dvc output carries hash: md5.

open as a page

In a DVC pipeline, what do `dvc.yaml` and `dvc.lock` each record, and why do both get committed to Git?

level: juniorimportance: must knowfreq 24%

basics

~20 s

dvc.yaml declares each stage's command, dependencies, outputs and tracked parameters; dvc.lock records what the last run actually used, with content hashes of dependencies and outputs and the parameter values. Committing both ties a commit to exact pipeline results.

open as a page

In DVC, why does a fresh `git clone` of a project lack its tracked data, and how do `dvc push` and `dvc pull` move it?

level: juniorimportance: must knowfreq 30%

basics

~20 s

Git carries only the small .dvc and dvc.lock pointer files; the bytes stay in each machine's DVC cache. dvc push uploads the cached objects a remote lacks; dvc pull downloads them and places them in the workspace.

open as a page

In DVC, after `git checkout` of last spring's tag, why are the workspace photos still this month's, and what restores the spring version?

level: middleimportance: must knowfreq 18%

basics

~20 s

git checkout only swaps the small .dvc pointer files; Git never tracked the photos. Running dvc checkout reads the restored pointers and rebuilds the workspace from the local cache, and dvc install adds a Git hook that runs it after every git checkout.

open as a page

In DVC, an S3 secret key set with `dvc remote modify` ended up in a Git commit — why, and where should remote credentials live?

level: middleimportance: must knowfreq 16%

basics

~20 s

Without a level flag, dvc remote modify writes .dvc/config, the project file that Git tracks. Secrets belong in the Git-ignored .dvc/config.local (written with --local) or outside DVC config entirely: environment variables, a credentials file or profile, SSH config.

open as a page

In DVC, a student edited one tracked camera-trap photo in place — what does `dvc status` report, and how do you keep or discard it?

level: middleimportance: should knowfreq 9%

basics

~20 s

dvc status lists data/camera-traps.dvc under changed outs as modified. To keep the edit, run dvc commit or dvc add and commit the updated pointer to Git; to discard it, run dvc checkout --force on the photo to restore the cached version.

open as a page

Where does DVC keep an experiment created by `dvc exp run`, and how do you keep the winning one and share it with a teammate?

level: middleimportance: should knowfreq 10%

basics

~20 s

Each DVC experiment is a hidden commit under a custom Git ref in .git/refs/exps, based on the HEAD it ran from. Keep the winner with dvc exp apply or dvc exp branch; share it with dvc exp push.

open as a page

In DVC, after `featurize.ngram_max` changes in `params.yaml`, why does `dvc repro` re-run featurize, train and evaluate but skip prepare?

level: middleimportance: should knowfreq 13%

basics

~20 s

DVC tracks params per key, so only featurize, which lists featurize.ngram_max, sees a value that differs from dvc.lock. Its new features change train's dependency hash, the new model changes evaluate's, and prepare's recorded hashes still match.

open as a page

In DVC, when do you use `dvc get`, `dvc import` or `dvc import-url` to bring data from another repository or a bucket into a project?

level: middleimportance: should knowfreq 12%

basics

~20 s

dvc get copies data from a DVC or Git repository once and tracks nothing. dvc import also writes an import .dvc file pinning the source repo and commit, refreshed by dvc update. dvc import-url does that for a storage URL.

open as a page

In DVC, the lab ran `dvc gc --workspace` to free disk space — why can last spring's tagged photo set no longer be checked out, and which flags would have kept it?

level: seniorimportance: should knowfreq 8%

basics

~20 s

dvc gc deletes every cached object its scope does not reference; --workspace keeps only what the current pointers name, so older tagged versions vanish unless another copy exists. Scope it with --all-tags, --all-branches or --all-commits, and preview with --dry.

open as a page

In a DVC pipeline, after fixing a bug in `src/textnorm.py`, a helper the featurize script imports, `dvc repro` says featurize didn't change. Why, and what do you fix?

level: seniorimportance: should knowfreq 8%

basics

~10 s

DVC hashes only the paths listed in a stage's deps and never follows the script's imports, so with textnorm.py undeclared nothing featurize depends on changed. Add it, or the src directory, to deps.

open as a page

In DVC, when several workstations and a GPU server mount the same NAS, when should they share one cache directory instead of each running `dvc pull`, and what must be configured?

level: seniorimportance: should knowfreq 8%

basics

~20 s

Share one cache when all machines mount the same storage: point cache.dir at it, set cache.shared group and a symlink cache.type, so data is linked, not transferred. Keep a remote for other machines and as a second copy.

open as a page

In a DVC project, how do `dvc params diff`, `dvc metrics diff` and `dvc plots diff` compare a pull request's branch with `main`?

level: middleimportance: nice to knowfreq 6%

basics

~20 s

All three take Git revisions: none means workspace against HEAD, one means that revision against the workspace, two compare each other. params diff lists old and new values, metrics diff adds numeric changes, plots diff overlays the versions.

open as a page

After a DVC 2.x project upgrades to DVC 3, why can adding a few photos to the tracked camera-trap folder double its cache, and what does `dvc cache migrate` change?

level: seniorimportance: nice to knowfreq 4%

basics

~20 s

DVC 3.0 caches under .dvc/cache/files/md5, hashes raw bytes, and by default does not de-duplicate against 2.x objects, so a changed 2.x directory is stored again in full. dvc cache migrate re-hashes old objects into the 3.0 location and links the old paths.

open as a page

In DVC 3.x, how do you run an overnight grid over learning rate and n-gram range with `dvc exp run --queue` and `dvc queue`, and what can break it?

level: seniorimportance: nice to knowfreq 4%

basics

~20 s

Queue the grid with dvc exp run --queue and sweep overrides in -S, then run it with dvc queue start. Each run executes in a copy under .dvc/tmp/exps, so untracked inputs and parallel jobs are the usual breakers.

open as a page

In DVC, CI's `dvc pull` succeeds for the project's own S3 remote but fails on a directory added with `dvc import` — why, and how do you fix it without committing secrets?

level: seniorimportance: nice to knowfreq 5%

basics

~20 s

dvc push skips data from dvc import, so dvc pull fetches it from the source repository's default remote, which CI may not reach. Select a reachable source remote with --remote and pass secrets via --config with a Git-ignored file.

open as a page