skip to content

In DVC, after `dvc add data/camera-traps`, what do you commit to Git, and where do the 1.4 million photos actually live?

level: juniorimportance: must knowfreq 30%

answer

  1. small YAML placeholder
  2. Git sees the pointer, not bytes
  3. an entry appended to .gitignore
  4. cache addressed by MD5 of contents
  5. directory hash ends in .dir

basics

~20 s

You commit data/camera-traps.dvc and data/.gitignore. The pointer records the directory's MD5 hash, size, file count and path; the photos move into DVC's content-addressed cache under .dvc/cache and are linked or copied back into the workspace.

solid answer

~40 s

`dvc add data/camera-traps` hashes every file, moves the contents into the project cache (in DVC 3, `.dvc/cache/files/md5/<first two hex characters>/<remaining 30>`) and links or copies them back into the workspace. It writes one pointer, `data/camera-traps.dvc`, whose `outs` entry holds `md5` (the directory's hash, ending in `.dir`), `size`, `nfiles`, `hash: md5` and `path`, and it appends `/camera-traps` to `data/.gitignore` so Git never stages the photos. You then `git add data/camera-traps.dvc data/.gitignore` and commit. The `.dir` object in the cache is a JSON list mapping each relative path to its file's hash, so one small pointer stands for all 1.4 million files. A plain `dvc add` uploads nothing: getting the bytes to another machine is a separate step.

code

bash · 4 lines
bash
dvc add data/camera-traps data/labels.csv
git add data/camera-traps.dvc data/labels.csv.dvc data/.gitignore
git commit -m "Track camera-trap photos and labels with DVC"
cat data/camera-traps.dvc

go deeper

for a junior

Recall the two files you commit after dvc add, the pointer and the ignore entry, and say plainly that the photos never enter Git.

for a middle

Explain the pointer's fields, how the cache path is built from the hash, and how a .dir listing lets one pointer stand for a whole directory.

for a senior

Show you know where the bytes really are at each moment: only in the local cache until pushed, which is why a committed pointer alone can leave a clone with nothing to check out.

for a principal

Weigh one pointer per large directory against finer-grained pointers: fewer Git changes and simpler history versus coarser change tracking and larger listings to rewrite on every edit.

## What `dvc add` does DVC (Data Version Control) is a command-line tool that versions large files beside a Git repository without putting their bytes into Git. Its everyday entry point is `dvc add`. Running `dvc add data/camera-traps` on a lab's two-terabyte folder of camera-trap JPEGs does six things: 1. **Hashes the contents.** Every file gets an MD5 hash of its bytes; for a directory, DVC also builds a listing of every file inside. 2. **Moves the contents into the cache.** The **cache** is DVC's local object store, `.dvc/cache` by default. In DVC 3 the objects sit under `.dvc/cache/files/md5/`. 3. **Puts the files back in the workspace** as links or copies, according to the `cache.type` setting (default `reflink,copy`). 4. **Writes a pointer file**, `data/camera-traps.dvc`, next to the target. 5. **Appends the target to `.gitignore`** in the same directory (`/camera-traps` in `data/.gitignore`), unless the project was initialised with `dvc init --no-scm`. 6. **Prints the `git add` command** you should run next. With `core.autostage` set to `true`, DVC stages the files itself; it still never commits. ## The pointer file A `.dvc` file is a few lines of YAML. For the photo directory it looks like this: ```yaml outs: - md5: 8d2b6f0c1a9e47d3b5c2e7f40a6d91c3.dir size: 2000000000000 nfiles: 1400000 hash: md5 path: camera-traps ``` | Field | What it records | |---|---| | `md5` | The content hash; for a directory, the hash of its `.dir` listing plus the `.dir` suffix | | `size` | Total bytes (the sum of all files for a directory) | | `nfiles` | Number of files inside, for a directory | | `hash` | The hash algorithm; DVC 3 writes `md5`, the only one supported | | `path` | The tracked path, relative to the `.dvc` file | Optional fields such as `desc`, `remote`, `push` and `cache` can be added, but the fields above are what `dvc add` writes for a local directory; a single file gets no `nfiles`, and an executable one also gets `isexec`. The pointer is the only thing Git needs: checking out an old commit brings back an old pointer, and the pointer names exactly which bytes belong in the workspace. ## How the cache is laid out The cache is **content-addressed**: an object's location is derived from its hash, never from its name. - A file whose hash is `ec1d2935f811b77cc49b031b999cbf17` is stored at `.dvc/cache/files/md5/ec/1d2935f811b77cc49b031b999cbf17` — the first two hex characters name a subdirectory and the remaining thirty name the file. - File names and directory trees are flattened away. Two photos with different names but identical bytes are stored **once**. - A tracked directory gets one extra object with a `.dir` suffix: a JSON array of `{"md5": ..., "relpath": ...}` entries, one per file. That is how DVC knows which cached objects make up `camera-traps/`. - `dvc init` lists the cache directory in `.gitignore`, so the cache itself is never committed. Because a directory is added as one target, the lab gets one pointer for 1.4 million files, while each photo is still cached individually. Commands that use tracked data can still target a single file inside the directory. ## What Git sees, and what it does not - Git versions `data/camera-traps.dvc` and `data/.gitignore`. The history of the pointer *is* the history of the dataset. - Git never sees a photo. A fresh clone has the pointer but an empty cache, so it has no photos until they are brought in from remote storage — a separate step with its own commands. - `dvc add` with no extra flags transfers nothing off the machine. Committing the pointer to a Git server backs up the pointer, not the data. - `dvc add --no-commit` writes the pointer but skips caching; `dvc commit` finishes the job later. ## Checking the result A quick way to confirm what happened, and a habit worth showing in an interview: - `cat data/camera-traps.dvc` shows the five fields above — a few hundred bytes for two terabytes. - `git status` lists only the new pointer and the changed `data/.gitignore`; the photo directory does not appear because Git now ignores it. - `dvc status` prints `Data and pipelines are up to date.`, meaning the workspace matches the pointer. - Listing `.dvc/cache/files/md5/` shows two-character directories, not photo names: the cache keeps no trace of the original file names outside the `.dir` listing. ## Mistakes interviewers listen for - Treating the `.dvc` file as the data, or as a compressed copy of it. It is a few hundred bytes of metadata. - Committing the photo directory to Git as well, which defeats the point and is what the `.gitignore` entry prevents. - Expecting one `.dvc` file per photo. A directory target produces a single pointer backed by a `.dir` listing. - Assuming `dvc add` makes the data available to colleagues. Until the bytes are pushed somewhere, they exist only in this machine's cache.

  • Why does DVC write one `.dvc` file for the directory instead of one per photo?
    Adding a directory creates a single pointer whose `md5` names a `.dir` object in the cache: a JSON list of every file's relative path and hash. Git's view stays one small file however many photos there are, while each photo is still cached individually, so commands can target one file inside the tracked directory. The DVC docs recommend directory targets for datasets with large file counts.
  • Two camera-trap JPEGs have different names but identical bytes. How many copies does the cache hold?
    One. The cache path comes from the hash of the contents only, so both entries in the `.dir` listing point at the same object. The workspace still shows two files.
  • What changes with `dvc add --no-commit`?
    The `.dvc` file is still written, but the contents are not stored in the cache. It suits data you are still exploring and do not want cached yet; a later `dvc commit` stores the contents and completes the operation.

A .dvc file is a cloakroom claim ticket: Git keeps the ticket, the cache keeps the coat. Handing someone the ticket does not hand them the coat, and a ticket whose coat was never checked in redeems nothing.

  • The .dvc file contains a compressed copy of the data.
  • dvc add commits the photos to Git automatically.
  • dvc add uploads the data to remote storage by itself.
  • Every photo in the directory gets its own .dvc file.
  • Pushing the .dvc file to a Git server backs up the photos.

Assumes DVC 3.x, checked against 3.67; since 3.0 cached objects live under .dvc/cache/files/md5/ and each .dvc output carries hash: md5.