skip to content

In DVC, why does a 2 TB photo set fill about 4 TB on workstations that cannot reflink, and what does changing `cache.type` cost?

level: middleimportance: should knowfreq 11%

answer

  1. the default is not a link
  2. reflink first, then copy
  3. once in cache, once in workspace
  4. links trade away in-place edits
  5. relink after changing the setting

basics

~20 s

DVC's default cache.type is reflink,copy: without reflink support it copies every file from the cache into the workspace, so the data exists twice. Setting hardlink or symlink removes the duplicate but makes workspace files read-only, so edits need dvc unprotect first.

solid answer

~40 s

`cache.type` defaults to `reflink,copy`. A reflink is a copy-on-write clone that costs no extra space and is safe to edit, but only some file systems support it (Btrfs, XFS and OCFS2 on Linux, APFS on macOS); elsewhere DVC falls back to `copy`, so the 2 TB sits once in `.dvc/cache` and once in the workspace. `dvc config cache.type hardlink,symlink` removes the duplicate — hardlinks when cache and workspace share a file system, symlinks otherwise — but editing a linked file in place would rewrite the cached object, so DVC makes linked workspace files read-only and you run `dvc unprotect` (which turns the file into a copy) before changing one. Existing files switch to the new type only after `dvc checkout --relink`, and `dvc version` lists which link types the file system supports.

code

bash · 8 lines
bash
dvc version                      # lists the link types this file system supports
dvc config cache.type hardlink,symlink
dvc checkout --relink            # re-create existing workspace files as links

# Later, to correct one photo safely
dvc unprotect data/camera-traps/IMG_0412.jpg
# ...edit the file...
dvc add data/camera-traps

go deeper

for a junior

Recall that DVC keeps data in a cache and puts the workspace version there by reflink or copy unless configured otherwise.

for a middle

Explain the four link types, why the default falls back to copy on many file systems, and why linked files become read-only.

for a senior

Show the operational trade: halving disk use against read-only files, dvc unprotect for corrections, a relink after the change, and tools that break on read-only files.

for a principal

Frame the choice as storage budget against working style: provision reflink-capable file systems, accept duplication, or accept write-once data and train people on unprotect.

## Why the data exists twice DVC keeps every tracked file in its **cache** (`.dvc/cache` by default) and puts the version the current commit needs into the **workspace**. How the workspace copy is made is the `cache.type` setting, and its default is `reflink,copy`: an ordered list meaning "try a reflink, and if the file system cannot do that, copy". A **reflink** (copy-on-write link) makes a second file name share the same data blocks; the file system copies a block only when one side writes to it. Reflinks are supported on a limited set of file systems — Btrfs, XFS and OCFS2 on Linux, APFS on macOS. On a workstation whose file system has none of these, DVC silently falls back to `copy`, and a two-terabyte camera-trap folder costs two terabytes in the cache **plus** two terabytes in the workspace. Checkouts are slow too: every switch of version rewrites the bytes. DVC prints a reminder when linking one file takes longer than ten seconds and `cache.type` is unset; `cache.slow_link_warning false` silences it. ## The four link types | `cache.type` | Saves space | Fast checkout | Safe to edit in place | |---|---|---|---| | `reflink` | yes | yes | yes | | `hardlink` | yes | yes | no | | `symlink` | yes | yes | no | | `copy` | no | no | yes | - **`hardlink`** — the workspace name and the cache object are the same file (one inode). Works only when cache and workspace are on the same file system. File systems cap links per file (NTFS 1,024, ext4 65,000); DVC falls back to the next type in the list when a dataset has that many identical files. - **`symlink`** — the workspace entry is a symbolic link pointing into the cache. Works across file systems, for example a repository on an SSD with the cache on a larger disk. - **`copy`** — supported everywhere, safe, and the most expensive in time and space. Run `dvc version` inside the project to see which link types the current file system supports. ## Seeing which type is in use - `ls -l` on a protected workspace file shows read-only permissions (`-r--r--r--`), the sign that it is a hardlink or symlink into the cache. - `ls -i` prints inode numbers: a hardlinked workspace file and its cache object share one. - A symlink shows its target path inside `.dvc/cache`. - A writable, independent file means a reflink or a copy; on a file system without reflink support it is a copy, and the disk usage confirms it. ## What switching costs Choosing `dvc config cache.type hardlink,symlink` removes the duplicate, but changes how people must work: 1. **Workspace files become read-only.** With a hardlink or symlink, writing to the workspace file writes to the cached object — the bytes filed under the old hash would no longer match it, corrupting that version. DVC protects against this by making linked files read-only. 2. **Edits need an extra step.** `dvc unprotect <path>` replaces the link with a real copy that can be edited safely; afterwards `dvc add` records the new version. Unprotecting copies data, so it is expensive on large files. Adding files to a tracked directory or deleting files from it does not need it. 3. **Existing files keep their old link type** until you run `dvc checkout --relink`, which re-creates every tracked file in the workspace with the configured type. 4. **Tools that rewrite files in place** (an image editor saving over a photo, a script that patches a CSV) fail on read-only files — usually the first symptom a team notices. ## Choosing for the lab - Moving the repository to a reflink-capable file system keeps both properties: no duplicate and safe edits. - Where that is not possible and the photos are effectively write-once, `hardlink` (same disk) or `symlink` (cache on another disk) halves disk use; the rare correction goes through `dvc unprotect`. - Where people routinely edit tracked files in place, the default copy is the safe choice and the disk cost is the price. ## Mistakes interviewers listen for - Saying DVC symlinks by default. The default is `reflink,copy`, and on many Linux file systems that means copy. - Believing hardlinked or symlinked files are safe to edit in place. - Changing `cache.type` and expecting existing workspace files to change without `--relink`. - Calling `copy` unsafe; it is the one non-reflink type where an in-place edit cannot reach the cache.

  • When is `hardlink` the better choice than `symlink`, and when is it impossible?
    DVC's docs call hardlinks the most efficient type when the repository and the cache share a partition, and symlinks the most efficient when they sit on different file systems — where hardlinks cannot be created at all. Both avoid duplicate data and both make workspace files read-only. A hardlinked file also looks like an ordinary file to tools that handle symlinks badly.
  • Does `dvc unprotect` make every later edit safe?
    It makes the targeted files safe: they become independent copies, so editing them cannot touch the cache. The cost is copying the data. After editing, `dvc add` records the new version, and the file is linked again under the new hash according to `cache.type`.

  • DVC symlinks workspace files to the cache by default.
  • The default reflink setting saves space on every Linux file system.
  • Hardlinked workspace files are safe to edit in place.
  • Changing cache.type relinks existing workspace files immediately.
  • Copy mode is risky because edits leak into the cache.