After a DVC 2.x project upgrades to DVC 3, why can adding a few photos to the tracked camera-trap folder double its cache, and what does `dvc cache migrate` change?
answer
- 3.0 changed hashing and location
- old objects outside files/md5
- pointers lacking a hash field
- no automatic de-duplication
- re-hash, move, link back
basics
~20 sDVC 3.0 caches under .dvc/cache/files/md5, hashes raw bytes, and by default does not de-duplicate against 2.x objects, so a changed 2.x directory is stored again in full. dvc cache migrate re-hashes old objects into the 3.0 location and links the old paths.
solid answer
~50 sDVC 3.0 changed how files are hashed — 2.x converted CRLF to LF in files it judged to be text before hashing, 3.0 hashes raw bytes — and where objects live: `.dvc/cache/files/md5/` instead of directly under `.dvc/cache/`. A 3.x release still reads 2.x pointers (outputs without `hash: md5`) from the old location, but keeps the two layouts apart to avoid hash collisions and by default does not de-duplicate between them. Once a 2.x-tracked directory changes, the next `dvc add` re-hashes it the 3.0 way and stores all its files in the new location, unchanged photos included. `dvc cache migrate` re-hashes the old objects, moves them atomically into the 3.0 location and links each old path to its new object, so the data exists once wherever a link type works (copied where none does). `--dvc-files` also rewrites the workspace's DVC files to the 3.0 format; Git history is not rewritten.
code
bash · 4 linesdvc cache migrate --dry # how many 2.x objects would be re-hashed and moved
dvc cache migrate --dvc-files # migrate objects and rewrite workspace DVC files
git add data/camera-traps.dvc data/labels.csv.dvc
git commit -m "Move DVC files to the 3.0 format"go deeper
Recall that DVC 3 keeps its cache under .dvc/cache/files/md5 and that projects started on 2.x have data in an older location.
Explain the two 3.0 changes, raw-byte hashing and the new location, and how a pointer without a hash field marks a 2.x output.
Show why a changed 2.x directory is stored again in full, when to run dvc cache migrate with --dry and --dvc-files, and what it leaves untouched.
Plan an upgrade for a large shared dataset: when to migrate, the disk headroom it needs, re-uploading to remote storage, and holding every machine on one DVC major version.
## What DVC 3.0 changed in the cache DVC keeps tracked data in a content-addressed **cache**, where an object's path is derived from its hash. Release 3.0 changed two things about that cache at once, and both still matter for any project whose history began on 2.x. 1. **Hashing.** DVC 2.x tried to detect text files and converted Windows CRLF line endings to LF *before* hashing, so a CSV hashed the same on Windows and Linux even though the stored bytes differed. That misfired on binary files mistaken for text. DVC 3.0 treats every file as binary: a CRLF `labels.csv` and an LF one are now different data. 2. **Location.** DVC 3 stores objects under `.dvc/cache/files/md5/<2 hex>/<30 hex>`. Objects cached by 2.x stay directly under `.dvc/cache/<2 hex>/<30 hex>`. Pointer files show which world an output belongs to. DVC 3 writes `hash: md5` into each output it records; an output **without** a `hash` field is read as a 2.x one, hashed the old way and looked up in the old location. | | Recorded by DVC 2.x | Recorded by DVC 3.x | |---|---|---| | Cache path | `.dvc/cache/ab/cdef…` | `.dvc/cache/files/md5/ab/cdef…` | | Pointer output | `md5` (with `size` and `nfiles` where recorded) and `path`, no `hash` field | `md5`, `size`, `nfiles` (directories), `hash: md5`, `path` | | Line endings | CRLF normalised before hashing text | Raw bytes hashed | ## Why the cache doubled To avoid hash collisions between the two schemes, DVC 3 keeps them separate and does not de-duplicate between them by default. It reads 2.x data where it lies and duplicates only new or modified data — but for a directory output, "modified" means the whole directory: - The lab's photo folder was added under 2.x, so its pointer has no `hash` field and its 1.4 million objects sit in the old location. - A student adds 300 new photos and runs `dvc add data/camera-traps`. The output has changed, so DVC upgrades it to the 3.0 hash and builds the directory again in the 3.0 location. - Every photo, changed or not, is now stored under `files/md5/` while the 2.x copies remain — roughly another two terabytes of cache. ## What `dvc cache migrate` does `dvc cache migrate` forces the de-duplication the upgrade skipped: 1. Each object in the old location is **re-hashed** with the 3.0 algorithm. 2. It is **moved atomically** into the 3.0 location. 3. A **link** from the old path to the new object is left behind, so 2.x pointers still resolve. On file systems with no usable link type, the data is copied instead, and nothing is de-duplicated. Options: - `--dry` prints how many files would be re-hashed and migrated, without doing it. On a large cache, the run itself may take a long time. - `--dvc-files` also rewrites every DVC file in the **workspace** (`.dvc` files, `dvc.yaml` and `dvc.lock` entries) to the 3.0 format, so DVC uses only the 3.0 location. Commit those changes with Git. Git history is never rewritten: checking out an old commit still yields 2.x-format pointers. Remote storage has no equivalent command, because links between old and new locations are not possible on many storage systems; after migrating locally, the 3.0 objects have to be uploaded again and the outdated ones cleaned from the remote separately. ## Before and after the upgrade - Look for outputs without `hash: md5` in `.dvc` files and `dvc.lock`; each one still points into the 2.x layout. - Run `dvc cache migrate --dry`; it reports how many files would be re-hashed and migrated, which is a fair estimate of the time and headroom the real run needs. - Make text outputs deterministic: the DVC docs advise writing files with an explicit `\n` or `\r\n` rather than the platform default, so a stage run on Windows and on Linux produces one version, not two. ## Mistakes interviewers listen for - Claiming DVC 3 cannot read a 2.x cache. It can; it just does not de-duplicate against it. - Saying DVC 3 switched hash algorithms. It still uses MD5 — the change is that line endings are no longer normalised, plus the new location. - Expecting `dvc cache migrate` to touch old commits, or remote storage. - Forgetting the line-ending change when the same CSV is produced on Windows and Linux: under 3.x the two copies are different versions.
- Why can the lab's `labels.csv` show a different hash under DVC 3 even though nobody edited it on purpose?If the file has CRLF line endings, DVC 2.x hashed it after converting them to LF, while DVC 3 hashes the raw bytes, so the 3.0 hash differs. A stage or tool that writes the CSV with platform-dependent line endings also yields different versions on Windows and Linux; the fix is to write one line-ending style explicitly.
- After `dvc cache migrate --dvc-files`, can you still check out a commit from before the upgrade?Yes. Git history is not rewritten, so the old commit's pointers are still in 2.x format, and DVC 3 reads them through the old cache paths — which now link to the migrated objects.
- DVC 3 cannot read data cached by DVC 2.x.
- dvc cache migrate rewrites old Git commits into the new format.
- DVC 3 switched from MD5 to a different hash algorithm.
- Upgrading to DVC 3 moves the old cache automatically.
- dvc cache migrate also migrates the data in remote storage.
Assumes DVC 3.x, checked against 3.67; the files/md5 cache location, raw-byte hashing and dvc cache migrate all arrived with DVC 3.0, and 2.x pointers remain readable.