skip to content

In DVC, when several workstations and a GPU server mount the same NAS, when should they share one cache directory instead of each running `dvc pull`, and what must be configured?

level: seniorimportance: should knowfreq 8%

answer

  1. move bytes or point at them
  2. an external cache directory
  3. group permissions on new files
  4. link instead of copy
  5. one copy is not a backup

basics

~20 s

Share one cache when all machines mount the same storage: point cache.dir at it, set cache.shared group and a symlink cache.type, so data is linked, not transferred. Keep a remote for other machines and as a second copy.

solid answer

~40 s

With a remote, each machine `dvc pull`s its own copy into its own cache. With a **shared cache**, every project on the mounted storage points `cache.dir` at one directory (`dvc cache dir /mnt/nas/dvc-cache`), so data added on one machine is already there for the others with no transfer. Set `dvc config cache.shared group` so new cache files are `444` and directories `775`, make the directory group-owned (fixing existing files by hand), and set `cache.type symlink` so the workspace links into the external cache instead of copying. Linked files become read-only, and a `dvc gc` in one project can delete objects another needs. The remote still serves CI and remote machines, and it is the only second copy.

code

bash · 9 lines
bash
# on shared storage, group-owned
sudo mkdir -p /mnt/nas/dvc-cache
sudo chown -R :lidar /mnt/nas/dvc-cache

# in the project
dvc cache dir /mnt/nas/dvc-cache       # or: dvc cache dir --local ... per machine
dvc config cache.shared group          # new files 444, directories 775
dvc config cache.type symlink          # link from the external cache instead of copying
git add .dvc/config && git commit -m "Use shared DVC cache on the NAS"

go deeper

for a junior

Recall that a cache directory can live outside the project and be shared by everyone who mounts it, so nobody has to pull the same data twice.

for a middle

Explain the setup steps: dvc cache dir, cache.shared group, a symlink link type, and why files linked from the cache become read-only.

for a senior

Judge when a shared cache pays off, guard against gc deleting other projects' data and against permission drift, and keep a remote for CI and recovery.

for a principal

Plan storage for several teams on one NAS: shared caches per group or one for all, who may garbage-collect, and where the independent copy lives.

## Two ways to share bytes DVC can share tracked data between machines in two ways. - **A remote:** each machine keeps its own cache and moves objects with `dvc push` and `dvc pull`. Every machine stores a full copy of what it uses, and every new checkout costs a transfer. - **A shared cache:** every project and clone on the same mounted storage uses one cache directory outside its workspace, called an external cache. An object added on the GPU server is immediately available to the workstations; nothing is pushed or pulled between them. The robotics team has both shapes at once: the office NAS is reachable from the workstations and the GPU server as a mount, while CI and laptops off the office network can only reach the S3 remote. ## Setting up a shared cache 1. Create a directory on the shared storage, owned by a group every user belongs to. 2. If projects already have local caches, move their contents in (`mv .dvc/cache/* /mnt/nas/dvc-cache`) and fix permissions: directories `0775`, files `0444`, group ownership. 3. Point the project at it: `dvc cache dir /mnt/nas/dvc-cache`. An absolute path is stored as given; a relative one is stored relative to the config file. 4. `dvc config cache.shared group` makes DVC create new cache files as `444` and directories as `775`; without it, permissions follow the user's umask. 5. `dvc config cache.type symlink`, then commit `.dvc/config`. Existing workspaces relink with `dvc checkout --relink`. If the mount point differs between machines, write `cache.dir` with `dvc cache dir --local` on each machine instead, so the tracked file does not carry one machine's path. ## Why the link type matters The default `cache.type` is `reflink,copy`. Reflinks need a supporting file system on the same volume, so with a cache on a network mount DVC usually falls back to copying every file into the workspace, which defeats the point. Hard links also require the same partition. Symbolic links work across file systems, which is why DVC's guide pairs a shared cache with `symlink`. The price is that symlinked and hard-linked files are protected (read-only) because editing them in place would corrupt the cache; changing one means unprotecting or replacing it first. ## Hazards - **Garbage collection:** `dvc gc` in one project only knows that project's references and can delete objects another project still needs, so collection on a shared cache needs an agreed owner and scope. - **One copy:** a shared cache has no second copy. If the NAS volume fails, the data is gone unless it was also pushed to a remote. - **Permissions drift:** files written before `cache.shared group` was set, or by a tool that ignores it, can block other users. - **I/O path:** training reads now go over the network share; whether that is fast enough depends on the NAS, not on DVC. ## What a shared cache does not change - The pointer files are the same: `.dvc` files and `dvc.lock` still record hashes, and Git still carries them. - `dvc push` still uploads from the cache, now the shared one, to whichever remote is configured, so one push from any machine on the mount serves CI. - `dvc status --cloud` still compares the cache with the remote and is the way to see what the off-network machines cannot get yet. - Remote credentials are still per user or per machine; sharing the cache shares bytes, not access to the bucket. ## Choosing | Question | Shared cache | Remote plus `dvc pull` | |---|---|---| | Machines share a mount? | required | not needed | | Transfer per checkout | none | missing objects only | | Disk use | one copy for everyone | one copy per machine | | Works for CI and laptops off the network | no | yes | | Independent second copy | no | yes | | Main hazard | gc and permissions across projects | stale or unpushed data | Most teams with a NAS use both: the shared cache for machines on the mount, and a remote that someone pushes to for everyone else and for recovery. The NAS can even host that remote, as an SSH or mounted-path remote in a separate directory, but then it is not an independent copy either.

  • What exactly does `cache.shared group` change?
    It makes DVC create new cache files with `444` and new directories with `775`, so every member of the owning group can read others' objects and add new ones. It does not fix files created before it was set, and it does not set group ownership; that is done on the directory itself.
  • Why does a team with a shared cache still keep a remote?
    CI runners and laptops off the office network cannot mount the NAS, so they still `dvc pull` from a remote. And the shared cache is a single copy: if that volume is lost, only data that was pushed to a remote survives.
  • The NAS is mounted at different paths on the GPU server and the workstations. Where do you set the cache path?
    Per machine, with `dvc cache dir --local <path>`, which writes `cache.dir` to the Git-ignored `.dvc/config.local`. Writing it with the default level would commit one machine's path to `.dvc/config` for everyone.

  • A shared cache directory removes the need for any remote
  • Pointing cache.dir at a shared path makes DVC fix permissions on existing files
  • Running dvc gc in one project is safe on a shared cache
  • A shared cache has to be configured as an ssh:// remote
  • A shared cache gives the team a second, independent copy of the data