skip to content

Remotes & Shared Storage

dvc remote add points a project at S3, GCS, Azure Blob or SSH, dvc push and pull move cached bytes, and secrets stay in .dvc/config.local. Interviewers probe shared caches and leaked credentials.

on this pageshow

explore

questions

5

In DVC, why does a fresh `git clone` of a project lack its tracked data, and how do `dvc push` and `dvc pull` move it?

level: juniorimportance: must knowfreq 30%

answer

  1. Git carries pointers, not bytes
  2. a registered storage location
  3. cache to remote, remote to cache
  4. pull is fetch plus checkout
  5. status with --cloud before pushing

basics

~20 s

Git carries only the small .dvc and dvc.lock pointer files; the bytes stay in each machine's DVC cache. dvc push uploads the cached objects a remote lacks; dvc pull downloads them and places them in the workspace.

solid answer

~40 s

A clone brings code, `.dvc` files and `dvc.lock`, which record each output's path and content hash, but the data lives in the cache (`.dvc/cache` by default), which Git ignores. A DVC remote, registered with `dvc remote add -d ci-s3 s3://example-lidar-scans/dvc` and committed in `.dvc/config`, is where those bytes are shared. `dvc push` finds the outputs the current metafiles reference, works out which cached objects the remote is missing and uploads only those; it never uploads code or metafiles, which `git push` handles. `dvc pull` is `dvc fetch` (remote to cache) followed by `dvc checkout` (cache to workspace). `dvc status --cloud` shows what the remote lacks before you push.

code

bash · 11 lines
bash
# once, in the project
dvc remote add -d ci-s3 s3://example-lidar-scans/dvc
git add .dvc/config && git commit -m "Add DVC remote"

# after tracking new data
dvc status --cloud     # lists objects the remote lacks as new
dvc push               # uploads only those objects
git push               # code and pointer files travel separately

# on a fresh clone
dvc pull               # fetch into the cache, then checkout into the workspace

go deeper

for a junior

Recall the split: Git moves code and pointer files, a DVC remote holds the bytes. Know that push uploads from the cache and pull downloads and places data in the workspace.

for a middle

Explain pull as fetch plus checkout, how the remote is chosen (output field, -r, core.remote), and what new, deleted and missing mean in dvc status --cloud.

for a senior

Show you can diagnose a CI pull that finds missing objects, and guard against it with status checks or the pre-push hook from dvc install rather than reminders.

for a principal

Weigh where the team's remotes live, which machines may write to them, and how push discipline is enforced across many repositories sharing one bucket.

## What a clone brings and what it leaves behind In a DVC project, Git versions the code and the small **metafiles**: `.dvc` files and `dvc.lock`, which record each tracked output's path and content hash. The bytes themselves live in the project's **cache** (by default `.dvc/cache`), and DVC keeps that directory out of Git. A teammate who runs `git clone` therefore gets every pointer and none of the data. Something else has to move the bytes between machines, and that is a **DVC remote**. - A remote is a storage location registered in DVC configuration: an S3, Google Cloud Storage or Azure Blob container, an SSH server, a WebDAV endpoint, or a plain directory such as a mounted network share. - DVC does not provide or recommend a storage service; the team brings its own. - The remote is laid out like the cache, by content hash, so it is not a browsable copy of the workspace. ## Registering a remote ```bash dvc remote add -d ci-s3 s3://example-lidar-scans/dvc git add .dvc/config && git commit -m "Add DVC remote" ``` `dvc remote add` writes a `remote` section to `.dvc/config`; `-d` / `--default` also sets `core.remote`, the remote every data command uses when none is named. The URL's scheme decides the storage type: `s3://`, `gs://`, `azure://`, `ssh://`, `webdav://` or `webdavs://`, or a local or mounted path. An S3-compatible server keeps the `s3://` URL and adds an `endpointurl`. A pip install may need the matching extra, such as `dvc[s3]`. Committing `.dvc/config` is what tells every clone where the data lives. ## dvc push: from your cache to the remote Without targets, `dvc push` collects the outputs referenced by the current `dvc.yaml` and `.dvc` files, looks up their hashes in the cache, asks the remote which objects it is missing, and uploads only those. It does not upload code, `dvc.yaml` or `.dvc` files; that is `git push`'s job. Useful options: - `-r <name>` pushes to a named remote instead of the default. - `-j <number>` sets parallelism; the default is `4 * cpu_count()`, and `dvc remote modify <name> jobs` changes it per remote. - `-a`, `-T` and `-A` widen the scope to all branches, all tags or all commits. The remote a command uses is resolved in this order: 1. a `remote` field written on the output in `dvc.yaml` or the `.dvc` file; 2. the `-r` / `--remote` option; 3. `core.remote`, the default set by `dvc remote add -d` or `dvc remote default`. With none of these set, the command stops with a "no remote specified" error that suggests `dvc remote default` or `-r`. ## dvc pull: fetch, then checkout `dvc pull` has the same effect as `dvc fetch` followed by `dvc checkout`. The fetch half downloads missing objects from the remote into the cache; the checkout half places them in the workspace, linking or copying according to the cache's link-type setting. `dvc fetch` on its own is useful when you want data for several branches in the cache without touching the workspace. Targets (a path, a `.dvc` file or a stage name) limit what is pulled, which matters on a dataset of millions of files. `--allow-missing` lets a pull succeed when some objects are absent from the remote, which is useful in CI jobs that need only part of the data but hides real gaps if used everywhere. ## Seeing what a remote lacks `dvc status --cloud` (`-c`) compares the cache with the default remote; `-r <name>` compares with another one and implies `--cloud`. When nothing differs it prints that the cache and the remote are in sync. Otherwise it lists states: | State | Meaning | Fix | |---|---|---| | `new` | in the local cache, not in the remote | `dvc push` | | `deleted` | in the remote, not in the local cache | `dvc pull` or `dvc fetch` | | `missing` | in neither the cache nor the remote | regenerate it, then push | ## The classic failure A teammate commits a new `lidar-scans.dvc`, runs `git push`, and forgets `dvc push`. CI clones the commit and `dvc pull` reports the files as missing, because the pointer arrived and the bytes never left the laptop. The fix is a `dvc push` from the machine whose cache holds them; the guard is running `dvc status --cloud` before pushing code, or `dvc install`, whose `pre-push` Git hook runs `dvc push` whenever `git push` runs.

  • What does `dvc fetch` do that `dvc pull` does not, and when would you use it?
    `dvc fetch` downloads objects from the remote into the cache only; `dvc pull` is fetch plus `dvc checkout` into the workspace. Fetch is handy after a fresh clone when you want several branches' or tags' data cached (`-a`, `-T`), before comparing metrics across commits, or when you want to keep the workspace untouched.
  • CI's `dvc pull` reports files missing that a teammate has on a laptop. What happened, and how do you prevent it?
    The teammate committed and pushed the pointer with Git but never ran `dvc push`, so the remote never received the objects. Run `dvc push` from the machine that has them. To prevent it, check `dvc status --cloud` before `git push`, or run `dvc install`, whose `pre-push` hook runs `dvc push` on every `git push`.
  • A project has an SSH remote and an S3 remote. Which one does a bare `dvc push` use?
    An output's own `remote` field wins first, then `-r`, then `core.remote`. If no output names a remote and no default is set, `dvc push` stops with a "no remote specified" error suggesting `dvc remote default` or `-r`; it never picks one of the two on its own.

  • git push uploads the DVC-tracked data together with the pointer files
  • dvc push also uploads the code and .dvc files to the remote
  • dvc pull only fills the cache and leaves the workspace untouched
  • dvc push re-uploads every tracked file each time it runs
  • DVC provides a hosted default remote, so dvc remote add is optional
open as a page

In DVC, an S3 secret key set with `dvc remote modify` ended up in a Git commit — why, and where should remote credentials live?

level: middleimportance: must knowfreq 16%

basics

~20 s

Without a level flag, dvc remote modify writes .dvc/config, the project file that Git tracks. Secrets belong in the Git-ignored .dvc/config.local (written with --local) or outside DVC config entirely: environment variables, a credentials file or profile, SSH config.

open as a page

In DVC, when do you use `dvc get`, `dvc import` or `dvc import-url` to bring data from another repository or a bucket into a project?

level: middleimportance: should knowfreq 12%

basics

~20 s

dvc get copies data from a DVC or Git repository once and tracks nothing. dvc import also writes an import .dvc file pinning the source repo and commit, refreshed by dvc update. dvc import-url does that for a storage URL.

open as a page

In DVC, when several workstations and a GPU server mount the same NAS, when should they share one cache directory instead of each running `dvc pull`, and what must be configured?

level: seniorimportance: should knowfreq 8%

basics

~20 s

Share one cache when all machines mount the same storage: point cache.dir at it, set cache.shared group and a symlink cache.type, so data is linked, not transferred. Keep a remote for other machines and as a second copy.

open as a page

In DVC, CI's `dvc pull` succeeds for the project's own S3 remote but fails on a directory added with `dvc import` — why, and how do you fix it without committing secrets?

level: seniorimportance: nice to knowfreq 5%

basics

~20 s

dvc push skips data from dvc import, so dvc pull fetches it from the source repository's default remote, which CI may not reach. Select a reachable source remote with --remote and pass secrets via --config with a Git-ignored file.

open as a page