skip to content

With Git LFS, how do you stop a CI job that clones the repository from downloading binaries no build step opens?

level: middleimportance: must knowfreq 44%

answer

  1. what runs per file at checkout
  2. an environment variable set before cloning
  3. pointers left in the working tree
  4. pull only the paths you need

basics

~20 s

Set GIT_LFS_SKIP_SMUDGE=1 before cloning so checkout leaves pointer files instead of downloading content, then run git lfs pull with --include for only the paths the build reads. A shallow clone does not reduce LFS downloads.

solid answer

~40 s

By default the LFS smudge step downloads every LFS object in the checked-out tree during `git clone`, one file at a time. In CI, export `GIT_LFS_SKIP_SMUDGE=1` before the clone: Git still checks out the tree, but LFS-tracked paths hold their small pointer files. Jobs that need some binaries then run `git lfs pull --include="levels/tutorial,shaders"` (or `-X` to exclude), and a committed `.lfsconfig` with `lfs.fetchexclude` keeps every clone away from a heavy folder. `--depth 1` does not help: LFS already downloads only the checked-out commit's objects, so the bytes come from the current tree, not from history. On hosts that meter LFS bandwidth, such as GitHub, CI downloads count against the repository owner, so this is a billing fix as well as a speed fix.

code

bash · 5 lines
bash
export GIT_LFS_SKIP_SMUDGE=1
git clone https://git.example.com/acme/game.git
cd game
# only this job reads binaries, and only these two folders
git lfs pull --include="levels/tutorial,shaders"

go deeper

for a junior

Recall that a normal clone downloads LFS content for the checked-out files, and that GIT_LFS_SKIP_SMUDGE=1 set before the clone leaves pointer files instead.

for a middle

Explain the sequence: skipped clone, then git lfs pull with --include or --exclude, and why a committed .lfsconfig rule applies to every clone while command-line flags apply to one run.

for a senior

Show you can cut a pipeline's LFS traffic job by job, explain why depth does not change LFS bytes, and tie the change to the host's bandwidth meter.

for a principal

Frame the tradeoff between per-job download rules, shared caches of the LFS store and restructuring which jobs need binaries at all, weighed against the bandwidth bill.

## Why a plain clone downloads every binary When a repository uses **Git LFS**, Git itself stores only small **pointer files** for the tracked paths; the real content lives on the **LFS server**. During `git clone` or `git checkout`, Git runs the LFS **smudge** step for each tracked path in the checked-out tree, and that step downloads the object if it is not already in the local LFS store (usually `.git/lfs/objects`). That is exactly right for an artist who opens the files. It is wasteful for a CI job that compiles code, runs unit tests or lints scripts: the clone pulls every texture, level and audio file in the current tree, and the job never opens one of them. A pipeline that clones the repository forty times an hour pays that download forty times an hour. ## Skipping the download: `GIT_LFS_SKIP_SMUDGE` The environment variable `GIT_LFS_SKIP_SMUDGE` tells Git LFS to skip the conversion from pointer to content. With it set to `1` (or `true`, `on`), Git still checks out the full tree, but every LFS-tracked path contains its pointer — three short text lines naming the object — instead of the binary. - The variable must be set **before** the clone or checkout runs; setting it afterwards changes nothing already written. - It affects both ways Git invokes LFS (`git lfs smudge` and `git lfs filter-process`). - A job that needs no binaries is finished: it never downloads LFS content at all. ## Downloading only what the job needs When a job needs some binaries, fetch them explicitly after the skipped clone: | Tool | Scope | Where it lives | |---|---|---| | `git lfs pull --include=<paths>` / `-I` | download and write only matching paths | one command in the job | | `git lfs pull --exclude=<paths>` / `-X` | everything except matching paths | one command in the job | | `lfs.fetchinclude` / `lfs.fetchexclude` | default filters for every fetch and pull | Git config or a committed `.lfsconfig` | Paths are comma-separated and matched with gitignore-style wildcards. The command-line options override the configured values for that one invocation, and an empty value clears them. Because `.lfsconfig` accepts `lfs.fetchinclude` and `lfs.fetchexclude`, a team can commit a rule such as "never fetch the raw source art folder" so every clone, CI or not, honours it. `git lfs pull` is broadly `git lfs fetch` followed by `git lfs checkout`, and it batches and parallelises downloads, which is faster than the per-file smudge a normal clone performs. ## Why a shallow clone does not fix it A common first attempt is `git clone --depth 1`. That trims **Git history**, which can matter for a large repository, but it does not touch the LFS download: by default LFS fetches objects for the **checked-out commit only**, never for older versions. The gigabytes come from the current tree, so a full clone and a depth-one clone download the same LFS content. How shallow and partial clones work is Git's own subject; here the point is only that they do not reduce LFS bytes. Another tempting key is `lfs.skipdownloaderrors` (or `GIT_LFS_SKIP_DOWNLOAD_ERRORS`). It makes checkout succeed when downloads **fail**, leaving pointers in place; it still attempts every download, so it saves no bandwidth and hides real errors from scripts. ## Why it matters on the bill Hosts that meter LFS usually meter **download bandwidth**. In GitHub's billing documentation, every LFS download counts against the **repository owner**, and the page names CI downloads explicitly. Forty clones an hour of a tree carrying 2 GB of LFS content is about 80 GB an hour of metered transfer. Other hosts set their own quotas, but the arithmetic is the same. ## Matching jobs to downloads Not every job needs the same content, so decide per job rather than per pipeline: - **Compile, lint and unit-test jobs** usually read no LFS content at all: skip the smudge step and never pull. - **Asset validation or packaging jobs** read a few folders: pull exactly those with `--include`. - **Full content builds** may need most of the tree: pull it, but keep the local LFS store between runs so only objects that changed since the last run are downloaded. - **Mirroring or backup jobs** are the rare case that needs every version: they run `git lfs fetch --all` deliberately, and nobody else should. A batched `git lfs pull` is also faster than the smudge path: smudge handles one file at a time as Git writes the tree, while pull asks the server for many objects in one request and downloads them in parallel. ## A checklist for a binary-heavy pipeline 1. Export `GIT_LFS_SKIP_SMUDGE=1` in every job before the clone. 2. Add `git lfs pull --include=...` only to the jobs that read binaries, listing the narrowest paths. 3. Commit `lfs.fetchexclude` in `.lfsconfig` for folders no build ever needs. 4. Keep the local LFS store between runs where the CI system allows a cache, so a repeated pull finds objects already present (the caching mechanism itself belongs to the CI tool). 5. Watch the host's bandwidth report after the change to confirm the drop.

  • With GIT_LFS_SKIP_SMUDGE set, what does a build step see if it opens a file it never pulled?
    It reads the pointer: a few lines of text starting with the LFS spec version line, not the asset. Tools then fail confusingly, for example an image loader reporting a corrupt header. Either add the path to the job's `git lfs pull --include`, or make the step fail fast when a file is still a pointer.
  • Where should include and exclude rules live: the command line, Git config or .lfsconfig?
    Per-job choices belong on the command line (`-I`/`-X`), which override configured values for that invocation. A rule every clone should follow belongs in a committed `.lfsconfig` as `lfs.fetchexclude` or `lfs.fetchinclude`, both on the short list of keys that file accepts. A developer's local Git config still overrides `.lfsconfig` when one machine needs something different.
  • Does setting lfs.skipdownloaderrors achieve the same saving?
    No. It only lets the smudge step tolerate failed downloads and leave pointers where they failed, so a checkout reports success. Every download is still attempted, so no bandwidth is saved, and scripts can no longer tell that content is missing.

  • Cloning with --depth 1 stops Git LFS from downloading large files.
  • lfs.skipdownloaderrors is how you skip LFS downloads in CI.
  • Setting GIT_LFS_SKIP_SMUDGE after the clone removes the downloaded binaries.
  • CI downloads never count against a host's LFS bandwidth quota.
  • A skipped clone fetches the real binaries automatically when a build opens them.