A team reports that their image build 'never hits the cache' even though they changed almost nothing between runs. How would you diagnose which instruction invalidates first, and what are the usual causes?

## Step 1: find the first miss, ignore the rest Cache invalidation cascades: once one instruction misses, everything after it must rebuild. So a build log full of misses usually contains exactly **one** interesting event — the first one. Debugging any later step is wasted effort. `docker buildx build --progress=plain .` prints each step with `CACHED` where the cache was used. Read top-down and stop at the first step without it. Compare two consecutive builds' logs; the boundary moves as you fix causes. Useful supporting probes: `docker buildx du` shows what the builder is actually holding; `docker history <image>` shows the layer chain and sizes; a `--no-cache` run gives you a cold-build baseline so you know what the cache is worth. If the whole build is uncached in CI but cached locally, the problem is the environment, not the Dockerfile. ## Step 2: interrogate the inputs of that instruction ### If it is a COPY or ADD The key is a content checksum over the matched files (content, path, mode/ownership under BuildKit). Something in that set genuinely changed. Common real causes: - **No or weak `.dockerignore`.** `COPY . .` with `.git` included means any fetch, any branch switch, any new commit object changes the context. `node_modules`, `target/`, `build/`, `.venv`, `coverage/` and stray OS metadata files do the same. Excluding them is usually the single biggest fix — it also shrinks context upload time. - **Generated files inside the context.** A pre-build step that writes a version file, a timestamp, a `.env`, or compiled assets guarantees a new checksum every run. - **Permission churn.** A CI checkout that yields different file modes than a developer's machine changes the metadata portion of the key, so cache exported by one environment never matches the other. - **Line-ending or formatting rewrites** (CRLF conversion, an autoformatter running before the build) change bytes. ### If it is a RUN The key is the expanded command string plus the environment. Look for: - **A volatile referenced `ARG`.** `ARG GIT_SHA` or `ARG BUILD_TIME` consumed near the top of the file changes every build and takes the entire stage with it. Move it below the expensive steps. - **A changed `ENV`, `WORKDIR`, `USER` or `SHELL`** above the step. - **Deliberate cache-busting left behind** — someone's temporary `--build-arg CACHEBUST=$(date +%s)` that stayed in the pipeline. ### If the miss is at the very top - **The base image moved.** `FROM node:22` is a mutable tag. With `--pull`, or on a runner that resolves it fresh, you can be building on a different digest than the cached layers assumed. Pinning by digest makes this deterministic and turns 'mystery cold build' into an explicit, reviewed bump. - **Platform mismatch.** Cache is per platform. A cache produced for `linux/amd64` gives nothing to a `linux/arm64` build, and multi-platform builds need cache for each. ## Step 3: environment causes — when nothing in the Dockerfile is wrong If the file is fine and it caches locally but never in CI: - **A fresh builder per job** with no `--cache-from`, so there is simply no cache to hit. - **Import/export reference mismatch** — the job writes `app:cache-$BRANCH` and reads `app:cache-main`, or reads a tag no job ever wrote. - **`mode=min` on a multi-stage build**, so builder-stage layers were never exported. - **Eviction / GC.** Provider caches evict under size pressure; a shared builder with a keep-storage limit or a scheduled prune drops old entries. Long gaps between builds then look like random cold builds. - **Parallel jobs racing** to export to the same reference, each overwriting the other's entry. - **A stray `--no-cache`** in the pipeline definition, often added during an incident and never removed. ## Step 4: verify with a controlled experiment Build twice back to back with no changes. If the second build is fully cached, the Dockerfile is fine and the problem is per-run input or environment. If the second build still misses, something inside the build is non-deterministic in a way that feeds a key — usually a generated file or an argument. ## Step 5: fix in order of leverage Add or tighten `.dockerignore`; move volatile arguments and labels to the bottom; split wide copies into narrow ones (manifest first); pin base images by digest; export cache with `mode=max` and a scope that actually matches on import; give a persistent builder a sane GC policy. Then re-measure hit rate and wall-clock time — the point is faster builds, not a prettier log.