skip to content

A custom cacheable task gets cache hits locally but never on CI. As a senior engineer, how do you reason about and fix the cause?

level: seniorimportance: should knowfreq 35%

answer

  1. local hit + CI miss = non-relocatable
  2. absolute path -> RELATIVE
  3. timestamp/SHA file -> normalization ignore
  4. classpath -> @Classpath
  5. caching.debug diff per-input hashes

basics

~20 s

Local hits but cross-machine misses almost always mean the cache key isn't relocatable: absolute paths, machine-specific input content, or timestamps leak in. Fix with relative path sensitivity and input normalization so keys match across machines.

solid answer

~50 s

Hits locally (same checkout) but misses on CI is the classic **non-relocatability** signature: something machine-specific is in the cache key, so the same logical inputs hash differently on two machines. I'd reason through the candidate leaks: (1) **absolute paths** — file inputs at `@PathSensitive(ABSOLUTE)` (or a default that keeps absolute paths) embed the checkout directory; switch to `RELATIVE`. (2) **volatile file content** — generated resources with timestamps/build numbers/commit hashes; strip them with `normalization { runtimeClasspath { ignore(...) } }` or by not feeding them as inputs. (3) **classpath order/timestamps** — use `@Classpath`/`@CompileClasspath` instead of plain `@InputFiles`. (4) **environment-dependent inputs** — JDK path, OS, env vars captured as `@Input`. I'd confirm by comparing the two keys: a build scan or `-Dorg.gradle.caching.debug=true` prints each input's hash, so I diff local vs CI to see which input differs, then normalize exactly that one. The principle: the key must encode semantics, not the machine.

code

bash · 4 lines
bash
# Capture per-input hashes on each machine, then diff to find the leak
./gradlew :app:myCacheableTask \
  -Dorg.gradle.caching.debug=true --info 2>&1 | tee local-keys.txt
# (run same commit on CI, save ci-keys.txt, diff the two)

go deeper

for a junior

Recognize the symptom (works locally, not on CI) points to paths in the key, but may not lead the full diagnosis.

for a middle

List the common leaks (absolute paths, timestamps, classpath order) and apply RELATIVE + normalization fixes.

for a senior

Systematically diff per-input hashes via caching debug/build scan, identify the exact leak, and fix without introducing false hits.

for a principal

Drive normalization standards and CI seeding strategy so cross-machine relocatability holds across the whole org, and review for correctness regressions.

## The signature **Local cache hits, zero CI hits** means each machine computes a *different* cache key for inputs that are logically identical. The task is cacheable (it stores entries) but the entries aren't **relocatable**. The whole value of a shared/remote cache is cross-machine reuse, so this is the bug that makes caching useless in practice. ## Reasoning checklist (most common first) 1. **Absolute paths in file inputs.** If a file input keeps absolute path information, `/home/ci/agent/work/...` differs from `/Users/dev/proj/...`. Fix: annotate file inputs `@PathSensitive(PathSensitivity.RELATIVE)` (or `NONE`/`NAME_ONLY` where appropriate). 2. **Volatile content inside inputs.** A `build-info.properties`, manifest with `Build-Date`, or a generated file embedding the git SHA changes every build, so the input hash changes every time. Fix: `normalization { runtimeClasspath { ignore("...") ; metaInf { ignoreAttribute("...") } } }`, or don't declare the volatile file as an input. 3. **Classpath noise.** Jar entry order and intra-jar timestamps differ across machines. Fix: declare classpaths with `@Classpath`/`@CompileClasspath` (which normalize), not plain `@InputFiles`. 4. **Environment captured as input.** A `@Input` holding `System.getProperty("java.home")`, an absolute tool path, or `System.getenv()` leaks the machine. Fix: capture only the semantic value (e.g. JDK *version*, not its path) or mark genuinely irrelevant ones `@Internal`. 5. **Output declarations including absolute-path side data** that round-trips into the next task's input. ## How to confirm — diff the keys The decisive move is to make the two keys observable and diff them: ```bash ./gradlew myTask -Dorg.gradle.caching.debug=true --info ``` With caching debug on, Gradle logs, per task, each input property and its individual hash. Capture this on a local run and a CI run for the same commit, then diff. Exactly the input whose hash differs is your leak — you don't have to guess. Build scans present the same per-input hashes in a nicer UI and can compare two builds. ## The mental model A cache key should encode the **semantics** of the inputs (what content/values actually affect the output), never the **environment** that produced them (paths, timestamps, machine identity). Every fix above is an instance of removing environment from the key while preserving semantics. Over-correcting — normalizing away something that *does* affect the output — swings you into **false hits**, so each normalization must be justified against "does this change the result?" ## After the fix Once keys match, CI runs that push to the shared cache seed entries that developers (and other agents) restore. Validate by running the task on a clean checkout on a second machine and confirming `FROM-CACHE`.

  • How would you actually pinpoint which input differs between local and CI rather than guessing?
    Enable -Dorg.gradle.caching.debug=true (or use a build scan) to log each input property's individual hash, run the same commit on both machines, and diff the per-input hashes. The property whose hash differs is the non-relocatable leak.
  • After fixing path sensitivity you now get a CI hit that produces wrong output. What likely happened?
    You over-normalized — e.g. set NONE/NAME_ONLY or ignored a resource that genuinely affects the result — so two different real inputs collide on one key (a false hit). Tighten normalization back so the meaningful input is in the key.

Like hashing a function with the machine's hostname accidentally mixed in — same arguments, different hash on every host, so no two machines ever share a memo.

saying these in an interview costs you the question

  • Blaming the remote cache server instead of the task's key composition.
  • Fixing by disabling caching rather than making the key relocatable.
  • Over-normalizing to force hits and introducing false hits / wrong outputs.

context

resolving links…