A team enables shared remote caching in their monorepo build tool. Soon after, engineers start seeing CI report a task as passing via a cache hit, even though the same code fails when run fresh locally. What's the likely root cause of this class of bug, and how do you prevent it going forward?

## The bug class This bug class — "CI says green via cache, reality says red" — is one of the most dangerous failure modes in a shared-caching build system precisely because it looks identical to a healthy fast build from the outside. Understanding it requires being precise about what a **cache key** actually represents versus what a task's **real inputs** are. ## What the key represents, and what it misses The orchestrator computes a cache key from a declared input set: it hashes the source files the task lists as inputs, the transitive hashes of its declared dependencies, the exact command and flags, and sometimes the toolchain version. That key is a proxy for "everything that could affect this task's output" — but it's only accurate if the declared set genuinely matches the actual set of things the task's behavior depends on. A task can secretly depend on something outside that declared set: - an environment variable read by the test framework to toggle a feature flag; - the system clock via `Date.now()` embedded in an assertion; - a live network call to a staging service whose response varies; - unordered map/set iteration in a language where that's not guaranteed; - a random seed not fixed for the test run. In each of these, the cache key stays identical across runs where that hidden factor actually differs. The tool, seeing an unchanged key, concludes it already has the answer and replays the old result instead of re-executing, and if the hidden factor's new value would have produced a failure, that failure is masked. It doesn't just slow down bug discovery — it removes any signal that a bug exists at all, until someone hits the actual broken behavior in production and starts asking why CI didn't catch it, at which point "well it did pass" is technically true and deeply unhelpful. ## Why the risk exists at all This exists as a risk precisely because caching's whole value proposition is trusting a hash instead of re-executing — the faster and more aggressive the caching, the more damage an undetected hermeticity gap does, since a single bad cache entry gets replayed for every subsequent matching key until someone invalidates it or the input set finally changes for an unrelated reason. ## The trade-off: enforcement cost versus risk tolerance The trade-off engineering teams navigate here is **enforcement cost versus risk tolerance**. The strongest prevention is structural: run tasks inside a sandbox that only exposes declared inputs and blocks everything else by construction — - no network access, - no filesystem access outside the declared input set, - environment variables scrubbed to an explicit allow-list. Bazel does this by default for build actions and can be configured to sandbox test execution too, which is precisely why organizations with strict correctness requirements lean on it. The cost is real: sandboxing breaks tests or build steps that legitimately need network access (e.g., integration tests against a real staging API) unless those are explicitly carved out as non-cacheable or given a controlled, mocked substitute — which is more upfront engineering work than just letting the test run unconstrained. ## Convention instead of enforcement Lighter tools without built-in sandboxing (Nx, Turborepo) instead rely on convention and configuration: task authors declare `inputs`/`outputs` explicitly, and the org has to establish a practice of treating any environment- or network-dependent test as something to either fix (mock the dependency, fix the seed) or explicitly mark as non-cacheable / always-run. This is cheaper to set up but depends on ongoing engineering discipline rather than a structural guarantee, so it degrades as team size grows and fewer people remember the convention. ## Diagnosing an actual incident Diagnosing an actual incident typically starts by: 1. **forcing a cache bypass** (most tools offer a `--no-cache` or equivalent flag) to confirm the task genuinely behaves differently fresh versus cached; 2. then **auditing the task's declared inputs** against what it actually reads/calls at runtime — commonly surfaced by grepping the task's source for `process.env`, `Date.now()`, `Math.random()`, or outbound HTTP calls, and checking whether each is either fixed/mocked or explicitly declared as a cache-relevant input. ## Where it shows up A concrete, well-known real-world pattern of this exact bug class: teams using Bazel without sandboxing enabled for a given target (a common misconfiguration when a team disables sandboxing to "fix" a build that fails inside the sandbox, rather than fixing the underlying hermeticity violation) have reported exactly this symptom — a target that reads local user-specific config outside its sandboxed inputs, producing cache hits that mask a bug only reproducible with a clean environment, discovered days later when a new hire's fresh machine failed a "passing" target. The fix in that pattern is almost always to restore sandboxing and explicitly declare the missing input, not to disable caching altogether.