Someone plants a compiler shim in a shared runner's tool cache. How do you detect that builds were tampered with?

## Why this class is nasty A shim is a small program placed where a build expects to find a real tool - a compiler, a linker, a language-version launcher. It calls the real tool and adjusts something on the way past. On a shared persistent runner, the tool cache is often writable by the build account, so one repository's job can plant a shim that every later job on that host silently uses. What makes this hard to see is that all the usual evidence stays clean. The source is what was reviewed. The lockfile pins what was declared. An inventory of what is inside the artifact describes the components that were **declared and packaged**, so a tool that altered code generation on the way through typically does not appear in it. This is precisely the gap that a record of **how the artifact came to be** exists to close, as opposed to a record of **what is inside it**. ## Detection, in order of usefulness ### 1. Rebuild somewhere clean and compare Build the same commit on a fresh single-use machine from a known image, and compare the output to the suspect artifact. Any difference that is not explained by non-determinism in the build is your signal. This works only as far as your builds are comparable at all - if timestamps, file ordering and embedded paths vary run to run, you may need to compare disassembly or specific sections rather than whole files. Even a partial comparison is powerful, because the attacker's change is usually in a place a diff will reach. ### 2. Build records that live off the host For every artifact, record and store externally: the job identity, the commit, the host or runner identity, the build image identity, and the toolchain versions actually used. Two anomalies then become queryable rather than invisible - an artifact built on a host that should not have taken that job, and a build whose reported toolchain does not match the image it claimed to run in. This only works if the record is produced by something the job cannot rewrite and is stored where the host cannot reach back and edit it. ### 3. Host integrity monitoring Baseline the tool cache, any language shim directory and everything on the default `PATH`, and alert on writes. On a multi-tenant build host, a write to a shared toolchain path by a job is almost never legitimate. Ship those events off the machine as they happen. The reason is the same one that makes this incident hard: whoever planted the shim had write access to the host, so anything stored only on the host - including its own logs and its own baseline database - is evidence you cannot fully trust. ## Bounding the damage The first question after detection is always "what else did this touch", and it is answerable only if you kept **job-to-host assignment records**. Without them, you cannot say which repositories built on that machine and are reduced to treating every artifact from the whole pool in the window as suspect. With them, the blast radius is every job that ran on that host from the earliest possible plant time - which you should treat as the last time you know the host was clean, not the time you first noticed - plus everything those artifacts were promoted into or bundled inside downstream. ## Response - **Destroy the host, do not clean it.** You do not know everything that was installed, and a cleanup performed with the attacker's tooling still on `PATH` proves nothing. Rebuild from a known image. - **Rotate every credential that host could reach**, including any long-lived tokens on disk, machine identities and cached registry credentials. - **Rebuild and republish** affected artifacts from clean infrastructure, and compare the new outputs against what you shipped. - **Preserve a disk image first** if you want forensics, but do it before rebuilding, not after. ## Prevention beats all of it The reason this question is asked at senior level is that the good answer ends by conceding that detection here is expensive and partial. The cheaper controls are structural: deliver the toolchain inside the build image rather than caching it on the host, make toolchain paths not writable by the job account, keep mutually untrusted repositories off the same host, and use single-use machines so a plant has no next job to catch. ## Saying it in an interview Open with "how would I even know" - naming that the source, the lockfile and a component inventory are all blind to a substituted tool - then give the three detections, then bound the blast radius from assignment records, then destroy rather than clean. A candidate who jumps straight to "we would see it in the scan results" has missed what happened.