A build passes on your team's long-lived shared CI machines but fails on a freshly provisioned one. What kinds of leftover state on a reused runner cause that, and what removes the whole class of problem?

## The two failures of a reused runner A persistent runner executes many jobs on the same machine, in the same working directory, as the same user. That produces two distinct failures with one cause. The first is the *dirty workspace*: a build succeeds only because of something an earlier job left behind, so it fails anywhere else — on a colleague's laptop, on a new machine, or the day the runner is replaced. The build has an undeclared dependency, and the shared runner has been hiding it. The second is *cross-job disclosure*: whatever one job leaves readable, the next job can read. If those two jobs belong to different teams, one team's pipeline has just read the other's credentials with no exploit involved. ## Where the state actually hides Candidates usually name the workspace and stop. The full list is longer, and the entries below the workspace are the ones that cause the confusing failures. - **The workspace directory.** Untracked files, generated sources, stale compiled output, an old dependency tree that satisfies an import nobody declared. A clean checkout into a dirty directory is not a clean state. - **Globally installed tools.** A job that ran `npm install -g` or `pip install --user` or added a system package leaves that tool on the PATH for every later job. - **Version-manager state.** A tool-version manager switched to a version and left it as the default, so later jobs get a runtime nobody selected. - **Home-directory configuration and credentials.** This is the dangerous one: a package-registry token, a container-registry login, a cloud CLI's cached session, a git credential helper store, an ssh-agent socket. These are files written into the agent user's home by ordinary, correct steps, and they simply stay there. - **Processes and ports.** A database or mock server started for tests and never stopped will make the next job's tests pass — or fail with a port collision. - **Container and image state.** Cached images that no longer match their tag, dangling volumes, and eventually a full disk. - **The machine itself.** OS packages, DNS or hosts-file edits, kernel parameters, and a clock or locale nobody set deliberately. ```bash # Job 1, last week, on a reused runner: npm install -g some-internal-cli # global, not in package.json npm config set //registry.internal/:_authToken "$TOKEN" # writes ~/.npmrc # Job 2, today, same machine, different team: some-internal-cli build # "works" only because Job 1 installed it cat ~/.npmrc # and Job 1's token is still readable here ``` ## What actually removes the class The structural answer is a **single-use runner**: the instance is created (or registered) for exactly one job and destroyed when that job ends, whether it passed, failed, was cancelled, or the machine was killed mid-step. Nothing survives, because the thing that would hold the state no longer exists. Platforms implement this as an ephemeral or just-in-time registration mode, usually driven by an autoscaler that provisions a fresh VM or container per queued job. Weaker substitutes, in descending order of strength: 1. **Run each job inside a fresh container** on a persistent host. The workspace, PATH, home directory and processes are new; the host disk and any mounted caches are still shared, and a privileged job still escapes. 2. **Wipe the workspace at job start** rather than at the end. Better than a trailing cleanup, but it touches only the first item on the list above. 3. **A cleanup step at the end of the job.** The weakest, because it is skipped precisely when it matters — a cancelled job, a crashed step, a runner that lost its connection. Never rely on it for anything security-relevant. If you must keep persistent runners, at least separate pools by trust domain, so a job that leaves a credential behind can only leak it to jobs that were already entitled to it. ## The cost, and how to pay it honestly Single-use runners give up the warm state that made builds fast: the dependency cache, the container image cache, sometimes the source checkout itself. That is a real cost, and the answer is to make the fast path explicit rather than accidental — restore dependencies from the platform's cache service at the start of the job, pull base images through a local registry mirror, and bake slow-moving toolchain into the runner image. The point of the exercise is not that caching is bad; it is that a cache you declare is reproducible, while a cache that is merely left over is not.