Each CI job runs in its own container on a shared host - what contamination does that not prevent?

## The assumption being tested "Every job runs in its own container, so jobs are isolated from each other" is one of the most common half-truths in CI. A container gives the job a fresh filesystem view and a fresh process tree. It does not give it a fresh **machine**, and the interesting contamination lives in the parts of the machine you deliberately shared to make builds fast or possible at all. ## What crosses the boundary ### Mounts you added on purpose Build caches only pay for themselves if they outlive the job, so they are bind-mounted or volume-mounted from the host: the dependency cache, the compiler or language tool cache, sometimes the workspace itself. A job writes there; the next job on that host reads there. If the mount is writable and shared across repositories, a job can plant a package or a tool in it and every later job on the host consumes it. The container boundary is irrelevant to a path that was mounted through it. ### A shared container daemon Jobs frequently need to build images. A common shortcut is to hand the job access to the host's container daemon so it can do that. The consequence for **cross-job** contamination is the point here: that daemon manages every container on the host, so a job holding it can enumerate other containers, inspect their configuration and environment, execute inside them and copy files out of them. That includes a **concurrently running** job from another repository - its checked-out source and the secrets that job was given. Once two jobs share a daemon, they are in the same trust domain no matter how neatly each sits in its own container. ### The local image store and mutable tags Build containers are started from images pulled or cached on the host. A tag is a **name**; a digest is a **content identifier**. If a job can write to the host's local image store, it can make a familiar tag resolve to different content for the next job's base image. Referencing base images by digest closes that specific hole; referencing by tag leaves it open. ### The host's network position and identity A container inherits where the host sits. Internal services reachable from the host are reachable from the container, and any machine identity the host holds - a credential the platform exposes to workloads on it - is typically reachable from inside the container too. That is not contamination between jobs in the filesystem sense, but it is access the job did not earn. ### Shared kernel and shared resources All containers on a host share one kernel and one set of resource limits. That produces two effects worth naming: a noisy or hostile job can degrade the others, and the strength of the boundary depends on how the container was configured - what was mounted, what privileges were granted, what the job account can do. This tree does not own the isolation primitives themselves; what it owns is the conclusion that the boundary is only as strong as what you did **not** share. ## Time versus concurrency A reused runner has two contamination axes and they need different answers. | Axis | What it means | What helps | | --- | --- | --- | | Across time | Job A finishes, job B starts on the same host and reads what A wrote | Destroy and reimage the host between jobs; scope caches per repository or trust tier | | Concurrently | Job A and job B run at the same time on one host | Do not share a daemon or a writable mount between trust domains; run mutually untrusted work on separate machines | People usually design for the first and forget the second, which is why the concurrent case is a good interview probe. ## What to actually do - Decide the **trust domain** first: if two workloads must not reach each other, one host each, not one container each. - Do not hand a job the host's container daemon across trust domains; if image building is needed on mixed pools, that need belongs on isolated hosts. - Make shared caches read-only to the job where possible, or key them per repository and per trust tier so a lower-trust job cannot write what a higher-trust job reads. - Pin base images by digest so a mutable tag in the local store cannot be repointed under you. - Assume the host's network position and machine identity are available to the job, and scope them accordingly. ## Saying it in an interview The crisp line is: a container is a boundary against **accident**, not automatically a boundary against an **adversary**, and every mount, socket and shared store you added for speed is a hole you punched in it on purpose.