GitLab CI jobs on an autoscaled runner fleet almost never get a cache hit, even though `cache:` is configured correctly. Why, and what fixes it?

## Two halves again: the key and the location GitLab's caching splits cleanly. `.gitlab-ci.yml` says what to cache and how to name it; the runner says where that archive physically lives. Job authors only ever see the first half, which is why "the cache config is correct" and "there are no cache hits" are perfectly compatible statements. After the script finishes, the runner's helper archives the cached paths and stores the archive under a path derived from the key. On the next job with the same key, it looks the archive up and extracts it before the script runs. Everything interesting happens in the lookup. ## Where the archive goes by default With no cache configuration on the runner, storage is **local to the runner host**. For the docker executor that means a volume on the machine running the daemon; for the shell executor a directory under the runner's cache path. The consequence is simple: a cache hit requires the next job to run on the *same host*. That works fine for a couple of long-lived Docker runners — in fact it works very well, because there is no network transfer at all. It fails completely for: - **Autoscaled fleets** where a VM is created for a job and destroyed after it. - **The Kubernetes executor**, where each job is a fresh pod, possibly on a fresh node. - **Any multi-host fleet**, where the job simply lands somewhere else half the time and hit rate becomes a coin toss. The failure is quiet. The job log shows the cache being created and uploaded every time, the pipeline is merely slower than it should be, and nobody notices because nothing is red. ## Distributed cache The fix is to point the runner at object storage: ```toml [[runners]] name = "k8s-runner" executor = "kubernetes" [runners.cache] Type = "s3" Shared = true [runners.cache.s3] ServerAddress = "s3.amazonaws.com" BucketName = "gitlab-runner-cache" BucketLocation = "eu-central-1" AuthenticationType = "iam" ``` Two details carry the weight. `Type` selects the backend — S3-compatible storage (including MinIO) is the common one, with GCS and Azure also supported. `Shared = true` removes the runner-token component from the cache object path, so *every* runner in the fleet reads and writes the same entries; leave it false and each runner effectively keeps its own private namespace in the same bucket, which reproduces the original problem with an extra network hop. Credentials should come from an instance role or workload identity (`AuthenticationType = "iam"`) rather than static keys pasted into `config.toml`, since that file sits on every runner host. ## The tradeoffs nobody mentions Distributed cache is not free: - **Latency and egress.** Every job downloads and re-uploads an archive. For a 2 GB `node_modules`, that can be slower than a fresh install from a nearby package mirror. Measure before assuming caching helps. - **Unbounded growth.** Object storage keeps every key forever unless you add a lifecycle policy to expire old objects. Cache buckets quietly become the largest line on a storage bill. - **Trust.** Anything a job can write, a later job will execute — cache poisoning is real. Scope caches so an untrusted merge-request pipeline cannot write an entry that a protected-branch job will then restore, and never cache build outputs you intend to ship: promote a built artifact instead of trusting a cache to reproduce it. - **`Shared = true` means shared.** Two projects using the same key now collide. Keys must carry enough identity to keep projects and branches apart. ## Diagnosing it Read the job log rather than guessing. It states whether a cache was found for the key and where it was fetched from, and whether the archive was created and uploaded at the end. Two consecutive jobs on the same branch that both report creating the cache and never restoring it are telling you the storage is local while the compute is disposable. Cross-check with the executor: if the fleet creates a pod or VM per job, local cache was never going to work.