A fleet of 40 CI build agents behind a single NAT gateway intermittently fails image pulls during peak hours with HTTP 429 from the registry. Walk through how you diagnose it and what you change so it stops recurring.

## Read the actual error before theorising HTTP 429 from a registry is a specific claim: the server accepted your request and refused it for quota. That rules out most of the usual suspects — DNS, MTU, proxy interception, TLS. Get the raw line from the build log or `journalctl -u docker`; `toomanyrequests` names the cause unambiguously. Distinguish it from 401/403 (auth) and 5xx (upstream trouble), because the remediation is completely different. ## Why 'intermittent, at peak' is the tell Anonymous registry quota on Docker Hub is bucketed **per source IPv4 address**. Forty agents behind one NAT gateway present one address, so the fleet shares a single counter that refills on a time window. Under light load the fleet stays under the ceiling; at peak — a merge queue, a nightly matrix, an incident-driven redeploy — concurrency multiplies pulls per unit time and the bucket empties. The failures then look random, hitting whichever job happens to pull next, which is why teams misdiagnose it as 'flaky network'. A second amplifier: ephemeral agents. If every job starts from a clean container/VM with an empty image cache, *every* job pays full pull cost. Persistent agents with warm caches quietly hide the problem until you switch to ephemeral runners — a classic 'it broke when we migrated the runners' story. ## Confirm attribution Check whether pulls are authenticated at all: is there a `docker login` step, does `~/.docker/config.json` on the agent contain an auth entry, does the Kubernetes-based runner have an image pull secret? Then measure: fetch a token for the throwaway `ratelimitpreview/test` repository and read the `ratelimit-limit` / `ratelimit-remaining` headers from an agent during peak. Watching remaining go to zero closes the case, and repeating the check after `docker login` proves the change worked. ## The fix, in cost order **1. Authenticate.** A machine account with a scoped read-only access token, injected as a CI secret, moves the fleet off per-IP accounting. This alone often ends the incident. Never use a human's personal credentials — rotation and offboarding will bite you. **2. Cache.** A pull-through mirror in the same network as the agents turns N pulls into approximately one upstream pull per image per cache window, and cuts pull latency substantially. Give the cache upstream credentials, or it becomes the new single IP hitting the anonymous ceiling. **3. Reduce demand.** Bake the ten images that account for most pulls into the agent base image or a warm cache volume. Pin base images **by digest** so builds are reproducible and cheap to reason about. Audit for pathological patterns: a `docker pull` in every matrix leg, a compose file re-pulling sidecars, `--pull=always` where it is not needed. **4. Buy headroom.** A paid registry plan or a hosted registry with generous limits is usually cheaper than the engineering time lost to flaky builds. Frame it that way to management. ## Make it stay fixed Export metrics: count of 429 responses seen by agents, mirror hit ratio, upstream pull volume. Alert on the 429 rate crossing zero rather than waiting for red builds. Add a CI preflight that fails fast with a clear message ('registry quota exhausted') instead of a confusing mid-build failure. Document the machine account and its token rotation, because an expired token silently reverts the whole fleet to anonymous pulls — and the symptom will look exactly like this incident again. ## What not to do Do not add blind retries: the window is time-based, so retries prolong exhaustion and hide the signal. Do not spread agents across extra NAT addresses to dodge per-IP counting — it is quota evasion, it violates terms, and it fails again the moment the fleet grows.