How would you design a Docker disk-reclamation policy for a shared build fleet?

## Decide what the host is for before you decide what to delete The first architectural move is refusing to write one policy. Two host classes behave in opposite ways: - **Build runners.** The build cache is the valuable object — it is what keeps a large Python inference image with CUDA wheels building in three minutes instead of half an hour. Images produced locally are disposable the moment they are pushed. Containers are short-lived. Volumes are usually irrelevant. - **Service hosts.** There is no build cache. Images are the valuable object, because deleting the wrong one means an outage-length re-pull at the worst moment, and volumes hold state that must never be swept. Running both classes on the same machine is the root cause of most disk incidents I would design away: a build filling the disk should never be able to threaten an order-checkout API on a 340 ms p99 budget. If they must share, they at least should not share a filesystem. ## Make reclamation continuous, not heroic A policy that only runs when someone is paged is not a policy. The shape that works is a timer, several times a day, running the individually-scoped prunes with age filters, in increasing order of risk, non-interactively: ``` docker container prune -f --filter "until=48h" docker image prune -f docker image prune -a -f --filter "until=336h" --filter "label!=retention" docker builder prune -f --filter "until=168h" ``` Three design choices are embedded there. First, individual commands rather than `docker system prune`, so each step's blast radius is visible and one of them can be dropped per host class. Second, **age filters everywhere** — `until=` is what converts a destructive command into a retention rule, and the numbers are policy: two days of stopped containers, two weeks of images, a week of unused cache. Third, **label-based exclusion**: builds stamp `LABEL retention=keep` on the images that must never be re-pulled at 3 a.m. — the heavy base images and the current release — and `--filter "label!=retention"` spares them. Labels are how you express intent that an age filter cannot. ## Budget the build cache instead of wiping it Deleting build cache is free of correctness risk and expensive in CI minutes, which makes it a capacity decision rather than a cleanup. The better instrument than a scheduled wipe is a size budget: give the cache an allowance and let the daemon's own garbage collection evict least-recently-used records to stay inside it. `docker builder prune` also accepts a space-keeping flag for the same purpose when you drive it yourself. Either way, size the allowance from what you are buying: measure build duration with a warm cache versus cold, cost that in runner-minutes, and set the budget where the marginal gigabyte stops paying for itself. A team that cannot state that number has picked its retention by superstition. ## Make images cheap to lose The reason aggressive image pruning is frightening is usually that the local daemon has become the only copy of something. Fix the cause rather than the symptom: immutable, digest-addressable tags pushed to a registry for anything that could ever need a rollback, and a pull-through mirror close to the fleet so that re-pulling a 6.2 GB base costs seconds of LAN traffic instead of minutes of internet. Once that is true, `image prune -a` degrades from a hazard into a slow build, and the policy can be far more aggressive on space. ## Guardrails, ownership and blast radius A few rules I would make non-negotiable across the fleet: - **No `--volumes` in any automated job**, on any host that is shared or holds state. Volume reclamation is a human decision with a named owner, because the registry cannot give data back. - **No hand-deletion under the data root.** Every reclaim goes through the API, or the daemon's metadata and the filesystem disagree. - **The same policy is configuration, not tribal knowledge** — it ships with the host image, so a new runner is born with it rather than acquiring it after its first incident. ## Close the loop with signals Finally, export the buckets — the four rows of `docker system df` — as metrics per host alongside filesystem free space, and alert on the trajectory rather than a fixed threshold: a runner that will hit 90% in four hours is actionable, a runner that is at 78% is not. The success criterion for the whole policy is that nobody ever runs a prune during an incident, because the timer already did, and that the failure mode when the policy is too aggressive is a slower build rather than a service that cannot start.