Rolling back by redeploying the image tag `indexer:stable` brings the same broken build back. Why?

### What actually happened A release of a document-indexing service shipped build `7e1c94d`, and the pipeline did what it always does: pushed the build tag, then re-pointed the alias with `docker tag ... indexer:stable` and a second push. The build turned out to hang on shutdown — the Elixir release ignored SIGTERM, so every stop hit the 4-second grace period and was killed, losing partial index state. Twenty-three minutes into the incident somebody runs the rollback: redeploy `indexer:stable`. The same broken container comes back. This is not a tooling bug. It is the defining property of an alias. `stable` is not "the last good build" — it is "whatever was pushed to that name most recently", and the release you are trying to undo is what pushed it. **You cannot roll back to a name that the thing you are rolling back moved.** ### Why it can look even stranger than that Two effects often overlap and make the diagnosis muddier. First, **local caching**. `docker run registry.internal:5000/indexer:stable` uses a local image with that tag if one exists; it does not contact the registry to check whether the tag has moved. So on a host that already pulled the image, a rollback can appear to "work" while doing nothing at all. Force the question with an explicit `docker pull`, or `docker run --pull=always`. Second, **skew across hosts**. Hosts that pulled the alias before the release still hold the old bytes; hosts that pulled after hold the new ones. The fleet is then running two different builds under one name, and "is the rollback done?" has no single answer. This is the same root cause wearing a different costume. ### Diagnosing it in under a minute Work from the identity of the bytes, not from the name: ``` # what is this host actually running? docker inspect --format '{{.Image}}' indexer # what does the local tag point at, and does the registry agree? docker image inspect --format '{{.Id}}' registry.internal:5000/indexer:stable docker pull registry.internal:5000/indexer:stable docker image inspect --format '{{.Id}}' registry.internal:5000/indexer:stable ``` If the pull changes nothing and the container is still the bad build, the alias is pointing at the bad build. If the pull *does* change it, you had a stale cache as well. Either way, the fix path is the same. ### The fix during the incident Deploy the previous build by a name that nobody moved: ``` docker pull registry.internal:5000/indexer:2b83f0e docker run -d --name indexer registry.internal:5000/indexer:2b83f0e ``` That requires knowing which build was previous, which is the part teams most often lack. The registry does not keep the history of an alias — one pointer, no log. The answer has to come from somewhere that does keep history: the deployment record that says which immutable tag was rolled out and when. If your deploy tool records only "deployed indexer:stable", it has recorded nothing useful, and the incident now includes an archaeology step. Move the alias back **after** the rollback is verified healthy, not before. Otherwise anybody pulling `stable` mid-incident — a developer, a smoke test, a host that happens to restart — gets the build you are trying to remove. ### The fix afterwards Three changes turn this class of incident off: 1. **Deployments name immutable build tags.** The alias may exist, but nothing in the deploy path resolves it. This is the single change that does most of the work. 2. **Deploy records store the resolved build tag**, so "the previous release" is a lookup rather than a reconstruction. Ideally the rollback command reads it directly. 3. **Alias moves happen after a healthy deploy**, as a separate step, and are equally logged. A useful sanity question for any release process: *if the alias were deleted right now, could we still redeploy every release from the last month?* If the answer is no, the aliases are load-bearing and rollback is a rebuild in disguise. ### The distinction worth stating out loud Mutable names are fine — they are convenient and people want them. What is not fine is a mutable name in the rollback path. Rollback is the operation you perform when you are already wrong about something, under time pressure, so it must depend on the fewest possible things that can have changed since. A name written once by a build that has already run is about as few as it gets.