Rollback Readiness

## The release is bigger than the artifact When a team says "we can roll back", they usually mean "we can redeploy the previous image". But a release is everything that changed in production as part of shipping that version: - the **code artifact** (image, package, bundle), - the **configuration** it reads - timeouts, pool sizes, endpoints, routing weights, resource limits, - **schema and data** changes, - **client-side assets** already cached in browsers or CDNs, - **messages already published** in a new format, - and any **runtime state** the new version created. A rollback reverses only the parts that are versioned and coupled to the thing you are reverting. Everything else stays where it is. The specific failure in the question is the most common one: the image is content-addressed and precise, and the configuration is not versioned with it at all. ## Why moving-HEAD configuration breaks rollback If configuration is rendered from whatever the config repository currently holds, then the pair (code, config) that actually runs is a function of *when* you deploy, not of *what* you deployed. Two consequences follow: 1. **Rollback is incomplete.** Redeploying the old digest gives you old code against new configuration - a combination that may never have been tested and can be worse than either release. 2. **Rollback can be a no-op.** If the regression came from a configuration change (a timeout cut from 5s to 500ms, a connection pool halved, a traffic weight moved), the image is innocent and rolling it back changes nothing. Teams then conclude "the rollback didn't work" and start debugging the wrong layer, which is expensive precisely when it is most expensive. The same reasoning applies to configuration edited directly in a cloud console or an admin UI: there is no revision to return to, only somebody's memory of the previous value. ## Making the release one revertible unit The goal is that a release has a single identity, and that identity resolves to everything the release changed. Practically: - **Pin both halves in one record.** The deployable unit is a manifest that names the artifact digest *and* the exact configuration revision - not "latest config", a specific commit or version. Deploying release N means applying that record; rolling back means applying record N-1. - **Store it as versioned desired state.** If your production state is expressed as a repository or an equivalent versioned store, the rollback is a revert of one commit that moves both digest and config together, and the whole change is visible in one diff. - **Make config changes go through the same path as code.** A configuration change that skips the release process is a deploy with no rollback story. Reviewed, versioned, and rolled out the same way is what makes it reversible. - **Treat runtime-mutable levers deliberately.** Some things are *supposed* to change without a deploy. That decoupling is their purpose, but it also means the artifact rollback will not move them - they need their own change log and their own way back. ## The parts that still will not come back Even a perfectly coupled release record does not undo everything: - **Client-side assets** already downloaded keep running in users' browsers until their cache expires, so a rolled-back server must still tolerate the previous release's clients. - **Messages already published** in a new format sit in queues and topics; the rolled-back consumer has to be able to read or safely park them. - **Data written by the new version** stays written. The practical rule is that a rollback restores your deployment, not the world's memory of it. Design each release so the previous version can survive contact with whatever the new one left behind - that compatibility, not the deploy tooling, is what makes rollback safe. ## How to know it works This class of defect is invisible until you exercise it. Roll back a release deliberately in a non-emergency, and check that both the code version and the effective configuration values match the previous release. Teams that only try during an incident discover the coupling gap at the worst possible moment.