A Helm rollback restores the previous revision's manifests but not the database it migrated — how do you keep releases reversible?

### The claim to interrogate "We can always roll back, it's one Helm command." `helm rollback` re-applies a previous revision's stored manifests and records the result as a new revision — quick, reliable, and entirely about Kubernetes objects. If the release that is being undone also changed a database, the rollback restores the old pods and leaves them pointing at the new schema. Whether that combination works is decided by how the migration was written, weeks earlier, by someone who was not thinking about this moment. So the real design question is not "can we roll back" but "what is the compatibility contract between adjacent releases". ### The contract worth enforcing State it as a rule teams can check in review: **the previous release's code must run correctly against this release's schema.** If that holds, the rollback path is real. If it does not, the release is a one-way door whether or not anyone labelled it one. The technique that satisfies the rule is expand/contract. A change like renaming a column becomes a sequence: add the new column and write to both; backfill; switch reads; and only in a later release, once no running version reads the old column, drop it. Each individual release is reversible, because the version before it still works against the schema in front of it. ### What it costs, honestly Three releases instead of one. Application code that carries both shapes for a while, which is real complexity and real test surface. And a contract step that ships after the interesting work is done, which is exactly the kind of task that never gets prioritised — estates end up carrying columns nobody reads and dual-write paths nobody remembers. If you mandate expand/contract you have to fund the cleanup: an owner, a due date, and a way to see which contractions are outstanding. There is also a latency cost. A team that could have shipped a rename on Tuesday now ships it across three deploys. For a low-traffic internal service that may be poor value, and saying so is part of the judgement. ### Decoupling the migration from the release A Helm `pre-upgrade` hook binds the migration to the deploy, which is what gives you the ordering guarantee — the new code cannot start against the old schema. It also means every deploy is a schema event, and a migration failure is a deploy failure. The alternative is to run migrations as their own step: a separate release, a pipeline stage, or an operator, with the application deploy following once the schema is known good. That buys independent failure domains and lets a schema change be reviewed and timed on its own; it costs you the guarantee, because now nothing structurally prevents someone deploying the code first. Which model you pick can legitimately differ by service class — the shared platform chart the infrastructure team owns and the fast-moving product services need not answer this the same way. ### One-way doors Some changes cannot be made reversible at a sensible price: a destructive transformation, a data-format change with no inverse, an engine migration. The mistake is not having them. The mistake is having them unknowingly. Name them in advance. A one-way release gets a different safety story: a rehearsed restore with a measured restore time, a window, a switched-off traffic path, sometimes a shadow write and a comparison run before the cutover. It should also be visible operationally, so the person paged at 02:00 learns from the release itself that rolling back will not help, rather than from trying it. ### Rehearse the thing you claim to have A rollback path nobody has exercised is a belief, not a capability. The cheap version is a staging environment where you deploy N, then run `helm rollback` to N-1 and assert the old version serves correctly against the new schema. It is a small test and it catches the class of error — a `NOT NULL` column with no default, a dropped column still referenced by the previous image — that otherwise surfaces only in an incident. ### Forward fix versus rollback Mature teams often conclude that for stateful services the primary recovery is fix-forward, and rollback is the exception rather than the default. That is a defensible position, but only if it is a decision rather than a discovery. If fix-forward is the plan, invest where it pays: small releases, fast pipelines, feature flags that decouple activation from deploy, and enough observability to know within minutes that the last release is the problem. ### What to say in an interview The strongest answer distinguishes three layers. Helm restores manifests. The schema decides whether restored code runs. The organisation decides which changes are allowed to break that, and pays for the alternative. A candidate who only says "use expand/contract" has the technique; a candidate who also names its cleanup debt, its latency cost, and the one-way cases it does not cover has the judgement.