Would you make --rollback-on-failure mandatory for every production helm upgrade in CI?

## What the mandate is actually proposing "Always pass `--rollback-on-failure`" is a proposal to make one recovery strategy — revert — the automatic response to every deploy failure, chosen by a flag rather than by a person. Whether that is right depends on a property of each chart that the flag itself cannot inspect: *is the previous revision still a correct thing to be running after this attempt partially happened?* ## The case for making it the default - **Bounded exposure.** Without it, a failed upgrade leaves the release in a mixed state for however long it takes a human to notice and decide. With it, the window is the timeout. - **Uniform end state.** Every failed deploy ends in the same place: the last known-good manifest. That is enormously easier to write runbooks against than "it depends what failed". - **It scales with fan-out.** A multi-tenant chart installed once per team namespace, upgraded across dozens of namespaces in one run, cannot have a human triage each failure. Self-restoring failures plus exit codes is the only shape that works. - **It forces waiting.** Because the flag defaults the wait strategy to watcher, adopting it also ends the pattern of pipelines reporting success before the rollout has converged. Some teams want the flag mostly for that. ## The case against making it universal **Irreversibility.** This is the one that actually decides it. Rolling back restores *manifests*, not *effects*. If a pre-upgrade hook has already migrated a database schema, the automatic restore puts the old application code back in front of a new schema — a state nobody tested, and often worse than the failed upgrade it replaced. The same applies to a Job that consumed a queue, an external system that was called, or data already written in the new format. For those charts, forward-fix is the only defensible policy, and the flag encodes the wrong one by default. **Timeout as verdict.** With the flag on, the timeout stops being a patience setting and becomes the definition of failure. Size it from typical rollouts and you will roll back healthy releases whose images were slow to pull or whose startup probes are long; size it very generously and the pipeline slot is held for that long on every failure. Neither is free, and a single organisation-wide number will be wrong for both the fastest and the slowest workload. **Evidence destruction.** On a release that does not exist yet the flag uninstalls rather than restores, taking the failed pods and their logs with it. Teams bringing up new services hit this repeatedly, conclude the tooling is hostile, and start disabling the flag everywhere — including where it was doing good. **Failure during recovery.** The restore is itself an apply that can fail, and a failed automatic rollback leaves a release in a worse and less-expected state than the failure that triggered it. Rare, but it is the scenario where the mandate has made things harder rather than easier. ## How I would actually write the policy 1. **Classify charts, not environments.** The question is "is reverting safe for this thing?", which is a property of the chart's hooks and its data model. Two tiers is usually enough: *revertible* (stateless services, config-only releases, anything whose previous revision is unconditionally safe) and *forward-only* (anything with a state-mutating hook or a schema contract). Default the first tier on, the second off, and require a reason to move a chart between tiers. 2. **Make the timeout a per-chart value with a floor.** Derive it from observed healthy rollout durations with headroom, not from a shared default. A chart whose pods take minutes to warm caches needs its own number. 3. **Separate first-install from upgrade.** New releases install without the flag, with an explicit diagnostic capture and cleanup step. Established releases upgrade with it. 4. **Keep the human path.** Automatic rollback is a first response, not an incident process. The pipeline must still fail loudly, page, and leave the failed revision visible in history for the follow-up. 5. **Watch the metric that matters.** Count automatic rollbacks per chart. A chart that rolls back regularly is not being protected by the flag, it is being masked by it — the flag is absorbing a quality problem that should be fixed upstream in tests or in the rollout strategy. ## The honest summary Automatic rollback is a good default for the majority of charts and a genuinely bad one for the minority that change shared state. A mandate with no exemption path optimises for the common case and quietly creates the worst incidents; a mandate with a documented exemption path, per-chart timeouts, and a separate rule for first installs is close to right. What I would not accept is "always on" being treated as a substitute for progressive delivery — the flag reverts a bad release *after* it has been fully applied, which is a strictly weaker guarantee than never sending it all the traffic in the first place.