Would you set --rollback-on-failure on every helm upgrade your CI runs, across a dozen service charts?

This is a policy question wearing a flag's clothes, and the interesting part is that the obvious answer — "always roll back, it protects production" — trades away the thing that stops the same outage recurring. ## What the flag does `--rollback-on-failure` tells Helm that when an upgrade fails it should revert the release to the previous revision as part of the same operation. In Helm 4 this is the flag's real name; `--atomic` survives as a deprecated alias, so a fleet's scripts written against Helm 3 keep working but should be renamed. Setting it also defaults `--wait` to the `watcher` strategy — a consequence worth stating out loud, because it changes behaviour for every chart that previously ran without waiting: those upgrades now observe readiness, can now fail on a slow rollout, and will now revert. Turning the flag on fleet-wide is not one change, it is two. ## The case for it For a stateless service, a failed upgrade that leaves half-applied objects or a crash-looping new version is a live incident, and reverting in the same operation shortens it to the length of one rollout. It also keeps the release record honest: rather than accumulating failed revisions that nobody cleans up, the release returns to a known-good, deployed state that the next upgrade can build on. ## The case against it Three arguments, in rising order of severity. 1. **Evidence.** Triage of a failed upgrade lives in objects that the revert destroys: the pods that would not become ready, their events, the Job a failed hook left behind. If your pipeline reverts before anyone looks, every failure report reads "the deploy failed and then it was fine", and a flaky, intermittent failure never gets diagnosed. The release history preserves the failed revision's manifest and description, which is genuinely useful, but a manifest is not a reason. 2. **Side effects.** Reverting the release reverts the Kubernetes manifests and nothing else. If a pre-upgrade hook ran a schema migration and a later step failed, the automatic rollback puts the *old* application code back in front of *new* schema. That combination can be worse than a failed upgrade that simply never replaced the running pods, and it is a per-chart property: a chart with an irreversible pre-upgrade hook wants human judgement, not an automatic revert. 3. **The previous revision must actually be deployable.** Rollback re-applies the stored manifest of the prior revision, which references images and configuration that must still exist. A fleet that garbage-collects old image tags aggressively has a rollback path that quietly stops working, and the discovery moment is an incident. ## A defensible position Decide per chart, and say so with reasons rather than reaching for uniformity. | Chart | Policy | Why | | --- | --- | --- | | Stateless services whose rollout time you have measured | Enable it | Set `--timeout` per chart from observed rollout duration with headroom, never from a shared default | | Charts with migrating or otherwise irreversible hooks | Leave it off, fail loudly, and page | The correct next action is a decision, not a revert | | A shared library chart consumed by twelve service charts | Consistency of *mechanism* rather than of setting | Express the timeout and failure policy as values the library chart's deployment tooling reads, so each service declares its own budget in one reviewed place instead of twelve pipeline files drifting apart | A budget below the honest rollout time plus automatic rollback produces a service that can never deploy — for an email-digest builder whose replicas drain a queue before reporting ready, a seven-minute rollout under a five-minute default budget is exactly that trap. ## Capture before you revert The strongest version of the answer refuses the binary. Because Helm performs the revert inside the same operation, a pipeline cannot interleave collection with it. So for services that need fast recovery *and* diagnosable failures, do not use the flag: run the upgrade without it, and on failure have the pipeline collect `helm status <release>`, `helm get hooks <release>`, the workload's pods, logs and events into the build's artefacts, and only then run the revert itself as an explicit step. That is a few more lines of pipeline in exchange for every failure being explainable, and it makes the recovery an auditable action rather than an invisible one. ## What to measure Whatever you choose, the fleet-level signals are the same: how often upgrades fail, how many of those failures were deadline expiries rather than genuine rejections, and how many were reverted without anyone reading a diagnosis. A high count in the last bucket means your automation is converting outages into mysteries, and that is the argument for changing the policy — not a preference about flags.