A `helm upgrade` fails with `field is immutable` — what happened, and what state is the release left in?
answer
- The refusal comes from the cluster, not Helm
- Some object fields are write-once
- Selector, Job pod template, volumeClaimTemplates
- A failed upgrade is still a stored revision
- Patching cannot help; replacing can
basics
~20 sThe Kubernetes API server rejected the write because the chart's new manifest changes a field that cannot change after the object was created — a Deployment's spec.selector is the classic one. Helm records the attempt as a failed revision and the live object keeps its old spec.
solid answer
~40 sThat message comes from the API server, not from Helm. `helm upgrade` renders the chart, sorts the objects by kind and sends them one at a time; when it reaches an object whose new spec changes a write-once field, the server refuses it and the upgrade stops. The three you actually meet are a workload's `spec.selector`, a Job's pod template, and a StatefulSet's `volumeClaimTemplates`. The release is now partly applied: objects Helm wrote before the rejected one are live, the rejected object is untouched, and Helm has stored a **new revision with status `failed`** while the previous revision is still the deployed one. Nothing is undone unless the upgrade ran with `--rollback-on-failure`. Getting out means changing the chart so the field stays put, or accepting that the object has to be deleted and recreated.
go deeper
Be ready to say plainly that the cluster refused the write, name one field it happens on, and state that the old object is still running. Knowing that Helm stored a failed revision and changed nothing on the rejected object is the whole answer at this level.
Explain the mechanics: Helm renders, sorts by kind, applies one object at a time and stops at the first hard failure, so the release ends up partly applied with a failed revision on top of the last deployed one. Name the three fields this bites on.
An interviewer expects you to move from the message to a plan — revert the field, recreate the object, or rename it — and to say what each costs in availability. Mention checking the partial state of the release before you decide, rather than re-running the command.
Own the prevention angle: which chart changes are allowed to touch a selector, whether a server-side dry run is a required check before a chart bump reaches a cluster, and who is allowed to authorise an object recreation in production.
### The refusal is the cluster's, not Helm's Helm has no notion of which fields may change. `helm upgrade` renders the chart, sorts the rendered objects into Helm's fixed kind order, and writes each one to the Kubernetes API server. The server validates every update, and for a set of fields it accepts only the value written when the object was created. The refusal text — `field is immutable` — is the server's, and Helm surfaces it under a failed upgrade. That is why no combination of Helm flags makes an ordinary update of such a field succeed: the write is a legal Helm operation and an illegal Kubernetes one. Anything Helm can do about it is about replacing the object, not about persuading the server. ### The three you will actually meet **A workload's `spec.selector`.** The labels a Deployment or StatefulSet selects on are fixed at creation. In a chart this is rarely somebody hand-editing a selector; it is a helper template feeding it. A third-party ingress-controller chart that moves from 4.11.3 to 4.12.0 may rewrite the label helper the selector renders from, and then every release of that chart is rejected on upgrade. There is a subtler variant worth knowing: charts commonly truncate a generated name to 63 characters, so a release whose name is 71 characters long renders a selector value that is a cut-off string. Change anything that lands before the cut — a prefix, an added component word — and the truncated value changes even though nobody touched the selector block. **A Job's pod template.** A Job that sits in `templates/` without hook annotations is an ordinary release object, so Helm tries to update it on every upgrade. Its pod template cannot change, so the first upgrade that bumps that Job's image is rejected. Charts avoid this by giving the Job a name that changes per release or by moving it out of the ordinary lifecycle entirely. **A StatefulSet's `volumeClaimTemplates`.** Growing a claim, adding a second volume, switching a storage class: all rejected on an existing StatefulSet. ### What state the release is in afterwards Helm applies in kind order and stops at the first hard failure, so the release is partly applied — everything ordered before the rejected object is already live in the cluster. Helm still writes a revision record: `helm history` shows a new revision with status `failed`, and the revision below it, the last successful one, is the one marked deployed. `helm status` reports the release as failed. Helm does not undo the objects it already wrote unless the upgrade was run with `--rollback-on-failure`; without it you are left with a half-new release and a failed revision on top. That matters for the next attempt. Helm computes the next upgrade against the manifest stored with the last deployed revision, which still describes the old shape — so a second run of the same command reproduces the same rejection, usually after re-applying the same partial set. Retrying is not a strategy. ### Getting out There are three honest ways forward. Revert the field so the chart renders the value the object already has — often the right answer when a chart bump changed a selector you never wanted changed, and you can pin the values that feed it. Recreate the object, which is what `helm upgrade --force-replace` does: Helm stops patching and removes and re-creates, so the new value lands and everything the object manages goes down and comes back. Or give the object a new name, so the chart renders a fresh object and the old one is removed as part of the same upgrade — a controlled cutover rather than a delete-and-recreate in place. ### What not to do Do not reach for `--force-conflicts`. It is a different Helm 4 flag that settles field-manager ownership under server-side apply, and immutability is not an ownership question — the server refuses the value no matter who is asking. Do not edit or delete the release record Secret to make the failure go away; the object in the cluster is what is refusing the change, and the record is only Helm's memory of it. And do not conclude from a failed upgrade that nothing was applied. ### Seeing it before it bites A server-side dry run (`helm upgrade --dry-run=server`) sends the rendered objects to the API server for validation without committing them, which is where an immutable-field rejection is cheapest to discover. Rendering the new manifest and diffing it against the manifest stored with the current revision is the other habit: if a selector block or a Job pod template moved, you know before you touch the cluster.
- Does Helm undo the objects it already applied before the rejection?No. Helm writes in kind order and stops at the failure, leaving the earlier objects live. It only reverts automatically if the upgrade ran with `--rollback-on-failure` (`--atomic` is the deprecated alias for that flag). Otherwise you are looking at a partly-applied release, and returning to a known state is a deliberate rollback you run yourself.
- Which revision does `helm history` show as deployed once the upgrade has failed?The previous one. Helm records the attempt as a new revision with status `failed` and leaves the last successful revision marked deployed, so the history shows a failed revision sitting on top of a deployed one. The next upgrade is computed from the deployed revision's stored manifest, which is why simply re-running the same command reproduces the same rejection.
- How would you have caught this before running the upgrade?Run the upgrade as a server-side dry run so the API server validates the rendered objects without committing them; immutable-field rejections surface there. Alongside that, render the new chart version and diff it against the manifest stored with the current revision — a moved selector block or a changed Job pod template is visible in the diff, and that check is cheap enough to sit in CI on every chart bump.
saying these in an interview costs you the question
- Says Helm itself refuses the change rather than the API server
- Assumes a failed upgrade rolls itself back automatically
- Thinks --force-conflicts clears an immutable-field rejection
- Believes nothing was applied because the upgrade failed
- Says deleting the release record Secret repairs the object
- Expects retrying the identical upgrade to eventually succeed