Across many teams' Helm releases, how do you stop chart changes from hitting immutable-field rejections?

### The shape of the problem at scale One team hitting an immutable-field refusal is an afternoon. A platform where thirty releases share a chart, and the chart changed, is an incident that arrives one release at a time — and each team hits it fresh, improvises, and some of them improvise with a flag that recreates objects in production. The leadership question is therefore not "how do I fix this upgrade" but "how does this class of failure stop being a surprise". ### Freeze what feeds a selector The single largest source of self-inflicted breakage is a workload's selector being rendered from something that was allowed to evolve. If the selector renders from the same helper as the full label set, then anything that touches that helper — adding a version label, renaming a component, adjusting a prefix — changes the selector and breaks every existing release of the chart. The policy is to treat the values that feed a selector as a published, frozen surface of the chart: a small stable set that nothing else is allowed to reach into, and a change to which is by definition a breaking chart release with a migration note, not a patch bump. The same discipline covers a subtler trap: generated names truncated to a fixed 63-character limit. A release whose name is 71 characters produces a truncated selector value, so a change that merely inserts a word earlier in the name changes what survives the cut. Any chart used with long release names should be reviewed with that in mind, and the fleet's longest release names are the ones to test against — not the short ones on your laptop. ### Keep un-updatable objects out of the upgrade path A Job whose pod template changes every release cannot live as an ordinary release object; it will be refused on the first upgrade that changes it. The organisational fix is a review rule rather than a flag: a Job rendered into `templates/` either gets a name that varies per release, or it is moved out of the ordinary lifecycle. That is worth encoding as a lint rule on the platform's own chart library, because it is mechanical and it catches the case before anybody meets the error. ### Gate the change, not the outage A server-side dry run puts the rendered objects through the API server's validation without committing them, and an immutable-field refusal shows up there. Making that a required check before a chart change lands — run against a cluster that actually holds a representative release, not an empty one — converts this failure class from a production incident into a review comment. It has real cost: the check needs cluster access from CI and a representative environment, and it is worth being honest that it only catches what the objects in that environment expose. For consumed charts, the equivalent gate is version discipline: pin the version, read the vendor's upgrade notes for the majors, and stage the bump on one low-stakes release with the dry run before it fans out. A third-party ingress-controller chart moving from 4.11.3 to 4.12.0 is exactly the change that should meet one release first. ### Decide who may recreate an object Someone will eventually face a refusal in production with traffic on the floor, and the available options differ enormously in cost: recreate the object in place and take a gap, remove the object while its pods keep serving and let the next upgrade adopt them, or cut over to a renamed object. That decision should be made in advance per class of workload, not improvised. A reasonable stance is that stateless workloads with a maintenance window may be replaced in place with a change record; anything holding data or long-running requests requires a written plan and a named approver; and the destructive flag never appears in an automated pipeline. Say plainly in the runbook that the flag with `force` in its name deletes and recreates, because the command line does not say so and the graphs will. ### Know what you are buying The honest tradeoff is that all of this slows chart changes down. Freezing selector inputs makes some cosmetic label improvements impossible without a migration; a dry-run gate needs a cluster CI can reach; staged rollouts of a vendor bump mean the fleet is briefly on two versions. Weigh that against the alternative, which is not "no cost" but "cost paid at random, by whichever team upgrades first, in production". For a small estate the informal answer is fine. The point at which it stops being fine is when one chart change can reject upgrades across many releases at once, and the platform team no longer knows which teams are about to hit it.