Why does helm upgrade still fail on a removed apiVersion after the chart was fixed to emit the supported one?
answer
- An upgrade is a diff, not a fresh apply
- Helm reads the previous revision too
- Frozen text; nothing rewrites it
- The message says current release manifest
- Rollback reads stored manifests as well
basics
~20 sBecause Helm must decode the previous release's stored manifest on every upgrade, and that text is frozen at install time. Fixing the chart changes what you render, not the retired apiVersion already recorded with the live release.
solid answer
~50 sAn upgrade is a **diff**, not a fresh apply. Before Helm can decide what to create, patch or delete it reads the manifest stored with the current revision and builds objects from it — that is how it knows which resources have disappeared from the chart, and on the older client-side path it is also the basis of the merge patch. That stored text is literal and immutable; nothing rewrites it when a cluster stops serving a version. So a release installed a year ago still carries `networking.k8s.io/v1beta1` in its record, the mapper fails on it, and the upgrade aborts before your freshly-rendered manifest is ever considered. Helm's wording points at the *current release manifest* rather than the new one. `helm rollback` fails identically, because it also builds objects from stored manifests. The fix is to repair the stored record, not the chart alone.
code
bash · 6 lines# the chart is clean, the release is not
helm template ./platform -n payments | grep -c 'networking.k8s.io/v1beta1' # 0
helm get manifest fraud-scoring -n payments | grep -c 'networking.k8s.io/v1beta1' # 1
# and rollback reads stored manifests too, so it fails the same way
helm history fraud-scoring -n paymentsgo deeper
Remember that an upgrade compares the new render against the manifest stored with the last revision, so fixing the chart cannot fix text that was already written. Know that helm get manifest shows you that stored copy.
Explain why Helm needs the old manifest at all — deletion detection, and on the client-side path the merge patch — and why nothing in the system rewrites it. Be ready to say that Helm 4's server-side apply default does not remove the requirement.
Demonstrate the triage: compare the two manifests, recognise that rollback is not an escape, and pair the chart bump with a record repair so the next upgrade does not reintroduce the retired version.
Frame the latency of the failure as the real risk — releases nobody upgrades look healthy until an emergency deploy, so the organisational answer is auditing stored manifests before the API disappears, not diagnosing them afterwards.
### An upgrade reads two manifests, not one The instinct that a `helm upgrade` simply renders the chart and sends it to the cluster is what makes this failure so confusing. It does more than that. Helm keeps a record of every release revision, and that record contains the **rendered manifest** exactly as it was at the time — a plain block of YAML text. On an upgrade, Helm loads the record for the current revision, decodes its manifest into Kubernetes objects, and compares that set against the set it just rendered from the new chart. It needs that comparison for concrete reasons. Resources present in the old manifest and absent from the new one must be deleted — that is how removing a template from a chart removes the object from the cluster. And on Helm 3's client-side three-way strategic merge, the old manifest is one of the three inputs to the patch, alongside the new manifest and the live object. Helm 4 makes server-side apply the default write path (`--server-side` is a bool defaulting to `true` on install, and a string defaulting to `"auto"` on upgrade, which inherits whatever the previous release used — so a release first installed by Helm 3 stays on the client-side path). Either way, Helm still has to turn the previous manifest into objects to know what disappeared. Helm 4 did not remove this failure. ### Why the stored text never updates itself Building objects means resolving each `apiVersion`/`kind` pair against the cluster's discovery data, and that is where the removed version bites. The stored manifest is a snapshot of characters. No component in the system goes back and rewrites it: - the API server migrates *stored objects* to a version it still serves, but has no idea Helm's record exists; - `helm repo update` refreshes chart indexes, not release records; - a newer chart version changes what the **next** render emits, not what the last one wrote; - the live `Ingress` remains perfectly healthy and readable under the supported version, which is exactly why nobody notices until the next upgrade. So you can bump the chart, review a beautiful diff in which `networking.k8s.io/v1beta1` is nowhere to be seen, run the upgrade, and get the same error you got before the fix. The failure is upstream of your change. ### Reading the message correctly Helm's phrasing distinguishes the two sides — a failure building Kubernetes objects from the **current release manifest** is the stored copy, while a problem with what you rendered surfaces against the new manifest. That single word is the fastest triage signal on this error. Confirm it directly: ``` helm get manifest fraud-scoring -n payments | grep -c 'networking.k8s.io/v1beta1' helm template ./platform -n payments | grep -c 'networking.k8s.io/v1beta1' ``` A non-zero count on the first and a zero on the second is the exact shape of "the chart is fixed and the release is not". ### The consequences people miss **Rollback is not an escape hatch.** `helm rollback` also builds objects from stored manifests — both the revision it is leaving and the one it is targeting — so if revision 47 is unusable, rolling back to revision 44 usually fails too, and revision 44 is *more* likely to carry the retired version, not less. **The failure is latent.** A release that nobody upgrades sits there looking green for months. The error surfaces the first time someone runs an upgrade, which in practice is often during an incident, when they are trying to ship a fix to the fraud-scoring endpoint and discover that the deploy path itself is broken. That is the argument for auditing stored manifests before a cluster loses an API rather than after. **A chart bump is still necessary — just not sufficient.** If you repair the stored record but leave the chart rendering the retired version, the next upgrade writes the bad text straight back in. The two fixes are paired: bump the chart so future renders are correct, and repair the record so the next upgrade can read the past. **History has a shelf life.** Because Helm keeps a bounded number of revisions (`--history-max` defaults to 10), a release that is upgraded regularly eventually scrolls the offending revisions out of its history on its own. That is a reason to prefer upgrading affected releases early, while the retired version still resolves and the upgrade can actually run.
- Can you roll back to an earlier revision to get out of this?No. A rollback builds Kubernetes objects from stored manifests as well — both the current revision and the one you are targeting — so it hits the same mapping failure. Older revisions are usually worse, since they predate any chart fix and are more likely to carry the retired version. Rollback is not a repair path for this class of failure.
- Does helm template reproduce the error, and if not, why not?It normally does not. helm template renders locally and does not resolve kinds against a cluster unless you ask it to validate against a live API server, so it happily prints whatever apiVersion the chart emits. That is precisely why the chart looks fixed while the upgrade keeps failing — the two commands read different manifests.
- If you repair the stored record but leave the chart alone, what happens on the next upgrade?It writes the retired apiVersion straight back into the new revision's stored manifest, and the release is broken again the moment that revision becomes the one Helm has to read. The chart bump and the record repair are a pair: the bump fixes future renders, the repair fixes the past one that upgrades still depend on.
saying these in an interview costs you the question
- Thinks upgrading the chart alone clears the error
- Believes the API server migrates Helm's stored manifest
- Suggests helm rollback as the way out
- Says server-side apply in Helm 4 removed this failure
- Confuses the live object's version with the recorded text
- Assumes helm template failing or passing settles it