skip to content

Why must an already-published Helm chart version never be overwritten with new content?

level: seniorimportance: should knowfreq 47%

answer

  1. The tool does not enforce this
  2. Coordinates are the only identity downstream
  3. Two consumers, two caches, one number
  4. Locks pin the number, not the bytes
  5. Bump instead; versions are cheap

basics

~20 s

Nothing in Helm enforces it, but every consumer treats chart name plus version as an identity: cached indexes, pinned dependencies and stored release records all resolve by version. Overwriting makes one coordinate mean two different things. Bump the version instead.

solid answer

~50 s

Helm will not stop you - a classic repository is files, and overwriting a published `.tgz` is a file copy. The problem is that name plus version is the only identity anything downstream has. A consumer holding a cached `index.yaml` from before the overwrite and one who refreshes afterwards both install "2.7.3" and get different bytes; the `digest` recorded in the older cached entry no longer describes the file at that URL. A parent chart's lock pins name, repository and version, not content, so the same lock resolves differently over time. And the release record stores what was actually applied, so the cluster and the repository can disagree about what 2.7.3 was. The remedy is boring and absolute: chart versions are cheap, so bump the version for anything - even a typo in NOTES.txt - and make the publish job refuse to overwrite an existing version.

code

bash · 9 lines
bash
REPO=https://charts.internal.example/charts
VER=$(helm show chart ./digest-builder | awk '/^version:/{print $2}')

if curl -fsSL "$REPO/index.yaml" | grep -q "digest-builder-${VER}.tgz"; then
  echo "digest-builder ${VER} is already published; bump Chart.yaml" >&2
  exit 1
fi

helm package ./digest-builder

go deeper

for a junior

Learn the rule before the reasoning: once a chart version has been published, it never changes. Any fix, however small, ships as a new version number. Overwriting is the mistake, not the shortcut.

for a middle

Explain why the rule exists mechanically - clients resolve from a cached index by name and version, dependency locks pin coordinates rather than content, and the recorded digest stops matching the file. Be able to say that Helm itself enforces none of this.

for a senior

Show the incident shape: two environments on 'the same version' behaving differently, and the move to the stored release records to find out what actually shipped. Then describe the guard you would add to the publish path and the correct withdrawal procedure for a bad version.

for a principal

Own it as a contract with consumers. Decide where enforcement lives - pipeline check, artefact-store setting, or both - what your policy is when a version must be withdrawn, and how you communicate that, given removal never reaches copies already vendored or stored inside releases.

Immutability of published versions is a convention, not a mechanism. It is worth understanding both halves of that sentence: why Helm cannot enforce it, and why everything downstream falls apart when you break it. ## Why nothing stops you A classic chart repository is a directory of tarballs plus a generated `index.yaml` on a static host. Republishing `digest-builder-2.7.3.tgz` with different templates is `cp` over a file. Regenerating the index updates that entry's `digest` and `created`, and the catalogue now describes the new bytes as though they had always been 2.7.3. Pushing the same chart version to a registry as a tag is the same story unless the hosting platform has been configured to reject overwrites - a capability of the artefact store, not of Helm. ## What breaks, concretely **Split-brain by cache.** Clients resolve charts out of a locally cached copy of `index.yaml`, refreshed only by `helm repo update`. After an overwrite you have two populations: those whose cache predates it and those whose cache follows it. Both install `--version 2.7.3`. Their cached entries disagree on the digest, and only one of them describes the file now sitting at the URL. Nobody sees an error; they see two clusters running "the same version" with different behaviour. **Dependency resolution stops being reproducible.** A parent chart declares a subchart by name, version and repository, and the lockfile pins those coordinates - not the bytes of the tarball. Resolving that lock a month apart fetches whatever is published under that number today. A build that was green becomes red, or worse, stays green and ships something different. **Diagnosis loses its anchor.** Suppose the email-digest builder is misbehaving in one environment and fine in another, and both report chart 2.7.3 - one renders a HorizontalPodAutoscaler with the old minimum replica count and the other does not. The instinct is to fetch 2.7.3 and read it, but the chart you fetch is only the *current* content of that coordinate. The trustworthy source is the release record Helm keeps in the namespace, which stores the manifest that was actually applied along with the chart that produced it. For this chart the stored record runs about 1.1 MiB, and `helm get manifest` against each release is what settles the argument. Having to reach for that is itself the smell: the repository has stopped being able to tell you what 2.7.3 was. **Rollback and audit go soft.** "Revision 4 installed chart 2.7.3" is a useful statement only while 2.7.3 means one thing. After an overwrite, history records a coordinate rather than a content, and a rollback that reinstalls from a re-fetched chart is not necessarily a return to what was running. ## Do this instead Bump the version. Chart versions are free and they are not marketing: a corrected description, a fixed indentation bug, a changed default - all of them are a new patch version. This is also the cleanest use of the two version fields: the chart's `version` moves whenever the packaging changes, while `appVersion` tracks the application image and may sit still for several chart releases in a row. Make the pipeline enforce it. Before uploading, check whether the version already appears in the published catalogue and fail the job if it does. This turns "we agreed not to overwrite" into a property of the system, and it costs about four lines. If the artefact platform you publish to can be configured to reject writes to an existing version, turn that on as well - defence in depth against the one job someone runs by hand. ## Withdrawal, when you must Occasionally a published version has to go: it leaks a credential, or it is so broken that leaving it discoverable is worse than the disruption. The procedure is not a republish. Rotate anything exposed first - the tarball has already been downloaded into caches, vendored into parent charts and stored inside release records you do not control. Then remove the entry from `index.yaml` and delete the tarball (or delete the tag in the registry), publish the fix under a *new* number, and tell consumers. Removal is not retroactive: clients with a cached index still list the version and will simply fail to download it, which is the correct failure. What you must never do is put corrected content back under the old number, because that hands the split-brain problem to everyone who was doing the right thing by pinning.

  • Two clusters report the same chart version for a release but render different manifests. How do you prove what each one actually got?
    Stop asking the repository and ask the clusters. Helm stores a record per release revision holding the manifest it applied and the chart it came from, so `helm get manifest` on each release shows the real content side by side. If those differ while the chart coordinates match, the version was overwritten between the two installs - the repository can no longer answer the question, and the release records are the only remaining evidence.
  • A published chart version embeds a credential in its default values. What is the withdrawal procedure?
    Rotate the credential first; the tarball is already in caches, vendored subchart directories and stored release records you cannot reach. Then delete the tarball, remove its entry from the index, publish a fixed chart under a new version, and notify consumers. Do not republish the old number with cleaned content - anyone holding a cached index would silently receive different bytes for a version they pinned deliberately.
  • Is bumping appVersion enough when only the chart's templates changed?
    No. `appVersion` describes the application being deployed; the chart's own `version` is the coordinate that consumers pin, that the index keys on, and that the packaged filename carries. A template change with an unchanged chart version is exactly the overwrite you are trying to avoid, whatever you do to appVersion.

saying these in an interview costs you the question

  • Claims Helm refuses to overwrite a published version
  • Re-uploads the same version after fixing a typo
  • Thinks a lockfile pins the tarball's actual bytes
  • Assumes deleting an entry recalls downloaded copies
  • Bumps appVersion instead of the chart version
  • Believes stale client indexes refresh themselves

context

resolving links…