skip to content

You fix a bug in a shared Helm library chart. What has to happen before the releases that depend on it run the fix?

level: seniorimportance: should knowfreq 33%

answer

  1. Nothing in the cluster changes by itself
  2. The link happens before packaging
  3. The lock pins what a rebuild restores
  4. One rollout per consuming chart
  5. Emit the library version as a label

basics

~20 s

Every consumer must re-resolve its dependencies, repackage and upgrade its own release. A library chart is linked in before packaging, not fetched at install time, so publishing a new version changes nothing in a cluster on its own.

solid answer

~50 s

Publishing the library is step one of many. The library is resolved into the consumer when that chart's dependencies are updated - the resolved version is recorded in `Chart.lock` and the packaged library is vendored under `charts/` - and it then travels inside the consumer's own tarball. So the fix reaches a namespace only after the consumer re-resolves, bumps its own chart version, republishes, and someone runs an upgrade of that release. A version range in `dependencies:` does not make this automatic: rebuilding from the lock reproduces the pinned version, and nothing re-reads the range at install time. Across a 12-namespace fleet that is N pull requests and N upgrades. Treat it as a rollout: automate the bump, and emit the library version as a label so you can find releases still running the old helper.

code

bash · 3 lines
bash
helm dependency update ./billing-cron
helm package ./billing-cron
helm upgrade --install billing-cron ./billing-cron-1.9.3.tgz -n billing

go deeper

for a junior

The point to remember is that a library chart is copied into the consumer when its dependencies are resolved, so publishing a new library version does not change anything already running.

for a middle

Explain the mechanics: the lock records the resolved version, the packaged library is vendored under charts, and the consumer's tarball carries it. Distinguish re-resolving from rebuilding from the lock.

for a senior

Show the rollout thinking - N pull requests, render diffs as the review artefact, the consuming chart's own version bump, and a way to see from the cluster which release still runs the old helper.

for a principal

Own the policy: pinning versus ranges, who is on the hook for driving adoption, and what an urgent fix looks like when twelve teams must each ship before the fleet is consistent.

### Where the link actually happens The single fact that answers this question: a library chart is linked into its consumer *before* the consumer is packaged, not fetched when the consumer is installed. The consumer declares the library in its `dependencies:` block with a version or a range. Resolving those dependencies writes the chosen version into `Chart.lock` and places the packaged library under the consumer's `charts/` directory. When the consumer is packaged, that vendored copy goes inside the tarball. At install or upgrade time Helm renders what is in the tarball; it contacts no repository to see whether a newer library exists. That has an immediate consequence people get wrong: publishing `platform-common` 2.4.4 changes nothing anywhere. Not one release differs. The library is only a source of template text, and the text every running release renders from is the copy captured when its chart was built. ### The full path for one consumer For a single consumer - say the single-service chart with an Ingress and a HorizontalPodAutoscaler - the fix reaches production through: re-resolve dependencies so the lock and `charts/` pick up 2.4.4; bump the consumer's own chart version, because the rendered output has changed and the chart that produced it must be distinguishable; package and publish the consumer; and upgrade the release in each namespace where it runs. Skip the consumer's own version bump and you have two different artefacts claiming to be the same chart version, which destroys your ability to answer "what is running here". ### Why a version range does not save you A range such as `~2.4.0` is a statement about what is acceptable at resolution time, not a subscription. Two commands behave differently and the difference is the whole answer: re-resolving the dependency consults the repository and picks the newest version matching the range, rewriting the lock; rebuilding from the lock deliberately restores exactly the versions the lock pins, which is what CI should do so a build is reproducible. If your pipeline only ever rebuilds from the lock - and it should - then nothing picks up 2.4.4 until a human or a bot re-resolves and commits the new lock. This is a feature. It means a bad library release cannot roll itself out across a fleet, and each consumer's upgrade is a reviewable diff. It is also the cost: the fix is an N-chart rollout. ### Operating the rollout On a fleet of any size, three things make this bearable. First, automate the bump. A job that re-resolves each consumer, renders it before and after, and opens a pull request with the render diff attached turns twelve manual edits into twelve reviews of an actual behaviour change. The render diff is the review artefact that matters, because a library bump's blast radius is exactly the change in emitted YAML. Second, make the deployed version observable. Nothing in a cluster tells you which library version produced an object unless you arranged for it, so have the shared helper emit the library version as a label or annotation on everything it stamps. Then finding the stragglers is a cluster query rather than an audit of twelve repositories. Third, decide a pinning policy and hold it. Exact pins plus a bot give you reproducibility and a controlled rollout. Loose ranges only matter at the moment of re-resolution, and they mostly buy surprise: two consumers re-resolved a week apart end up on different library versions with no record of anyone deciding that. ### When the fix is urgent If the bug is a bad label or a wrong annotation on a security-relevant object, you cannot ship one central change and be done. Sequence it: fix the library, roll the highest-risk consumer first and verify the rendered diff in one namespace, then fan out. If a consumer cannot be rebuilt quickly, the fallback is to change that chart's own template rather than to wait for the library - the library is a convenience, not a control plane, and no part of the release path is blocked on it. ### One thing that is not affected Rolling back a consumer's release rolls back to a previous revision's stored manifest, so it undoes the library bump along with everything else in that revision - there is no separate library version to roll back, and no way to revert only the helper change while keeping the consumer's other edits. That is another reason to bump a library on its own commit and its own consumer chart version: it keeps the unit of change and the unit of rollback aligned.

  • Why does rebuilding a consumer's dependencies not pick up a newer library version?
    Because a rebuild restores the versions recorded in `Chart.lock` rather than consulting the repository - that is what makes a build reproducible. Re-resolving is the command that re-reads the range, chooses the newest match and rewrites the lock. CI should rebuild; a bump is a deliberate, committed change.
  • How would you find which releases still run the old library version?
    Have the shared helper stamp the library version as a label or annotation on every object it produces, then query the cluster for the old value. Without that, you are reduced to correlating each release's chart version against the lock file in each consumer's repository, which does not scale past a handful of charts.
  • Does the consuming chart need its own version bumped when only the library changed?
    Yes. The rendered output changed, so the artefact changed, and two tarballs with the same chart version producing different YAML is a reproducibility bug. Bump the consumer's chart version on every library bump, even when its own templates were untouched.

saying these in an interview costs you the question

  • Thinks consumers pick up the new version automatically
  • Believes Helm fetches dependencies at install time
  • Says a version range makes the upgrade happen
  • Leaves the consuming chart's version unchanged
  • Assumes one library release equals one fleet rollout

context

resolving links…