skip to content

A Helm post-renderer patches a chart's StatefulSet; a colleague's helm upgrade without the flag drops the patch. Why, and how do you prevent it?

level: seniorimportance: should knowfreq 38%

answer

  1. Values are remembered, flags are not
  2. The next upgrade renders the chart plainly
  3. Absent from desired state means removed
  4. The stored manifest still holds the patch
  5. Make the flag impossible to omit

basics

~20 s

Helm stores a release's values but not the fact that a post-renderer produced its manifest. An upgrade without the flag renders the chart plainly, so the patched fields are absent from the new desired state and Helm removes them. The fix is to make the flag impossible to omit.

solid answer

~50 s

Helm records the merged values with each revision, so `--reuse-values` can recover them; it records **nothing** about `--post-renderer`, which is a per-invocation flag. An upgrade that omits it renders the chart plainly, and the patched fields are simply missing from the new desired manifest — so Helm removes them from the live object, whether through server-side apply dropping fields its manager no longer declares or through the older client-side three-way merge diffing against the stored manifest. Nothing errors; the release goes to `deployed` and the patch is gone. Note that `helm get manifest` on the earlier revision still shows the patched YAML, because Helm stores the post-renderer's output, which is why a `helm rollback` to that revision restores the patch — it replays the stored manifest rather than re-rendering. The prevention is structural: one wrapper or pipeline step that always passes the flag, the plugin pinned in the deploy image, and a check that fails if the patch is absent from the rendered output.

code

bash · 9 lines
bash
# Which of the 12 tenant namespaces lost the patched field?
for ns in $(kubectl get ns -o name | cut -d/ -f2 | grep '^chat-'); do
  if ! helm get manifest chat-redis -n "$ns" | grep -q 'platform.example/tier'; then
    echo "MISSING in $ns"
  fi
done

# Bring one namespace back to the last post-rendered revision
helm rollback chat-redis 47 -n chat-eu-3

go deeper

for a junior

The takeaway is that --post-renderer is not remembered by the release. If a deploy needs it, every deploy needs it, so run deploys through the pipeline rather than typing helm by hand.

for a middle

Explain why the field is removed rather than merely left alone: Helm compares the previous declared manifest with the new one, and a field that is no longer declared is taken away.

for a senior

Show the whole loop — detect it with helm get manifest across the fleet, recover with a rollback to the last post-rendered revision, and prevent it with a single deploy entry point plus a rendered assertion in CI.

for a principal

Own the systemic point: deploy behaviour that lives in a command-line flag instead of in stored release state is unrecoverable and unauditable. Decide where such transformations are permitted at all and what must gate them.

This is the failure mode that makes post-renderers dangerous in the hands of a team rather than an individual, and it is worth walking through concretely. ## The scenario A chat fan-out service depends on a Redis chart you do not own, whose StatefulSet and PersistentVolumeClaim template do not expose the field your platform requires. Rather than fork the chart, you write a post-renderer that patches the rendered StatefulSet, and you roll it across a 12-namespace fleet — one release per tenant namespace. Everything is correct for months, up to revision 47 in the busiest namespace. Then an engineer needs a quick change in three of those namespaces, runs `helm upgrade` by hand with the right chart version and the right values file but without `--post-renderer`, and the patch silently disappears from those three StatefulSets. No command failed. No status is anything but `deployed`. ## Why Helm lets this happen Helm persists a release's **inputs** selectively. The merged values are stored with the revision, which is why `--reuse-values` and `--reset-then-reuse-values` exist and can reconstruct them on a later upgrade. The post-renderer is not an input in that sense: it is a flag on one invocation, and Helm keeps no record that it was used, which plugin it was, or what version of it ran. So the upgrade without the flag is not, from Helm's point of view, an incomplete command. It is a perfectly ordinary upgrade of the same chart with the same values, and it produces a desired manifest in which the patched field does not appear. ## Why the field is then actively removed This is the part candidates often get wrong: the patch does not merely stop being reasserted, it is taken away. Helm compares what it declared before with what it declares now. Under Helm 4's default write path, server-side apply, Helm's field manager owned the patched field on the live object. The new apply no longer declares it, so the API server removes it as a field the manager has released. On `upgrade` the `--server-side` value defaults to `"auto"`, inheriting the previous release's apply method — so a release first installed by Helm 3 stays on the older path, where Helm computes a three-way merge between the stored manifest of the previous revision, the newly rendered manifest and the live object; the field is present in the old stored manifest and absent from the new one, so the merge patches it out. Either way the outcome is the same and it is silent. ## What the release record still shows Helm stores the **post-renderer's output** as each revision's manifest, so `helm get manifest <release> -n <namespace> --revision 47` still shows the patched YAML for the revision that was deployed with the flag. Two useful things follow. First, you can detect the incident by diffing `helm get manifest` for the current revision against the previous one, or by comparing the same command across all twelve namespaces and finding the three that differ. Second, `helm rollback` to revision 47 restores the patch, because rollback replays that revision's stored manifest rather than re-rendering the chart — the post-renderer does not need to run, and does not need to be installed, for the rollback to bring the patched form back. ## Preventing it Discipline is not a control here, because the failing command looks completely normal. Make the flag impossible to omit: - **One entry point.** Deploys go through a pipeline step or a wrapper script that always passes `--post-renderer`; direct `helm upgrade` against those releases is not part of the runbook. - **Pin and provision the plugin.** In Helm 4 the post-renderer must be installed on every machine that deploys, at a known version, or two operators produce different desired states from identical inputs. - **Assert the patch.** Render with `helm template --post-renderer …` in CI and fail the job if the expected field is absent. That catches both a missing plugin and a transformation that silently stopped matching after an upstream chart bump renamed or restructured the object it targets. - **Detect drift across the fleet.** For a 12-namespace fleet, a scheduled job that pulls `helm get manifest` per namespace and compares the patched fields turns a silent regression into an alert. - **Prefer a value.** If the upstream chart grows a knob that does what your patch does, delete the post-renderer. A patch that no longer exists cannot be forgotten. The general lesson generalises past post-renderers: any deploy-time behaviour that lives in a flag rather than in the release's stored state is one hand-typed command away from being lost.

  • Would --reuse-values on that upgrade have saved the patch?
    No. `--reuse-values` recovers the values stored with the previous revision, and the patch was never a value — it was applied after rendering, downstream of every value. The upgrade would render the same plain chart from the same values and still omit the patched field. Nothing in the values family can recover a post-renderer.
  • Why does helm rollback restore the patch when the post-renderer is not installed on that machine?
    Because rollback does not re-render. It replays the stored manifest of the target revision, and the manifest Helm stored was the post-renderer's output. That makes rollback a genuine recovery path for this failure — though it also rolls back everything else in that revision, so it is a blunt instrument if the bad upgrade also carried a wanted change.
  • How would you catch this in CI rather than in production?
    Render the chart with `helm template --post-renderer …` in the pipeline and assert on the output — fail the job if the field the transformation is supposed to add is missing. That single check catches a missing or broken plugin, an unpinned plugin version, and a transformation whose selector stopped matching after the upstream chart restructured the object.

saying these in an interview costs you the question

  • Says --reuse-values would have kept the patch
  • Thinks Helm remembers which post-renderer a release used
  • Claims the field just stops being reasserted, not removed
  • Believes the upgrade would have failed loudly
  • Expects helm rollback to re-run the post-renderer
  • Relies on people remembering to pass the flag

context

resolving links…