skip to content

Which parts of a third-party Helm chart run code you never asked for?

level: middleimportance: should knowfreq 50%

answer

  1. Some manifests are not release members at all
  2. One annotation moves a Job ahead of everything
  3. Ordinary Deployments can run more than one image
  4. Render twice, once with --no-hooks, and diff
  5. A dry run shows them but never runs them

basics

~20 s

Manifests annotated helm.sh/hook — usually Jobs — run at install and upgrade time, before your workloads exist. So do initContainers and sidecars inside otherwise ordinary pod specs, and the pods helm test starts. All of them execute images the chart chose.

solid answer

~40 s

Three places in a chart execute something the value table never mentions. **Hook manifests**: any object annotated `helm.sh/hook: pre-install` (or one of the other eight events) is applied ahead of the release's own resources, ordered by `helm.sh/hook-weight`, and Helm waits for a hook Job to complete before continuing. **initContainers and sidecars** in ordinary pod specs — a Deployment that looks innocuous can mount a hostPath and run a setup image first. **`helm test` pods**, which run in-cluster against a real release. Isolate the hooks by rendering twice: `helm template ... > all.yaml` and `helm template ... --no-hooks > nohooks.yaml`, then diff. Read each hook's image, args, serviceAccountName and `helm.sh/hook-delete-policy` — without a delete policy the Job object survives the install and stays visible.

code

bash · 4 lines
bash
helm template checkout ./checkout-platform -f prod-values.yaml > all.yaml
helm template checkout ./checkout-platform -f prod-values.yaml --no-hooks > nohooks.yaml
diff nohooks.yaml all.yaml
grep -n 'initContainers:' all.yaml

go deeper

for a junior

Recall that a manifest annotated helm.sh/hook runs at install or upgrade time rather than being part of the release, and that hook Jobs execute an image the chart chose.

for a middle

Explain the mechanics: the hook events, hook-weight ordering, hook-delete-policy deciding whether the Job survives, and why --no-hooks plus a diff isolates the hook set from a large render.

for a senior

Demonstrate the judgement that a hook mutating external state is not covered by a Kubernetes rollback, and size timeouts and review effort around irreversibility rather than around object count.

for a principal

Own the policy question of which vendor charts may run privileged hooks in your clusters at all, and whether migrations belong inside a chart's install path or in a deliberate, separately approved step.

### The value table is not the attack surface A chart's `values.yaml` describes what the author expects you to configure. It does not describe what the chart runs. Three constructs execute images at times you did not choose, and all three are visible in the rendered output if you know to look for them. ### 1. Hooks Any manifest carrying the `helm.sh/hook` annotation is pulled out of the normal release and applied at a lifecycle event instead. Helm defines nine events — pre-install, post-install, pre-upgrade, post-upgrade, pre-delete, post-delete, pre-rollback, post-rollback and test. `helm.sh/hook-weight` orders them (lower first, as strings sorted numerically), and Helm waits for each hook to reach a ready/complete state before applying the next weight and, eventually, the release's own manifests. The consequences for a chart you did not write are concrete. A `pre-install` Job runs **before** any of the workloads you reviewed exist, so "it only starts the order-checkout API" is not what happens first. The Job runs an image the chart names, with args the chart chose, under a serviceAccountName the chart may also have created — which is why the same render pass that finds the hook should follow the ServiceAccount it references. `helm.sh/hook-delete-policy` decides the Job object's fate: `before-hook-creation` (the default behaviour) deletes a previous one before creating the new one, `hook-succeeded` deletes it on success, `hook-failed` on failure. With no policy that succeeds silently, the Job and its completed pod stick around — annoying, but also the only place its logs still exist. Hook resources are not tracked like ordinary release members: Helm does not manage them across upgrades the way it manages a Deployment, and `helm uninstall` does not necessarily clean them up. `helm get hooks <release>` lists them for an installed release. ### 2. initContainers and sidecars These hide in plain sight because the enclosing object is an ordinary Deployment or StatefulSet. An initContainer runs to completion before the main container starts, frequently as root, frequently with a volume mount the main container does not have — chown-ing a data directory, tuning a sysctl, fetching config. A sidecar runs for the pod's whole life and shares its network namespace. Both are images the chart picked, and neither appears in a kind inventory: `grep '^kind:'` shows one Deployment whether it holds one container or five. ### 3. `helm test` `helm test <release>` runs the manifests annotated `helm.sh/hook: test` as pods in the cluster, against the live release. It is a command an operator runs deliberately, but the pod specs come from the chart, so they belong in the same review. ### Isolating them in the render `helm template` includes hook manifests in its output, which is good, but they arrive mixed into a thousand lines of YAML and are distinguished only by an annotation. The clean trick is to render twice and diff: render once normally, once with `--no-hooks`, and the difference is exactly the chart's hook set. Then grep the whole render for `initContainers:` and read each pod spec's full container list. ### Why dry-running does not help here Neither `helm install --dry-run=client` nor `--dry-run=server` executes hooks. A dry run tells you a `pre-upgrade` Job exists and shows you its spec; it never tells you what the image inside it does. That gap is the whole reason the review is a reading exercise: you are judging an image reference and a command line, not observing behaviour. ### The failure mode this prevents Consider an umbrella chart bundling an order-checkout API and a worker subchart, installed with `--timeout 96s`. Its `pre-upgrade` hook Job runs a schema migration against the production database and takes longer than that. Helm's wait expires, the upgrade fails, and the migration is still running — half-applied, with a release now in a failed state and an already-mutated database that no rollback of Kubernetes objects can undo. Nothing in the value table said an upgrade would touch the database; the hook manifest did. Knowing a chart's hooks before you run it is what turns that from an incident into a decision: give the migration its own review, run it deliberately, or set a timeout that matches reality. The general rule: in a third-party chart, ask which images run, when, as whom, and whether their effects are reversible. Hooks answer "when", initContainers answer "which images you missed", and the serviceAccount reference answers "as whom".

  • A chart's pre-upgrade hook Job is still running when the install times out. What state are you in?
    Helm marks the upgrade failed, but the Job is a real object that keeps running unless something deletes it. Anything it already did — a schema migration, a data backfill — has happened and is not undone by rolling back the Kubernetes objects. That asymmetry is why a hook that mutates external state deserves review and a timeout sized to it, not a default.
  • Why is helm.sh/hook-delete-policy worth reading during a chart review?
    It decides whether the hook's Job object and pod survive. `hook-succeeded` deletes on success, `hook-failed` on failure, `before-hook-creation` clears the previous one first. A hook you may need to debug should keep its pod so the logs remain; a hook left with no policy accumulates completed Jobs in the namespace, which is untidy but at least auditable.
  • The chart's kind inventory shows only Deployments and Services. Is nothing else running?
    No — a kind count cannot see inside a pod spec. Each Deployment can carry initContainers and sidecars running images the chart chose, often with wider mounts or a root securityContext than the main container. Grep the render for `initContainers:` and read the full container list of every pod template.

saying these in an interview costs you the question

  • Believing values.yaml lists everything the chart runs
  • Thinking a dry run executes the chart's hooks
  • Assuming hooks run after the workloads are up
  • Missing initContainers because the kind is just Deployment
  • Expecting helm uninstall to clean up every hook resource
  • Treating helm test pods as if they ran outside the cluster

context