What happens to a chart's helm.sh/hook manifests when a controller applies helm template output?
answer
- Who reads those annotations
- The manifests are still in the output
- --no-hooks is the flag that removes them
- A fixed-name Job that nothing deletes
- helm test needs a release to exist
basics
~20 sThey are still rendered into the output unless --no-hooks is passed, so they get applied as ordinary objects in the same batch as everything else. No hook event fires, hook-weight orders nothing, and hook-delete-policy deletes nothing.
solid answer
~50 sThe annotations are instructions to Helm's installer, not to the cluster. `helm template` prints hook-annotated manifests like any other document unless you pass `--no-hooks`, so whatever applies that YAML creates them alongside the Deployment rather than before it. A `pre-upgrade` migration Job therefore starts at the same moment as the pods that assume the migration already ran; `helm.sh/hook-weight` no longer sequences anything; `helm.sh/hook-delete-policy` never fires, so a fixed-name Job survives and a later apply cannot re-run it; a `helm.sh/hook: test` Pod becomes an ordinary Pod created on every apply; and `post-delete` never happens at all because nothing uninstalls. `helm test` is also unavailable, since it needs a release record. The fix is to stop relying on hook semantics: make the work idempotent, name it after something that changes per release such as the image tag or `.Chart.AppVersion`, and let the applying system express the ordering.
code
yaml · 17 linesapiVersion: batch/v1
kind: Job
metadata:
name: ingest-gw-schema-migrate
annotations:
helm.sh/hook: pre-upgrade
helm.sh/hook-weight: "-5"
helm.sh/hook-delete-policy: before-hook-creation
spec:
backoffLimit: 2
template:
spec:
restartPolicy: Never
containers:
- name: migrate
image: registry.example.com/ingest-gw-migrate:2.7.3
args: ["--schema", "v41"]go deeper
Know that helm.sh/hook is an instruction to Helm, not to the cluster, and that rendering only produces YAML. If nothing runs Helm's install path, nothing acts on those annotations.
Explain that the hook manifests are still printed unless --no-hooks is passed, and walk through what that means: applied in the same batch, no ordering, no cleanup, no delete-time events at all.
Bring the second-release failure: a fixed-name Job nothing deleted, an apply rejected because a pod template cannot change in place, and a migration that quietly stopped running. Then say what you replace hooks with.
Decide the policy for charts you do not own. Third-party charts lean on hooks, so mandating a render-and-apply model means auditing them or accepting that some charts must keep Helm as their installer.
## What a hook actually is A hook is an ordinary manifest with an annotation on it. `helm.sh/hook` names the event, `helm.sh/hook-weight` gives a sort key within an event, and `helm.sh/hook-delete-policy` says when Helm may delete the object afterwards. Nothing in a Kubernetes cluster understands any of those strings. They are read by Helm's installer, which pulls those documents out of the render, applies them at the right moment, waits for them, and then applies or deletes according to the policy. Remove the installer and you remove the only reader. ## Rendered, not removed The first surprise is that hooks do not vanish from the output. `helm template` prints hook-annotated documents in the same stream as everything else - `--no-hooks` is the flag that omits them. So a controller that applies the render creates the hook objects as plain resources, in whatever order its applier happens to use, at the same time as the workloads. For a telemetry ingest gateway whose chart carries a `pre-upgrade` schema-migration Job, the concrete consequence is that the migration Job and the new gateway pods are created together. The pods start against the old schema, crash-loop or, worse, write records the new code assumes are already converted, until the Job finishes. Nothing failed; the ordering guarantee simply was not there to begin with. ## Delete policies and fixed names The second surprise arrives on the following release. Under Helm, `helm.sh/hook-delete-policy: before-hook-creation` means Helm removes the previous Job before creating the new one, which is what lets a hook keep a stable, predictable name across releases. Under render-and-apply nothing deletes it, so the Job with that name is already there. Re-applying an unchanged document changes nothing and the migration does not re-run. Re-applying one whose pod template changed - a new migration image tag, say - is rejected, because a Job's pod template cannot be edited in place. The pipeline reports a failure on a resource nobody thinks of as a resource. The usual escape is a name that changes when the work changes. Note that `.Release.Revision` is not available for this: an offline render always produces the first-install value, so a Job named after the revision has the same name forever. Use the image tag, `.Chart.Version` or `.Chart.AppVersion` instead - all of which move when you actually ship something new. ## Test hooks Manifests annotated `helm.sh/hook: test` are meant to be created on demand by `helm test` against a running release and then reaped. In a rendered pipeline they are simply part of the manifest set: a Pod that gets created on every apply, runs its assertions against whatever is live at that instant, and then sits in the namespace as a completed or failed Pod that nothing cleans up. And `helm test` itself is unavailable, because it looks up a release record that was never written. Teams usually pass `--no-hooks` at render time to keep test manifests out of the stream entirely, then run the same assertions as a job in the delivery system. ## Events that can never occur Some events have no counterpart at all in a render-and-apply model. `post-delete` and `pre-delete` presuppose an uninstall, and nothing uninstalls a set of applied YAML documents in Helm's sense. If your chart cleans up an external resource - deregistering the gateway from an upstream collector, say - that cleanup silently stops happening the day you switch delivery models, and nobody notices until the external system fills up with stale registrations. ## Designing for the model you are actually in There are three defensible positions. Keep Helm as the installer, and the hooks keep working. Or accept the rendered model and stop using hooks: pass `--no-hooks` so the manifests do not leak into the stream at all, make any setup work idempotent so it is safe to apply repeatedly, and express ordering with whatever primitive the applying system offers rather than with annotations it does not read. Or move the work out of the chart entirely, into a pipeline step that runs before the apply, where its ordering, its logs and its failure handling are all visible in one place. The answer an interviewer is listening for is that hook annotations are not a property of the manifests, they are a contract with Helm's install path - and that changing who applies the YAML silently voids that contract without producing a single error message.
- The chart names its migration Job after the release revision so each upgrade gets a fresh one. Why does that not help here?An offline render always produces the first-install revision value, because there is no release record to advance. Every render therefore emits the same Job name, and the second apply either does nothing or is rejected outright when the pod template changed. Name the Job after the migration image tag, `.Chart.Version` or `.Chart.AppVersion` instead - all of them move when the work itself moves.
- Would you pass --no-hooks when rendering, or leave the hook manifests in the output?Pass it, in most cases. Leaving them in means shipping objects whose annotations promise ordering and cleanup that nobody will provide - a trap for the next reader and, for test hooks, a Pod created on every apply. Excluding them makes the gap explicit, and then you deliberately reintroduce the work: an idempotent init step, a pipeline stage before the apply, or an ordering primitive the applying system actually implements.
- How do you run a chart's tests when there is no release to test?You cannot use `helm test` - it looks up a release record and there is none. Either keep a Helm-installed copy of the chart in a pre-production namespace and run `helm test` there, or lift the assertions out of the chart into a delivery-system job that runs against the deployed endpoints after the apply. The second is more common, because it also covers what the chart cannot see.
saying these in an interview costs you the question
- Thinks helm template strips hook manifests from its output
- Believes the cluster interprets helm.sh/hook annotations
- Expects hook-weight to order a plain apply
- Assumes hook-delete-policy still cleans up completed Jobs
- Says helm test works against objects applied from rendered YAML