After `helm uninstall`, a Job created by a chart's pre-install hook is still in the namespace. Why?
answer
- Not everything the chart made is tracked
- The release record holds two lists
- Uninstall walks only one of them
- The default policy fires before the next run
- Nothing runs that hook again after uninstall
basics
~20 sHelm does not track hook resources in the release manifest, so uninstall removes only the chart's ordinary resources. A hook object is removed solely by its own helm.sh/hook-delete-policy; with no matching policy it outlives the release.
solid answer
~40 sWhen Helm renders a chart, manifests annotated `helm.sh/hook` are recorded separately from the release manifest. Only that manifest is the release's tracked state, and `helm uninstall` deletes what is in it — hook objects were never in the list, so they are left behind, Pods and logs included. The only cleanup mechanism for a hook resource is `helm.sh/hook-delete-policy`. Its default, `before-hook-creation`, deletes the object just before that hook runs *again*, which after an uninstall never happens. To make a hook clean up after itself, add `hook-succeeded` (and usually `hook-failed`) so the object is deleted the moment the hook finishes. Anything the hook's own process creates at runtime is invisible to Helm entirely and has to be undone by the hook.
code
yaml · 15 linesapiVersion: batch/v1
kind: Job
metadata:
name: order-checkout-seed
annotations:
"helm.sh/hook": pre-install
"helm.sh/hook-delete-policy": hook-succeeded,hook-failed
spec:
template:
spec:
restartPolicy: Never
containers:
- name: seed
image: registry.example.com/order-checkout:1.9.3
args: ["seed-reference-data"]go deeper
Recall that Helm keeps hook manifests apart from the release manifest and that uninstall only deletes the latter. Naming helm.sh/hook-delete-policy as the thing that governs hook cleanup is the expected answer.
Explain the timing: the default policy deletes before the hook's next creation, so it can never fire after a teardown, while hook-succeeded and hook-failed fire at completion. Be ready to say which annotation you would add and why.
Show that you audit charts for this before they reach many namespaces, and that you weigh keeping a failed hook object for its logs against leaving litter in tenants' namespaces you will never revisit.
Own the contract question: a chart you publish is installed into namespaces you cannot inspect, so hook lifecycle is part of the chart's promise, not the operator's runbook. Decide what evidence a hook must leave and what it must never leave.
### Two lists, not one When Helm renders a chart it sorts the rendered manifests into two piles. Anything carrying the `helm.sh/hook` annotation is pulled out and recorded as a **hook**; everything else becomes the release **manifest**. Only that manifest is the release's tracked desired state — it is what an upgrade diffs, what `helm get manifest` prints, and what `helm uninstall` walks when it deletes. The hooks are stored beside it in the release record and replayed at the events they ask for. `helm get hooks <release>` prints them, and it keeps printing them long after the live objects are gone, because it is reading stored YAML rather than the cluster. ### Uninstall only walks the manifest Deletion is driven by that tracked manifest, so a hook object was never a candidate for removal in the first place. A pre-install Job that ran once, completed, and was never given a delete policy simply stays: the Job object, its Pods and their logs remain in the namespace after the release is gone. The same is true of everything else a hook renders — the ServiceAccount it runs as, the Role that lets it read a Secret, a ConfigMap holding the SQL it applies. None of it is release-managed, so none of it is release-deleted. ### The default policy has the wrong timing to help With no `helm.sh/hook-delete-policy` annotation, Helm behaves as if `before-hook-creation` were set. That value deletes an existing object of the hook's kind and name *immediately before Helm creates that hook again*. It is a pre-cleanup, and it fires only on the next run of that hook. After an uninstall there is no next run, so the default never fires and the object survives indefinitely. Most people who claim "Helm cleans hooks up automatically" are thinking of this policy and have missed that its trigger is the next install or upgrade, not teardown. ### Making a hook clean up after itself The completion-time policies are the fix, and they are set on the hook manifest itself: ```yaml metadata: name: order-checkout-seed annotations: "helm.sh/hook": pre-install "helm.sh/hook-delete-policy": hook-succeeded,hook-failed ``` `hook-succeeded` removes the object as soon as Helm judges the hook successful; `hook-failed` removes it when the hook fails. List both and the object never outlives its own run, so there is nothing for uninstall to leave behind. The cost is debuggability: with `hook-failed` set, the failed Job and its Pod logs vanish at exactly the moment you want to read them, which is why many charts settle on `before-hook-creation,hook-succeeded` and deliberately keep failures around for a human. Helm's own documentation also points at Kubernetes' time-to-live field on a Job as an alternative reaper for finished Jobs. ### Objects a hook creates at runtime are worse Everything above concerns objects Helm renders and applies. If the hook's own process calls the cluster API and creates something — a temporary Secret for a one-off credential, a debug Pod — Helm has no record of it at all, no annotation can govern it, and no policy will remove it. That work has to be undone by the hook itself, or by a delete-phase hook that carries its own delete policy so it does not become another leftover. ### A worked example A multi-tenant chart for an order-checkout API is installed once per team namespace, fourteen namespaces in all. Each release runs a pre-install Job that seeds reference data. Nobody annotated it, so every namespace accumulates one completed Job per install. A team decommissions its namespace's release with `helm uninstall`; `helm list` shows nothing, but `kubectl get jobs -n team-checkout-7` still shows the seed Job and one `Completed` Pod. Reinstall six weeks later and the picture gets confusing: the default `before-hook-creation` does delete that stale Job at last — because the hook is running again — so the leftover quietly disappears at the least informative moment, and the team concludes uninstall had cleaned it up all along. ### What to check when you see one Run `helm get hooks <release>` (or read the chart) and look at the annotations on the hook in question. If there is no `helm.sh/hook-delete-policy`, or it lists only `before-hook-creation`, the object is meant to survive between runs and nothing will remove it once the release is gone. Decide per hook whether the artefact is evidence you want to keep or litter you want gone, then say so in the annotation rather than deleting objects by hand after every teardown.
- Two hook Jobs from the same chart are gone after uninstall and a third is still there. What explains the difference?The two that vanished carried a delete policy matching how they ended — `hook-succeeded` for a good run, `hook-failed` for a bad one — so Helm removed them the moment the hook completed, long before the uninstall. The third either has no policy at all or lists only `before-hook-creation`, which deletes on the next run rather than at completion. With the release gone, that next run never comes.
- The hook Job itself created a temporary Secret while it ran. What removes that?Nothing Helm offers. Delete policies apply to objects Helm rendered and applied from the chart; an object the hook's process created through the API is unknown to Helm, absent from both the manifest and the hook list, and untouched by any annotation. The hook has to delete it before exiting, or a later delete-phase hook has to — and that hook needs its own delete policy so it does not become the next leftover.
The release manifest is the moving inventory: uninstall carries out everything on the list. A hook object is the packing crate nobody wrote down, so it stays in the empty room.
saying these in an interview costs you the question
- Claiming helm uninstall removes everything the chart ever created
- Believing hook manifests are part of the release manifest
- Expecting the default before-hook-creation policy to clean up at uninstall
- Saying a completed Job disappears on its own
- Treating manual kubectl delete as the fix rather than the chart
- Assuming --no-hooks on uninstall is what caused the leftovers