Why would `helm upgrade` keep failing in its pre-upgrade hook phase because the migration Job already exists, and how do you fix it?

### The shape of the failure A pre-upgrade hook that runs schema migrations is the classic case. The hook's Job has a stable name so it can be found and re-run; the chart's helper truncates that name at 63 characters, so it is byte-identical on every upgrade of a given release. The upgrade reaches the pre-upgrade phase, Helm tries to create the Job, and a Job with that name is already sitting in the namespace from a previous attempt. The hook phase fails, the upgrade aborts, and it fails the same way on every retry — nothing about the situation changes on its own. Helm cannot patch its way out either: a Job's pod template cannot be edited after creation, so "just update the existing one" is not available to it. ### Diagnosing it Work from the release outward. 1. `helm history <release> -n <ns>` shows the failed revision and confirms the release never got past the hook. 2. `kubectl get jobs -n <ns>` shows the leftover, and its completion count tells you which outcome left it there — a `Complete` Job means the hook succeeded once and nothing removed it, a Job with failures means the run that aborted the previous upgrade is still on disk. 3. `helm get hooks <release> -n <ns>` prints the hook manifests as rendered for that release. Read the `helm.sh/hook-delete-policy` annotation on the migration Job. This is where the answer almost always is. Two annotation states produce this failure. Either the policy list is `hook-succeeded` only — so a successful run cleans up, an unsuccessful one does not, and `before-hook-creation` is no longer in force because writing any value replaces the default — or the policy is `hook-succeeded,hook-failed`, which cleans up both outcomes but not a run that never completed: a Job whose Pod was evicted, a namespace whose quota blocked the Pod, or an upgrade the operator interrupted before Helm reached the deletion step. In both cases the pre-creation delete is missing, and that is the only policy that guarantees a clean slate at the start of a run. ### Why only some namespaces A multi-tenant chart installed once per team namespace makes this look like an environment problem. Three of fourteen namespaces fail; the other eleven upgrade fine. The eleven are simply the ones whose last migration run completed cleanly and deleted its Job. The three are where something went wrong once — weeks ago, possibly under a different chart version — and the wreckage has been waiting on the hook's name ever since. Nothing is special about those clusters; they are the ones that exercised the missing policy. ### Fixing it The unblock is manual and immediate: delete the leftover Job in each affected namespace and re-run the upgrade. Do read its Pod logs first — that Job is the record of why the earlier run failed, and once it is gone the reason is gone. The actual fix is in the chart: put `before-hook-creation` back into the policy list. ```yaml annotations: "helm.sh/hook": pre-upgrade "helm.sh/hook-weight": "0" "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded ``` Now every run starts by clearing whatever holds the name, whatever left it there, and the failure cannot recur. Keeping `hook-succeeded` stops completed Jobs accumulating; leaving `hook-failed` off means the next failure is still readable in the namespace, which is what you wanted in step 2 above. ### The alternative, and why it is usually worse The other way to dodge a name collision is to stop reusing the name — build the hook Job's name from `.Release.Revision` or another per-run value so each upgrade creates a distinct object. It works, and it preserves every run's logs. It also means nothing ever deletes anything: after a year of weekly releases each namespace carries a pile of dead Jobs and Pods, consuming object quota and making `kubectl get jobs` useless, and the leftovers survive uninstall because Helm does not track hook resources. If you take that route you need a deliberate reaping story — Kubernetes' own time-to-live field on a Job is the one Helm's documentation points at — rather than an implicit one. ### What not to do `--no-hooks` will get the upgrade through, and it will get it through by skipping the migration, which is how a schema-dependent release ships against an unmigrated database. Deleting the Job by hand as a runbook step is the same failure repeated on a schedule: it is the correct emergency action and an admission that the chart is unfinished. And be sceptical of a fix that only makes the hook Job's name more unique without saying who deletes it — uniqueness converts a hard failure into a slow one. ### Prevention for a chart many teams install Treat the hook's delete policy as part of the chart's contract, not as an operator's concern: consumers install into namespaces you do not see and will not clean up. Give every run-to-completion hook `before-hook-creation` so the chart is re-runnable from any state, decide explicitly whether failures are kept for evidence or reaped, and name hook objects from the release so one tenant's cleanup can never target another's object.