A common fix for a Kubernetes namespace stuck in Terminating is emptying spec.finalizers through the /finalize subresource; what does that actually do, and when is it acceptable?

## The shortcut everyone finds Search for "Kubernetes namespace stuck in Terminating" and the first answer is nearly always the same: fetch the Namespace as JSON, empty its `spec.finalizers`, and send it to the namespace's `/finalize` subresource. The namespace disappears within a second. Knowing **why** that works, and **what it skips**, is what separates a safe operator from a lucky one. ## Why `kubectl edit` does not work A **Namespace** being deleted has a `deletionTimestamp`, phase `Terminating`, and the entry `kubernetes` in `spec.finalizers`. That entry is owned by the **namespace controller** in `kube-controller-manager`, which removes it only after deleting all content. The API server protects this field. On an ordinary update of a Namespace — `kubectl edit`, `kubectl apply`, `kubectl patch` against the main resource — it copies `spec.finalizers` from the stored object, so any change you make to that list is silently discarded. The only write path that can change it is the dedicated **`/finalize` subresource**, `PUT /api/v1/namespaces/{name}/finalize`, the same endpoint the namespace controller itself calls. ```bash kubectl get namespace recsys-inference -o json \ | jq '.spec.finalizers = []' \ | kubectl replace --raw /api/v1/namespaces/recsys-inference/finalize -f - ``` Once a namespace has a `deletionTimestamp` and an empty `spec.finalizers` (and no `metadata.finalizers`), the API server deletes it from storage. ## What the shortcut skips Removing the finalizer does **not** run the purge. It tells the API server "the cleanup is done" when it is not. Consequences: 1. **Leftover objects stay in storage.** Any object that still existed in the namespace remains in etcd under that namespace's name. Nothing deletes it any more, because the namespace controller only processes namespaces that exist. 2. **They come back.** Create a namespace with the same name later and the old objects are there again — old custom resources, old Secrets, stale configuration for the recommendation-model inference server, now mixed into a new tenant's namespace. 3. **External resources leak.** Finalizers on the leftover objects usually stood for cleanup outside the cluster: a storage volume, a load balancer, a DNS record, a database. Skipping them means those are never released. 4. **The real fault is still there.** If the cause was an unavailable aggregated API, every other namespace deleted later will get stuck in the same way. | | Fixing the cause | Force-finalizing | |---|---|---| | Namespace content | Deleted by the namespace controller | Left in etcd | | Object finalizers | Run by their owners | Never run | | External resources | Released | Possibly leaked | | Same-name namespace later | Starts empty | Can show old objects | | Other stuck namespaces | Also resolved | Still stuck | ## When it is acceptable Force-finalizing is a legitimate last resort when **all** of these hold: - You read `status.conditions` and know the cause. - You listed what remains (`kubectl api-resources --verbs=list --namespaced -o name` piped into `kubectl get -n <ns>`) and it is either nothing, or objects you have deliberately accepted as leftovers. - Anything external those objects managed is already gone or has a manual cleanup ticket. - The namespace name will not be reused, or you have checked that nothing will resurface. A common legitimate case: on a 3-node kubeadm test cluster, an add-on was removed and its aggregated API can never come back; you delete the stale `APIService`, and any namespace still stuck after the next retry is inspected and, if empty, finalized. ## Guardrails for a shared cluster - **Restrict the permission.** Writing the subresource needs `update` on `namespaces/finalize`. Grant it only to cluster administrators; a tenant who holds it can make their namespace vanish while its objects stay behind. - **Audit it.** Calls to the `/finalize` subresource by anyone other than the namespace controller's own identity are worth an alert. - **Write a runbook.** Put the diagnosis first and the force step last, with the checklist above as a gate. - **Fix the lifecycle.** Uninstall add-ons in order: their custom resources first, their controllers next, their `APIService` together with its backend. ## Key takeaway The `/finalize` subresource does not delete anything. It removes the one marker that made the API server wait for the cleanup. Use it only after you have made sure the cleanup is truly unnecessary. In an interview, the strong answer names both halves: how the subresource works, and the checklist that has to pass before anyone is allowed to call it.