In Kubernetes, what happens when you run kubectl delete namespace, and why does the namespace show Terminating before it disappears?

## Two phases: `Active` and `Terminating` A Kubernetes **Namespace** reports its lifecycle in `status.phase`, which has exactly two values: - **`Active`** — the namespace is in normal use and new objects can be created in it. - **`Terminating`** — someone has asked for the namespace to be deleted and the cluster is cleaning it out. When you run `kubectl delete namespace recsys-inference`, the API server does **not** remove the Namespace object. On that first delete request it sets `metadata.deletionTimestamp` and flips `status.phase` to `Terminating` in the same write. The object stays readable, and `kubectl get namespace` keeps listing it until the cleanup is finished. By default `kubectl delete` also waits (`--wait=true`) for the object to be gone, which is why the command can sit there for minutes. ## The `kubernetes` entry in `spec.finalizers` A **finalizer** is a marker that says "some component still has work to do before this object may be removed". Namespaces carry a special one. When a Namespace is created, the API server adds the value `kubernetes` to the namespace's own `spec.finalizers` list (a Namespace-only field, separate from the generic `metadata.finalizers` list every object has). The rule the API server enforces is simple: a namespace with a `deletionTimestamp` is only removed from storage once `spec.finalizers` is **empty** (and, like any object, once its `metadata.finalizers` is empty too). The component that owns the `kubernetes` entry is the **namespace controller**, one of the controllers inside `kube-controller-manager`. Until it removes that entry, the namespace cannot go away. ## What the namespace controller does For every namespace that has a `deletionTimestamp`, the namespace controller runs a purge loop: 1. **Discover** every namespaced resource type the API server serves, including custom resources and types served by aggregated APIs. 2. **Delete pods first.** It deletes every Pod in the namespace and waits until none remain before touching any other type, so workloads do not lose their Secrets, ConfigMaps or Services while they are still shutting down. 3. **Delete everything else** — Deployments, Services, ConfigMaps, Secrets, PersistentVolumeClaims, custom resources — type by type. 4. **Record progress** in the namespace's `status.conditions` (for example `NamespaceContentRemaining` when objects are still present). 5. **Retry** until every type reports zero objects left. 6. **Finalize**: remove `kubernetes` from `spec.finalizers`. With the list empty, the API server deletes the Namespace object itself. The deletions in steps 2 and 3 are ordinary deletes, so each object still goes through its own graceful shutdown and its own finalizers before it really disappears. ## Why `Terminating` can last a while The namespace can only finish as fast as its slowest object. Suppose `recsys-inference` runs a recommendation-model inference server whose pods set `terminationGracePeriodSeconds: 1140` so that in-flight batch requests can drain. On a 3-node kubeadm cluster, deleting the namespace leaves it in `Terminating` for about **19 minutes**: the pods honour their grace period, and because pods go first, the namespace's other objects are not even deleted until those pods are gone. That is normal behaviour, not a hang. ## What you can and cannot do while it terminates | Action in a `Terminating` namespace | Result | |---|---| | Create a new object | Rejected by the `NamespaceLifecycle` admission plugin: "unable to create new content in namespace ... because it is being terminated" | | Read existing objects | Allowed; they are still being removed | | Update or delete existing objects | Allowed | | Delete the namespace again | Returns the namespace object without starting anything new | | Create a namespace with the same name | Fails until the old one is fully gone | A CI pipeline that deletes and immediately recreates a namespace of the same name therefore fails intermittently. It should wait for the old namespace to disappear (for example with `kubectl wait --for=delete namespace/recsys-inference --timeout=25m`) or use a fresh name. ## Namespaces you cannot delete The same `NamespaceLifecycle` admission plugin refuses to delete `default`, `kube-system` and `kube-public` at all ("this namespace may not be deleted"). Every other namespace, including ones an add-on created, can be deleted and will be purged. ## Version note Deleting pods before other content is the `OrderedNamespaceDeletion` behaviour. It has been GA and locked on since Kubernetes 1.34, and by 1.37 its feature gate has been removed, so the ordering is always on. On older clusters the controller deleted all types without that ordering.