skip to content

Namespace Deletion

7 roadmaps3 questions

Deleting a namespace only marks it Terminating: the namespace controller purges every object inside, and the namespace goes only once that sweep finishes. Interviewers use the namespace stuck in Terminating to see whether you can reason about finalizers.

part ofKubernetesoverview, primer and where to startread it →
on this pageshow

explore

questions

3

A Kubernetes namespace has been stuck in Terminating for 19 minutes; how do you find what is blocking the namespace controller's purge, and fix it?

level: middleimportance: must knowfreq 72%

answer

  1. the namespace explains itself
  2. status.conditions before any force
  3. whose finalizer, whose controller
  4. discovery must be complete
  5. kubectl get apiservices, AVAILABLE column

basics

~20 s

Read the namespace's status.conditions. They usually point to objects held by finalizers whose controller is gone, or to an unavailable aggregated API that makes discovery fail. Restore that controller or API, or remove the stale finalizer or APIService.

solid answer

~40 s

I start with `kubectl get namespace <ns> -o yaml` and read `status.conditions`, because the namespace controller writes why it cannot finish there. `NamespaceContentRemaining` and `NamespaceFinalizersRemaining` mean objects are still there, held by finalizers — usually an operator was uninstalled before its custom resources were deleted. I list them with `kubectl api-resources --verbs=list --namespaced -o name` piped into `kubectl get`, then restore the operator or remove that object's finalizer once I know what it protected. `NamespaceDeletionDiscoveryFailure` means an aggregated API is unavailable. The controller will not finalize a namespace it cannot fully list, so I check `kubectl get apiservices` for one with `AVAILABLE` `False`, then restore its backend or delete the stale `APIService`. Force-emptying `spec.finalizers` comes last.

code

bash · 4 lines
bash
kubectl get namespace recsys-inference -o jsonpath='{range .status.conditions[*]}{.type}={.status}: {.message}{"\n"}{end}'
kubectl api-resources --verbs=list --namespaced -o name \
  | xargs -n 1 kubectl get --show-kind --ignore-not-found -n recsys-inference
kubectl get apiservices

go deeper

for a junior

Remember that a stuck namespace records its reason in status.conditions, and that finalizers on leftover objects are the most common cause.

for a middle

Explain both blocking paths: object finalizers nobody clears, and a discovery failure from an unavailable aggregated API that prevents finalization cluster-wide.

for a senior

Show the triage order: conditions, leftover objects, APIService availability. Fix the owner before removing any finalizer, and check what each finalizer protected outside the cluster.

for a principal

Treat stuck namespaces as a sign of bad add-on lifecycle: define uninstall ordering and alerting on unavailable APIService objects so tenants' deletions never hang.

## What "stuck in `Terminating`" means A Kubernetes **Namespace** that is being deleted has phase `Terminating` and keeps the value `kubernetes` in its `spec.finalizers` list. That entry belongs to the **namespace controller** inside `kube-controller-manager`, which deletes every object in the namespace and removes the entry only when the namespace is empty. A namespace is **stuck** when that controller can never reach "empty": it keeps retrying, and the namespace never goes away. Suppose `recsys-inference`, which ran a recommendation-model inference server on a 3-node kubeadm cluster, has been `Terminating` for 19 minutes — longer than its pods' grace period. Time to find out why. ## Step 1: read the namespace's own `status.conditions` The namespace controller writes the reason into the namespace's `status.conditions`. Read them before anything else: ```bash kubectl get namespace recsys-inference -o jsonpath='{range .status.conditions[*]}{.type}{"\t"}{.status}{"\t"}{.message}{"\n"}{end}' ``` A condition with status `True` means that problem is present: | Condition type | Reason when `True` | What it tells you | |---|---|---| | `NamespaceDeletionDiscoveryFailure` | `DiscoveryFailed` | The controller could not list every API group; the message names the failing group/version | | `NamespaceDeletionGroupVersionParsingFailure` | `GroupVersionParsingFailed` | A served group/version string could not be parsed | | `NamespaceDeletionContentFailure` | `ContentDeletionFailed` | Delete calls for some resource types returned errors | | `NamespaceContentRemaining` | `SomeResourcesRemain` | Objects still exist; the message counts them per resource type | | `NamespaceFinalizersRemaining` | `SomeFinalizersRemain` | Remaining objects carry finalizers; the message names each finalizer and how many objects hold it | Two patterns account for most stuck namespaces. ## Cause 1: objects held by their own finalizers The controller has already sent a delete for every object, but an object with entries in `metadata.finalizers` is only removed after whoever owns each entry clears it. If the component that owns the finalizer is gone — typically an operator that was uninstalled before its custom resources were deleted — nobody ever clears it. The conditions show `NamespaceContentRemaining` and `NamespaceFinalizersRemaining`, naming the resource type and the finalizer string. To list what is left, enumerate every listable namespaced type: ```bash kubectl api-resources --verbs=list --namespaced -o name \ | xargs -n 1 kubectl get --show-kind --ignore-not-found -n recsys-inference ``` **Fix, in order of preference:** 1. Bring the owning controller back (reinstall the operator) so it finishes its cleanup and clears its own finalizer. 2. If that cleanup is genuinely no longer needed, remove the finalizer from that **specific object** after checking what it was protecting (an external load balancer, a storage volume, a database). Either way, the namespace controller notices on its next retry and finishes. ## Cause 2: an unavailable aggregated API An **aggregated API** is an API group that the API server proxies to a separate backend, registered through an `APIService` object. When that backend is down, discovery for its group fails. The namespace controller treats a partial discovery as non-fatal and still deletes everything it can see, but it will not remove its finalizer while discovery is incomplete: an unlistable group might hold objects in the namespace. Symptoms: - The condition `NamespaceDeletionDiscoveryFailure` is `True`, and its message names the failing group, for example `metrics.k8s.io/v1beta1`. - `NamespaceContentRemaining` may be absent: the namespace can look empty and still not go away. - `kubectl get apiservices` shows that group with `AVAILABLE` set to `False`. - **Every** namespace deleted while the API is unavailable gets stuck, not just this one, because discovery is cluster-wide. **Fix:** restore the backend that serves the group. If the add-on was removed on purpose and its `APIService` was left behind, delete that stale `APIService`. Once discovery succeeds, the pending namespaces finish without further action. ## What not to do first Emptying `spec.finalizers` by force through the namespace's `/finalize` subresource makes the namespace disappear, but it skips the purge: it treats the symptom and can leave objects and external resources behind. Diagnose with the conditions first, because the conditions usually name the exact resource type, finalizer or API group at fault, and fixing that also unblocks any other namespace waiting on the same cause. ## Summary checklist - Read `status.conditions` on the namespace; they usually name the culprit. - Finalizers remaining: find the objects, restore their controller or deliberately remove that object's finalizer. - Discovery failure: check `kubectl get apiservices`, restore or remove the unavailable one. - Re-read the conditions to confirm the purge completed.

  • Why can one unavailable aggregated API stop the deletion of a namespace that never used that API?
    The namespace controller must know every namespaced resource type to prove a namespace is empty. If discovery fails for one group, it cannot list that group's objects, so it records `NamespaceDeletionDiscoveryFailure` and keeps the `kubernetes` finalizer. Because discovery is cluster-wide, every namespace being deleted waits, whether or not it ever held those objects.
  • The namespace's conditions name a finalizer on 3 remaining custom resources, and their operator was uninstalled last week. What do you do?
    Find out what that finalizer protects — often an external resource such as a volume, DNS record or load balancer. Preferably reinstall the operator long enough for it to clean up and clear its finalizers. If the external side is already gone or will be cleaned by hand, patch the finalizer off those 3 objects. The namespace controller then finishes on its next retry.
  • How do you prevent namespaces from getting stuck like this when removing an add-on?
    Delete the add-on's custom resources, and wait for them to go, while its controller is still running. Then uninstall the controller, and delete its `APIService` in the same change that removes its backend. Order matters: the controller must outlive the objects it finalizes, and an `APIService` must never outlive its backend.

saying these in an interview costs you the question

  • A stuck namespace is always fixed by emptying spec.finalizers
  • Namespace deletion only waits on the pods inside it
  • An empty-looking namespace cannot be stuck on anything
  • A broken aggregated API only affects namespaces that used its resources
  • kubectl get all shows every object left in a namespace
  • Restarting kube-controller-manager clears a stuck namespace
open as a page

In Kubernetes, what happens when you run kubectl delete namespace, and why does the namespace show Terminating before it disappears?

level: juniorimportance: should knowfreq 58%

basics

~10 s

Deleting a Kubernetes namespace only marks it Terminating. The namespace controller deletes every object inside, pods first, then removes the kubernetes finalizer, and only then does the API server remove the namespace.

open as a page

A common fix for a Kubernetes namespace stuck in Terminating is emptying spec.finalizers through the /finalize subresource; what does that actually do, and when is it acceptable?

level: seniorimportance: should knowfreq 44%

basics

~20 s

Emptying a namespace's spec.finalizers through /finalize tells the API server the purge is done, so it deletes the namespace. Remaining objects stay in etcd and their finalizers never run. Use it only after diagnosing the cause and accepting the leftovers.

open as a page