Why would a 7-replica Kubernetes Deployment deleted with kubectl delete --cascade=foreground still exist with a deletionTimestamp twenty minutes later, and how do you unblock it safely?

## What `deletionTimestamp` really means In Kubernetes, a delete request does not always remove an object at once. If the object has entries in `metadata.finalizers`, the API server only **marks** it: it sets `metadata.deletionTimestamp` and leaves it in storage. A **finalizer** is a string key that some component promised to remove once its cleanup is done. The object leaves storage only when the list is empty. Three rules follow, and they shape every fix: - `deletionTimestamp` **cannot be unset**. A marked object is going away. There is no undo. - Once it is set, finalizers can **only be removed**. The API server rejects an update that adds a new one. - Anything that keeps a finalizer in place keeps the object alive. `kubectl delete` waits by default (`--wait` is true), so the command itself appears to hang. ## Why a foreground delete gets stuck Take the 7-replica feature-flag evaluation Deployment, `flag-eval`, on a 12-node GPU model-serving cluster, deleted with `kubectl delete deployment flag-eval --cascade=foreground`. That request sends `propagationPolicy: Foreground`: 1. The Deployment gets a `deletionTimestamp` and the **`foregroundDeletion`** finalizer. 2. The garbage collector in `kube-controller-manager` deletes the ReplicaSet with Foreground, so the ReplicaSet also gets `foregroundDeletion`. 3. The garbage collector deletes the 7 Pods. Each Pod's ownerReference has `blockOwnerDeletion: true`, so the ReplicaSet waits for every one of them. 4. Only when the last Pod is gone do the finalizers come off, first on the ReplicaSet and then on the Deployment. Twenty minutes later the Deployment still exists, so something **below** it has not finished. The usual causes: | Blocker | What you see | Why it blocks | |---|---|---| | Pod on an unreachable node | Pod `Terminating`, its node `NotReady` | the kubelet never confirms the containers stopped, so the Pod is not removed | | Pod with a custom finalizer | `metadata.finalizers` lists a key some operator owns | that operator is down or failing, so it never removes its key | | Garbage collector not making progress | nothing has a new `deletionTimestamp` | check the garbage collector's errors in the `kube-controller-manager` logs | ## Diagnosing top-down Follow the chain rather than guessing: 1. Read the owner: `kubectl get deployment flag-eval -o jsonpath='{.metadata.deletionTimestamp} {.metadata.finalizers}'`. Seeing `foregroundDeletion` confirms it is waiting on dependents. 2. List what still exists under it, with each object's deletion state, finalizers and node: ```bash kubectl get rs,pods -l app=flag-eval -o custom-columns=NAME:.metadata.name,DELETING:.metadata.deletionTimestamp,FINALIZERS:.metadata.finalizers,NODE:.spec.nodeName kubectl get nodes ``` 3. For a dependent that is still there, compare its `metadata.ownerReferences[].uid` with the owner's UID, so you know it really belongs to this chain. 4. For a Pod with a foreign finalizer, find the component that owns that key and check why it is not reconciling. ## Unblocking safely Fix the **lowest** blocker, not the top of the chain. - **Unreachable node.** Bring the node back, or remove its Node object if the machine is really gone. Once the Node is gone, the control plane cleans up Pods still bound to it. As a last resort, `kubectl delete pod <name> --grace-period=0 --force` removes the Pod from the API without waiting for the kubelet. The container **may still be running** on the partitioned machine, still holding its GPU and still answering requests if it can reach its clients. - **Custom finalizer.** Restore the owning operator so it finishes its cleanup and removes its key. Stripping the key by hand, for example with `kubectl patch pod <name> --type=merge -p '{"metadata":{"finalizers":null}}'`, **skips that cleanup**. Whatever the finalizer protected, such as an external registration or a lease, is leaked and must be cleaned up by hand. - **Force deletion is not finalizer removal.** `--force --grace-period=0` skips graceful termination only. A Pod whose finalizer list is not empty stays in storage anyway. Removing `foregroundDeletion` from the **Deployment** makes it vanish at once, but it only hides the problem. Its remaining dependents now point at a missing owner and are still garbage collected in the background, and the stuck Pod stays stuck for the same reason as before. ## Preventing the next one - Alert on objects whose `deletionTimestamp` is older than a few minutes. - Treat every operator that adds finalizers as a dependency of deletion: if it is down, deletion stops. - Prefer Background for routine cleanup. Use Foreground only when a caller must wait for the children to be gone.