A node upgrade is stuck: the drain has been retrying for an hour with "Cannot evict pod as it would violate the pod's disruption budget". Walk through your diagnosis and the options for unblocking it.

## The message is a symptom, not a cause `Cannot evict pod as it would violate the pod's disruption budget` means only that `disruptionsAllowed` was 0 for a PDB matching that pod. Five distinct root causes produce it, and they need different fixes. ## Step 1 — locate the budget ``` kubectl get pdb -A kubectl get pdb -n prod api -o yaml ``` Read `status`: `currentHealthy`, `desiredHealthy`, `expectedPods`, `disruptionsAllowed`. The gap between the first two tells you immediately whether you have a *budget arithmetic* problem or a *pod health* problem. ## Cause A — the budget can never be satisfied `maxUnavailable: 0`, `minAvailable: 100%`, or `minAvailable: N` where the Deployment runs exactly N replicas. Here `currentHealthy == desiredHealthy` with every pod perfectly healthy. Nothing will change on its own; waiting is futile. Fix: either scale the workload up so there is headroom (`kubectl scale deploy/api --replicas=N+2`, the least invasive option because it *adds* availability rather than removing protection), or correct the budget to `maxUnavailable: 1`. Scaling up is usually the right emergency action: it unblocks the drain while honoring the intent of the budget. ## Cause B — pods are not Ready Only pods whose `Ready` condition is true count toward `currentHealthy`. A workload with 6 replicas where 3 are CrashLooping has `currentHealthy: 3` against `desiredHealthy: 5` — the budget is *already* violated, so every eviction is refused. Ironically the pods blocking maintenance are the ones serving nothing. Check with `kubectl get pods -l <selector> -o wide` and look at the READY column, then at events and probe configuration. Sometimes readiness fails because a dependency is down, not because the pod is broken. Fix: repair the workload. Structurally, set `spec.unhealthyPodEvictionPolicy: AlwaysAllow` (GA in Kubernetes 1.31) on the PDB so running-but-unready pods may be evicted even when the budget is exhausted. That is the purpose-built escape from this deadlock and belongs in your default PDB template. ## Cause C — selector problems PDB selectors match labels, independent of ownership: - A selector like `app: web` may cover `web-canary` and `web-stable` at once, so one Deployment's health consumes the other's budget. - Two PDBs matching the same pod cause the eviction to be **refused outright**, regardless of either budget's numbers. `kubectl get pdb -A -o json` plus a check of which selectors match the blocking pod's labels finds this. - A PDB whose selector matches nothing has `expectedPods: 0`; harmless for drains but a sign the labels drifted, and it means a workload someone believes is protected is not. ## Cause D — no capacity for replacements An eviction only restores budget when a replacement pod becomes Ready — and the replacement must be **scheduled somewhere else**. If the cluster is full, or the pods have node affinity / topology spread / a specific zone requirement that only the draining node satisfied, replacements stay `Pending`, the budget never recovers, and the drain retries forever. Symptom: `disruptionsAllowed` pinned at 0 *and* pods in Pending with `FailedScheduling` events. The fix is capacity, not the PDB. ## Cause E — it is working, just slowly With `maxUnavailable: 1` and a 2-minute readiness warm-up, a node with 25 pods of that workload takes ~50 minutes. `disruptionsAllowed` cycling between 0 and 1 and a falling pod count on the node prove progress. Distinguish this before doing anything destructive. ## Ordered playbook 1. `kubectl get pdb -A` → which budget is at zero. 2. Are the matching pods Ready? If not, that is the cause. 3. Is `desiredHealthy == expectedPods`? Then the budget is unsatisfiable by construction. 4. Are replacement pods Pending? Then it is a capacity/scheduling problem. 5. Otherwise watch for two minutes — it may be pacing normally. Actions in increasing order of risk: **scale the workload up** → **fix readiness** → **amend the PDB** (raise maxUnavailable / add `AlwaysAllow`) → **temporarily delete the PDB** → **`--disable-eviction` or delete pods directly**. The last two discard the protection entirely and should be a conscious, time-boxed decision with the service owner, not a reflex. ## Preventing the recurrence Validate at admission that no PDB is unsatisfiable (`maxUnavailable: 0`, or `minAvailable` ≥ the workload's `minReplicas`), require budgets to be expressed relative to the fleet, standardize `unhealthyPodEvictionPolicy: AlwaysAllow`, and alert on any PDB sitting at `disruptionsAllowed: 0` for longer than a few minutes — that alert fires long before an upgrade window, which is exactly when you want to hear about it.