After Kubernetes nodes are drained, every pod creation fails with 'failed calling webhook', including the webhook's own pods. Why does this deadlock happen, and how do you break it?
answer
- admission runs before storage
- failurePolicy Fail with no backend
- replacement pods hit the same webhook
- webhook configs are exempt from webhooks
- exclude own namespace, then restore
basics
~20 sA webhook with failurePolicy Fail intercepts pod creation, and its backend has no running pods, so every create is rejected, including creates of its replacement pods. Break the loop by patching the webhook configuration to Ignore or narrowing its scope.
solid answer
~40 sThe API server calls every matching admission webhook before it stores an object. If a webhook's `failurePolicy` is `Fail` and its Service has no ready backends, every matching request is rejected with `failed calling webhook "<name>"`. When the webhook's own pods were evicted during a drain, their ReplicaSet has to create new pods, and those creates match the same webhook, so they are rejected too. Nothing recovers without someone stepping in. You see it as `FailedCreate` events on ReplicaSets and a `ReplicaFailure` condition. The way out works because Kubernetes never sends requests for ValidatingWebhookConfiguration or MutatingWebhookConfiguration objects to webhooks. You can therefore always patch the configuration: set `failurePolicy: Ignore`, or add a `namespaceSelector` that excludes the webhook's own namespace. Let the backend start, then put the configuration back.
code
bash · 5 lineskubectl get validatingwebhookconfiguration ledger-policy -o yaml > ledger-policy.bak.yaml
kubectl patch validatingwebhookconfiguration ledger-policy --type=json \
-p '[{"op":"replace","path":"/webhooks/0/failurePolicy","value":"Ignore"}]'
kubectl -n policy-system rollout status deployment/ledger-policy-webhook
kubectl replace -f ledger-policy.bak.yamlgo deeper
Remember that the API server calls admission webhooks before saving an object, and a webhook that cannot be reached can block writes.
Explain why failurePolicy Fail plus a webhook with no ready backend pods rejects the creation of its own replacement pods.
Show the recovery: recognise FailedCreate events, use the fact that webhook configurations are exempt, fail open or narrow the scope, then restore and verify.
Frame the tradeoff between enforcement and availability, and require exclusions, disruption budgets and a rehearsed break-glass patch for every cluster-wide webhook.
## The incident During the node-by-node upgrade of a 48-node cluster, the platform team drains the workers. A cluster-wide validating webhook checks every Pod creation. Its two replicas happen to be on the first two nodes drained. A few minutes later nothing new starts anywhere: the nightly ledger-reconciliation CronJob fails to create its pod, Deployments stop scaling, and even the webhook's own Deployment cannot bring its pods back. The API server is healthy and kubectl answers. The failure is in **admission**, not in the control plane's availability. ## Why it becomes a deadlock 1. The API server receives a Pod create and finds a matching webhook in a **ValidatingWebhookConfiguration** (or a MutatingWebhookConfiguration). 2. It calls the webhook's Service. There are no ready endpoints, so the call fails, or waits until the webhook's `timeoutSeconds` (10 seconds by default). 3. The webhook's `failurePolicy` is `Fail`, so the API server **rejects the request** instead of skipping the webhook. 4. The webhook's own ReplicaSet tries to create replacement pods. Those creates also match the webhook, so they are rejected as well. 5. Nothing changes until someone acts. The ReplicaSet controller keeps retrying with backoff. ## How to recognise it | Signal | Where | |---|---| | `FailedCreate` events mentioning `failed calling webhook "<name>"` | `kubectl describe replicaset` or `kubectl get events` | | A `ReplicaFailure` condition | Deployment or ReplicaSet status | | The webhook's Service has no ready endpoints | Its EndpointSlices | | `kubectl run` fails with the same message | Any namespace the webhook matches | | Pod creates are slow before they fail | The webhook timeout is being reached | This is different from `admission webhook "<name>" denied the request`. That message means the webhook **ran** and said no; that is a policy decision, not an outage. ## Breaking the loop Kubernetes deliberately **exempts webhook configuration objects from webhooks**. Requests for ValidatingWebhookConfiguration and MutatingWebhookConfiguration objects (and the admission-policy kinds) are never sent to admission webhooks. A cluster administrator can therefore always change them, even while every webhook is failing. Options, from least to most drastic: - **Narrow the scope.** Add a `namespaceSelector` (or `objectSelector`) that excludes the webhook's own namespace, so its pods can start while enforcement continues elsewhere. - **Fail open for now.** Patch `failurePolicy` to `Ignore`. This removes enforcement for everything the webhook covers until you restore it. - **Remove the configuration** temporarily, after saving it with `kubectl get -o yaml`. Once the backend's pods are Ready, restore the original configuration and confirm the blocked controllers catch up. ## Side effects to watch - **Static pods are not blocked.** The kubelet runs them from files, so the control plane keeps running. Their **mirror pods** may fail to appear in the API while the webhook is failing. - **Every write the webhook covers is blocked**, not only pods. A webhook that matches Deployments or ConfigMaps can stop the GitOps sync of the very fix you are pushing. - **Namespaces are not exempt by default.** A webhook matching all namespaces also intercepts `kube-system` unless its selector excludes it. ## Preventing a repeat Why `Fail` versus `Ignore` is chosen, and how to roll webhooks out safely, belongs to the webhook-authoring topic. From the troubleshooting side, the useful habits are: - keep an exclusion for the webhook's own namespace and for system namespaces; - give the backend enough replicas, a PodDisruptionBudget and spread across nodes so a drain cannot remove all of it; - write down the patch command in the runbook before you need it.
- How do you tell a failing Kubernetes webhook from one that is working and rejecting requests on purpose?Read the message. 'failed calling webhook "<name>"' means the API server could not get an answer: no endpoints, a timeout, or a TLS failure. 'admission webhook "<name>" denied the request' means the webhook ran and refused the object, which is a policy result. Only the first calls for incident action; the second needs the object or the policy changed.
- Why can the Kubernetes control plane keep running while this deadlock blocks all pod creation?On kubeadm clusters the control-plane components are static pods started by the kubelet from files on disk, so they never go through admission. Only their mirror pods, the API copies, go through admission, and those may fail to appear. The API server, scheduler and controllers keep running, which is why the fix can be applied through kubectl.
saying these in an interview costs you the question
- Webhook configuration objects are themselves blocked by a failing webhook
- The fix is restarting kube-apiserver so it forgets the webhook
- A failing webhook only affects the namespace it is deployed in
- kube-system is exempt from admission webhooks by default
- Denied-the-request and failed-calling-webhook errors mean the same thing