In Kubernetes, a request that creates an object can be intercepted by webhooks registered through both a MutatingWebhookConfiguration and a ValidatingWebhookConfiguration. In what order does the API server run them relative to each other and to the rest of its request handling, and why does that order matter?
answer
- authn → authz → mutate → schema → validate → etcd
- mutating serial, validating parallel
- validators must see the final object
- JSONPatch back from mutating only
- reinvocationPolicy: IfNeeded → be idempotent
basics
~20 sThe API server authenticates and authorizes, then runs mutating admission (webhooks may patch the object), then schema validation, then validating admission (accept or reject only), then writes to etcd. Mutation runs first so validators judge the final stored object.
solid answer
~50 sThe pipeline is: authentication, authorization, **mutating admission**, object schema validation, **validating admission**, then persistence to etcd. Mutating webhooks are called **serially** — one webhook's output is the next one's input — and reply with a base64-encoded JSON Patch that the API server applies. Validating webhooks are called **in parallel** and may only return allowed true/false plus a message or warnings; they cannot change the object. The ordering is deliberate: a policy check must see what will actually be stored. If a sidecar-injection webhook adds a container, a validating webhook enforcing "every container declares resource limits" has to run afterwards, otherwise it would approve an object that no longer complies once mutated. A direct consequence is that a mutating webhook may be invoked more than once for a single request when `reinvocationPolicy: IfNeeded` is set and a later webhook modified the object, so mutating logic must be idempotent.
code
yaml · 27 linesapiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
name: sidecar-injector
webhooks:
- name: inject.example.com
admissionReviewVersions: ["v1"]
sideEffects: None
failurePolicy: Ignore
timeoutSeconds: 5
reinvocationPolicy: IfNeeded
rules:
- operations: ["CREATE"]
apiGroups: [""]
apiVersions: ["v1"]
resources: ["pods"]
scope: "Namespaced"
namespaceSelector:
matchLabels:
inject: "enabled"
clientConfig:
service:
name: injector
namespace: platform
path: /mutate
port: 443
caBundle: LS0tLS1CRUdJTi...go deeper
Know the two phases by name and that mutating runs before validating, and that only mutating webhooks can change the object.
Explain serial mutation versus parallel validation, the JSON Patch response format, and give the sidecar-injection-then-policy-check example for why the order matters.
Add reinvocationPolicy and idempotency, ordering between configurations being undefined, and the design rule that defaulting belongs in mutating webhooks while enforcement belongs in validating ones.
Frame it as an extension-point contract: where to put policy so it cannot be bypassed, when to prefer in-process CEL policies over webhooks, and how phase ordering interacts with tenant-owned mutating webhooks in a shared cluster.
## Where admission sits in the request path Every write to the Kubernetes API server follows a fixed pipeline. The request is authenticated (who are you), authorized (may you do this), and then enters **admission control** — a chain of plugins compiled into the API server plus two plugins that call out over HTTPS to your own code: `MutatingAdmissionWebhook` and `ValidatingAdmissionWebhook`. Only after admission passes is the object serialized into etcd. Admission itself has two phases in a fixed order: 1. **Mutating phase** — plugins and mutating webhooks may change the incoming object. 2. **Object validation** — the API server validates the result against the built-in type schema (or, for custom resources, the CRD's OpenAPI schema) and rejects malformed objects. 3. **Validating phase** — plugins and validating webhooks may only accept or reject. So `MutatingWebhookConfiguration` always runs before `ValidatingWebhookConfiguration`. Nothing you configure changes that relationship. ## What each configuration object registers Both are cluster-scoped objects in `admissionregistration.k8s.io/v1`. Each contains a list of webhooks, and each webhook declares: - **rules** — which API groups, versions, resources and operations (CREATE, UPDATE, DELETE, CONNECT) trigger it; - **clientConfig** — either a `service` reference inside the cluster or an external `url`, plus a `caBundle` the API server uses to trust the TLS certificate; - **failurePolicy**, **timeoutSeconds**, **sideEffects**, **matchPolicy**, and selectors that narrow the match. The API server POSTs an `AdmissionReview` object containing the request's object (and, for updates, the old object, the user info and the dry-run flag). The webhook replies with an `AdmissionReview` response carrying the same `uid`, `allowed: true|false`, and — for mutating webhooks only — `patchType: JSONPatch` with a base64-encoded RFC 6902 patch. ## Serial mutation, parallel validation Mutating webhooks are invoked **one at a time** because each one's patch changes the object the next one sees. Within a single `MutatingWebhookConfiguration` the webhooks run in the order listed; across separate configurations the ordering is not something you should design around. If you need webhook A to observe webhook B's changes, set `reinvocationPolicy: IfNeeded` on A: the API server will call A again if any later webhook modified the object. Reinvocation means your webhook can see its own previous output, so patches must be idempotent — "add a sidecar if absent", never "append a sidecar". Validating webhooks are invoked **concurrently**, which keeps total latency near the slowest single webhook rather than the sum. Because they cannot mutate, order among them is irrelevant: a single rejection rejects the request. They may also return `warnings`, which kubectl prints without failing the call. ## Why the ordering is the interesting part The classic example is a service-mesh injector plus a policy engine. The injector mutates a Pod to add a proxy container; the policy engine then validates that every container has a non-root user and resource limits. Because mutation precedes validation, the injected container is subject to the same policy — which is what you want, and also a common source of surprise when the injected sidecar itself violates policy and the whole Pod is rejected. The inverse case matters too: a validating webhook can never repair an object, so "reject unless the label is set" and "add the label if missing" are two different design choices with different user experience. Defaulting belongs in mutating webhooks; enforcement belongs in validating webhooks. Mixing them — a mutating webhook that returns `allowed: false` for policy reasons — technically works but hides rejections in a phase where operators do not look for them. Also note what is *not* re-run: after validating admission, nothing else can change the object, so what a validating webhook approves is what lands in etcd (quota accounting aside, which is itself a built-in validating plugin). ## In-process alternatives Since Kubernetes 1.30 the built-in `ValidatingAdmissionPolicy` evaluates CEL expressions inside the API server, in the validating phase, with no webhook server, no TLS and no network hop. A mutating equivalent (`MutatingAdmissionPolicy`) followed later. They occupy the same phases in the same order, so the mental model above still applies — they simply remove the availability and latency risk of an external call for rules simple enough to express in CEL.
- Can one mutating webhook be called twice for the same request, and what does that imply for how you write it?Yes. If the webhook sets `reinvocationPolicy: IfNeeded`, the API server calls it again when any later mutating webhook changed the object. The webhook therefore sees objects it has already patched, so every patch must be idempotent — check whether the container, label or annotation is already present before adding it. Blindly appending produces duplicate sidecars.
- Two mutating webhooks patch the same field. Which one wins, and how would you design around that?Within a single MutatingWebhookConfiguration the webhooks run in the listed order, so the later one overwrites the earlier. Across separate configurations you should not depend on ordering. The robust design is to make the webhooks target disjoint fields, or to collapse conflicting logic into one webhook, rather than relying on invocation order.
- Can a validating webhook modify the object it is reviewing?No. Its response only carries `allowed`, an optional `status` with a message and reason, and optional `warnings`. Any `patch` field is ignored in the validating phase. If you need to change the object you must register a mutating webhook, which runs earlier.
Think of a manuscript passing through copy-editors and then a legal reviewer: the editors mark up the text one after another, and legal reads the finished draft — it would be pointless for legal to sign off on a version the editors are still rewriting.
saying these in an interview costs you the question
- Saying validating webhooks run first, or that the order is configurable
- Claiming validating webhooks can patch or default the object
- Assuming a mutating webhook is called exactly once per request, so appending a sidecar unconditionally is safe
- Thinking mutating webhooks run in parallel — they are serial precisely because each sees the previous patch
- Believing the API server merges a returned full object; it applies a base64-encoded JSON Patch