skip to content

Kubernetes runs mutating admission controllers before validating ones. Why is that ordering necessary, and what must the author of a mutating admission webhook keep in mind because of it?

level: seniorimportance: should knowfreq 38%

answer

  1. mutate → schema-validate → validate → persist
  2. validators must see the final object (else injection defeats policy)
  3. order among mutating webhooks unspecified; reinvocationPolicy: IfNeeded
  4. reinvocation ⇒ patches must be idempotent (double sidecar)
  5. sideEffects / dryRun; JSONPatch escaping ~1 for /

basics

~20 s

Mutation must finish before validation so that validators judge the final object — otherwise a webhook could inject a sidecar that a validator already approved the spec without. Mutating webhook authors must therefore assume other webhooks also mutate, run in unspecified order, may be re-invoked, and must produce a schema-valid object with idempotent patches and no side effects on dry-run.

solid answer

~60 s

The pipeline is: **mutating** plugins and webhooks → object schema validation → **validating** plugins, ValidatingAdmissionPolicies and webhooks → persist. Validation last is a security requirement: if a mutating webhook could run after validation, it could inject a privileged sidecar, a hostPath mount, or a different image *after* the policy said yes. Validators must see exactly what will be stored. For a mutating webhook author this means: - **Order among mutating webhooks is not guaranteed**, so never assume your patch sees the final object. If your patch depends on another's output, use `reinvocationPolicy: IfNeeded` — your webhook may then be called again after others mutate, so patches must be **idempotent** (injecting a sidecar twice is a classic bug; check whether it is already there). - Your output must still be **schema-valid** — the API server re-validates after mutation. - Declare `sideEffects: None` and honour `dryRun: true` by never touching external state, otherwise `kubectl --dry-run=server` causes real effects. - Your patch will face validating policies afterwards, so an injected sidecar must itself satisfy them.

code

yaml · 25 lines
yaml
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: sidecar-injector
webhooks:
  - name: inject.mesh.example.com
    admissionReviewVersions: ["v1"]
    sideEffects: None
    failurePolicy: Ignore
    timeoutSeconds: 3
    reinvocationPolicy: IfNeeded
    objectSelector:
      matchLabels:
        mesh.example.com/inject: "true"
    rules:
      - apiGroups:   [""]
        apiVersions: ["v1"]
        operations:  ["CREATE"]
        resources:   ["pods"]
    clientConfig:
      service:
        namespace: mesh-system
        name: injector
        path: /mutate
      caBundle: <base64-ca>

go deeper

for a junior

Know that mutating admission can change the object, validating admission can only accept or reject, and mutation happens first.

for a middle

Explain why validators must see the final object, and that the API server re-validates the schema after mutation.

for a senior

Cover unspecified ordering, reinvocationPolicy and the idempotency requirement, sideEffects/dryRun contracts, JSON Patch pitfalls, and the latency each webhook adds to every write.

for a principal

Weigh mutation as an architectural choice: implicit object rewriting versus explicit configuration, the availability cost of webhooks on the write path, and preferring in-tree defaulting or CEL where it suffices.

## The two sub-phases Admission is not one step. For any write the API server: 1. Runs **mutating** admission — built-in plugins first (things like `ServiceAccount`, which assigns the default ServiceAccount and mounts its token, `DefaultStorageClass`, `DefaultTolerationSeconds`), then registered mutating webhooks. Each may return a JSON Patch that modifies the object. 2. **Validates the object against the API schema**, so a webhook cannot smuggle in a structurally invalid object. 3. Runs **validating** admission — built-in plugins, ValidatingAdmissionPolicies (in-process CEL), and validating webhooks. These may only accept or reject; `ResourceQuota` runs here because it must count what will actually be stored. 4. Persists to etcd. ## Why the order is a security property, not a convenience Suppose validation ran first. A validating policy checks the pod: no privileged containers, approved registry, resource limits present — approved. A mutating webhook then runs and injects a container with `privileged: true` and a hostPath mount of `/`. The stored object violates the policy that just approved it, and no component ever saw the final state. Every policy in the cluster would be advisory. Running all mutation first, then validating the final object, closes that. It is the same principle as validating after deserialisation rather than before: **check the thing you are actually going to keep**. A second, subtler benefit: defaulting is mutation. Because built-in defaults are applied first, validators can write simple rules against fully-defaulted objects rather than repeating "or unset" everywhere. ## What this forces on mutating webhook authors ### Ordering is unspecified The API server does not guarantee an order among mutating webhooks — treat it as arbitrary. If your webhook adds a label and another webhook's decision depends on that label, you have a race in configuration, not in code. ### Reinvocation and idempotency Because of that, a webhook may set `reinvocationPolicy: IfNeeded`, telling the API server to call it again if the object was modified by a later webhook in the same pass. That is how a sidecar injector ends up seeing the fully-mutated pod. The price: **your webhook can be invoked more than once for the same request**, so every patch must be idempotent. The canonical bug is a service-mesh injector that appends its sidecar unconditionally and produces two identical containers — the pod is then rejected for duplicate container names, or worse, starts with two proxies. Always check for the marker (an annotation, or a container of that name) before patching. ### Your output is re-validated The post-mutation schema check means a malformed patch surfaces as an error on the write, often attributed confusingly to the user. Beyond schema, your injected content still faces validating policies: inject a container without resource limits into a cluster that requires them and every pod in that namespace starts failing, with the message pointing at the policy rather than at your injector. ### Side effects and dry-run Every webhook declares `sideEffects`. `None` means it changes no external state; `NoneOnDryRun` means it does, but honours the `dryRun: true` flag in the AdmissionReview. This matters because `kubectl apply --dry-run=server` and server-side quota simulation run the admission chain for real. A webhook that provisions an external resource during a dry run creates orphans, and one that ignores `dryRun` breaks tooling. Declaring `None` and then having side effects is a correctness bug the API server trusts you not to commit. ### Patch mechanics Webhooks return a base64-encoded JSON Patch with `patchType: JSONPatch`. Two practical traps: paths must escape `/` and `~` in keys (annotation keys contain slashes — `/metadata/annotations/example.com~1injected`), and adding to an array that may be absent requires creating the array first, since `add` to `/spec/containers/-` fails if `containers` does not exist. ### Cost Every mutating webhook adds a synchronous network round trip to every matching write, and reinvocation can double it. Combined with `failurePolicy`, mutating webhooks are the admission feature most likely to hurt cluster availability, which is why in-tree defaulting or CEL-based approaches are preferred when they suffice. ## Where the built-in plugins sit It is worth knowing that some controllers are both: `NamespaceLifecycle`, `LimitRanger` (mutating — applies defaults — and validating), `PodSecurity` (validating, implementing Pod Security Standards), `ResourceQuota` (validating, last). The enabled plugin list is an API-server flag, so "is that plugin on?" is a legitimate diagnostic question when a cluster behaves unexpectedly.

  • A service-mesh injector occasionally produces pods with two identical sidecar containers. What is the likely cause?
    The webhook sets reinvocationPolicy: IfNeeded and appends the sidecar unconditionally. When another mutating webhook modifies the pod after it runs, the API server calls the injector a second time and it appends the container again. The fix is idempotency: check for a marker annotation or for an existing container of that name and return an empty patch if it is already present.
  • Why does a mutating webhook that ignores the dryRun flag break kubectl apply --dry-run=server?
    Server-side dry run executes the full admission chain but discards the result instead of persisting. A webhook that provisions external state — registering an endpoint, allocating an address, writing to another system — during that pass creates real, orphaned side effects for an operation the user expected to change nothing. That is why each webhook declares sideEffects, and why anything with side effects must declare NoneOnDryRun and skip them when request.dryRun is true.

Assembly line then inspection: you cannot let anyone bolt on extra parts after the inspector has signed off, or the certificate describes a machine that never existed.

saying these in an interview costs you the question

  • Claiming validating webhooks can also modify the object.
  • Assuming mutating webhooks run in the order they are registered or alphabetically.
  • Writing non-idempotent patches while enabling reinvocationPolicy: IfNeeded.
  • Declaring sideEffects: None on a webhook that provisions external resources.
  • Forgetting that injected containers must themselves pass the cluster's validating policies (limits, non-root, approved registry).

context

resolving links…