Why must a Kubernetes mutating admission webhook's patch be idempotent under reinvocation?

## Why a webhook sees the same object twice Mutating admission webhooks run one after another, and each one is handed the output of the ones before it. That creates an obvious problem: a webhook that ran early made its decision without seeing a change a later webhook made. Kubernetes answers this with **reinvocation**. Each webhook entry carries `reinvocationPolicy`, which is `Never` by default. Set it to `IfNeeded`, and the API server may call that webhook again within the same admission request if a subsequent mutating webhook modified the object. The practical consequence is the whole point of this question: **your webhook can be handed an object it has already patched**. The safe assumption is not "it runs once", nor "it runs at most twice" - it is that the number of invocations is not yours to know, so the patch must be written to converge. ## What idempotent means for a patch A patch is idempotent when applying it to its own output produces no further change: f(f(x)) equals f(x). Sorted by that property: | Patch shape | Idempotent? | Why | |---|---|---| | Set a scalar field to a fixed value | Yes | The second application finds the value already there | | Add a map key if absent | Yes, if you check | Annotations and labels are keyed, so re-adding overwrites rather than duplicates | | Append an element to a list | **No** | Lists have no identity check; each call appends again | | Append keyed by name, skipping if present | Yes | The name is the identity you test before writing | | Write a freshly generated value each call | **No** | The object changes on every invocation, so it never settles | Setting `automountServiceAccountToken: false` on a Pod spec is the easy case. The field is a scalar; the second run reads `false`, has nothing to do, and returns an empty patch. Now take a rule that injects a volume, or a sidecar container, or an extra environment entry. Written as "append this element", it appends on every invocation. What you get is two containers with the same name - rejected outright - or two volumes, two args, two env entries, and a failure that looks nothing like a policy bug. Written as "if no element named `x` exists, add one", it is safe by construction and safe on UPDATE too. ## Reinvocation is not the only path to a second look Even with `reinvocationPolicy: Never`, your rule runs again on every UPDATE to the object, and by then the object already carries your earlier patch. A blind append is therefore a slow leak across an object's lifetime, not only a within-request hazard. `Never` only promises that the API server will not call you twice inside a single admission request. ## The other mutator Reinvocation exists because independently registered mutating configurations touch the same objects, and that raises the second failure mode. If another mutator sets the field you just set, the value that persists is the one written last. You do not get to choose the order in which independently registered configurations run, and nothing in the API server detects the conflict or reports it - there is no error, just a stored object whose value is not the one you returned. Two consequences follow. First, **verify from the stored object, not from your own patch**: read what is actually persisted for a sample of workloads rather than assuming the rule stuck. Second, **resolve contested fields by ownership, not by racing**. Turning on `IfNeeded` so you can re-assert a value that someone else keeps overwriting produces a fight in the cluster's request path; agreeing which policy owns which field is the fix. Reinvocation is there so an earlier webhook can react to a later one's change - it is not a mechanism for winning arguments. ## Writing the rule so this never bites - Express every patch as a desired end state, not as an action: "this field is false", "an element named `x` exists", never "append". - Never derive a patched value from something that changes per call - a timestamp, a random identifier, a counter. - Read the object you were sent, including your own prior work; do not reconstruct the original. - Test the rule by feeding it its own output. If the second pass returns a non-empty patch, the rule is not idempotent, and that test catches nearly every instance of this bug before it reaches a cluster.