skip to content

You diff a running Pod against the manifest you committed and find fields your team never wrote. What happened?

level: seniorimportance: nice to knowfreq 38%

answer

  1. the cluster copy is not the source
  2. something rewrote it on the way in
  3. a server-side dry-run shows it first
  4. the repository still carries the flaw
  5. another cluster, no policy, live again

basics

~20 s

An admission mutation patched the object before it was stored, so the cluster is compliant and the committed file is not. Confirm it with a server-side dry-run, then fix the manifest - the live object is not your source of truth.

solid answer

~50 s

A mutating admission policy rewrote the object on the way in, so what is stored differs from what you applied. Confirm it rather than guessing: `kubectl diff` and `kubectl apply --dry-run=server` both send the request through the admission chain, so the object they show already carries the patch, and a well-run platform stamps an annotation naming the rule that fired. An audit entry recorded at a level that includes the response body shows it too. Then fix the file. The manifest is what gets reviewed, promoted to the next cluster, and reused as a template - and where the policy is absent or scoped differently, the flaw is live again. Leaving it also means a reconciler comparing your repository against the cluster reports the same drift forever, until someone silences it with an ignore rule that then hides real drift.

code

yaml · 11 lines
yaml
# kubectl get pod api-7c9f -o yaml   (live, trimmed)
metadata:
  annotations:
    policy.example.com/mutated: "default-automount"   # left by the rule that patched it
spec:
  automountServiceAccountToken: false   # not present in the committed manifest
  containers:
    - name: api
      image: registry.example.com/api:1.4.2
      ...
...

go deeper

for a junior

Know that what runs in the cluster can differ from the YAML you committed, because admission control may rewrite an object on its way to being stored.

for a middle

Be able to find the change: a server-side dry-run, an annotation the rule leaves behind, or an audit entry that captures the object as persisted.

for a senior

Explain why the live fix is not enough - promotion to a cluster without the policy, a reviewer approving the unsafe shape, and a reconciler reporting drift forever.

for a principal

Argue for making mutations legible across the organisation: every patched field traceable to a named rule and reported back to its author, not quietly applied.

## What you are looking at Admission control sits between an authorized request and the write. A mutating policy returns a patch, the API server applies it, and **the patched object is what gets stored**. Nothing rejects your request, nothing appears in your terminal, and the object you get back on the next read carries fields you never typed. From the developer's chair this reads as a colleague editing the cluster behind your back; from the platform's chair it is the guardrail working exactly as designed. ## Confirming what did it, rather than guessing Three things to reach for, cheapest first. - **A server-side dry-run.** `kubectl apply --dry-run=server` and `kubectl diff` send the real request through the real admission chain and return the object as it *would* be persisted, without persisting it. That is how you see the mutation on a manifest before it reaches the cluster, and how you prove which of your fields survive. - **An annotation on the object.** A convention worth insisting on from your platform team: every mutating rule stamps an annotation naming itself. Without it, a patched field is indistinguishable from a field a teammate added. - **The audit log.** An API server audit entry recorded at a level that captures the response body shows the object as stored, which pins the change to a specific request rather than to a suspicion. What will not tell you: the kubelet's logs, the object's `resourceVersion` (an opaque store-level counter, not a mutation count), or reading the webhook configuration - its match rules tell you which requests are intercepted, never what the patch contains. ## Why fixing the cluster is not fixing anything The temptation is to shrug: the running object is compliant, the mutation re-applies on every create and update, so nothing is broken. Four things are. 1. **The artifact is still wrong.** The manifest is what a reviewer reads, what a template gets copied from, and what a new service starts as. Every review of it approves the unsafe shape, because in practice the shape works. 2. **Portability.** Apply the same file to a cluster without that policy, or into a namespace its scoping does not match, and the flaw is live with nothing to catch it. The safety lived in the cluster, not the artifact, and artifacts travel. 3. **Permanent drift.** A reconciler comparing the repository against the live cluster sees a field it did not write. It applies the manifest, admission patches it again, and the diff returns. Teams paper over it with a rule that ignores differences on that path - and now a genuine unauthorized change to the same field is invisible too. 4. **Nobody learned.** The next manifest from the same team has the same shape, because the feedback never reached a human. So the fix is in the file: write the field the policy would have written, commit it, and the mutation becomes a no-op for that workload. Do not hand-edit the live object to match the repository - the patch is re-applied on the next write anyway. ## Judgment: if you disagree with the mutation Sometimes the patched field is wrong for your workload. The move is not to fight it - there is no way to win a race against an engine in the request path - but to take it to whoever owns the rule and ask for an exemption through whatever path they provide. If the patch changes how your application behaves rather than filling in a default it had no view on, that is a strong argument that the rule should have denied and explained rather than repaired silently, and it is worth making. ## What a senior candidate says that a middle one does not A middle answer identifies the mutation and finds it. The senior answer adds the cost of the divergence and closes it: fix the source, ask the platform team to make the patch legible so the next person does not spend an afternoon on it, and treat the persisted object as evidence of what the cluster does - never as the record of what your team intended.

  • How do you see what a policy will do to your manifest before applying it?
    Use a server-side dry-run - `kubectl apply --dry-run=server` or `kubectl diff`. Both send the request through the real admission chain and return the object as it would be stored, without storing it. A client-side dry-run never reaches the API server, so it shows nothing about admission.
  • A reconciler reports permanent drift on the field a policy mutates. What is the right fix?
    Write the compliant value into the manifest so the policy's patch becomes a no-op and the diff disappears. Configuring the reconciler to ignore that path is the tempting shortcut, but it also blinds you to a genuine unauthorized change on the same field.
  • What should the platform team change so this costs the next developer nothing?
    Make the mutation legible: stamp an annotation naming the rule on every patched object, and route the fact that a patch fired back to the person who submitted it. A guardrail that quietly repairs and never reports trains teams to keep writing the shape it repairs.

saying these in an interview costs you the question

  • Hand-edits the live object to match the committed file
  • Concludes the cluster is fine so the manifest needs no change
  • Assumes a teammate edited the object rather than a policy
  • Silences the reconciler's drift report instead of fixing the source
  • Thinks a client-side dry-run reveals admission mutations

context