How does a Kyverno mutate rule patch PersistentVolumeClaims that already exist in the cluster?
answer
- the rule points at something else
- not in the admission path
- who is allowed to write it
- targets plus the background controller
- sweeping on policy update
basics
~20 sBy declaring mutate.targets, which points the patch at objects other than the one being admitted. Kyverno applies it from its background controller rather than in the admission request, so the change is asynchronous and needs RBAC on the target kind.
solid answer
~50 sAn ordinary mutate rule only touches the object in the admission request, so it can never reach a PVC that was created last month. Adding a `targets` list to the `mutate` block turns it into a mutate-existing rule: the rule is still triggered by a matched event, but the patch is applied to the resources the targets select — by kind, namespace and name, which may be computed from variables. Kyverno applies those patches from its **background controller**, outside the admission path, and that changes three things. The user who triggered it sees no error and no added latency. The controller needs RBAC to get and update the target kind, or nothing happens. And failures surface only as events and policy report entries, so you have to go looking. Setting `mutateExistingOnPolicyUpdate` additionally re-applies the rule when the policy itself changes, which is how you sweep the estate that already exists.
go deeper
Know the boundary: an ordinary mutate rule only sees the object in the current request, so existing resources need a different mechanism. Recognising that limit is enough at this level.
Explain how targets redirect the patch to other objects, that the work happens in a background controller rather than in the admission response, and what that means for error visibility.
Diagnose the silent-failure case end to end: RBAC on the target kind, controller logs, events and reports, plus the blast radius of a target selector with no name.
Own the policy question of rewriting other teams' live objects: what you sweep, what you only default at admission, how you announce it, and how you keep the change reversible.
## The problem You want every PersistentVolumeClaim to carry a `backup-class` annotation that your backup operator reads. A normal Kyverno mutate rule stamps it onto PVCs at admission — but the cluster already holds hundreds of PVCs created before the policy existed, and those hold the data that actually matters. Admission-time mutation is, by construction, blind to them: it only ever sees the object in the current request. ## The mechanism: mutate.targets Kyverno's answer is a `targets` list inside the `mutate` block. The rule still has a `match` block that says *what triggers it*, but the patch is applied to whatever the targets select: ``` mutate: targets: - apiVersion: v1 kind: PersistentVolumeClaim namespace: "{{ request.object.metadata.namespace }}" patchStrategicMerge: metadata: annotations: +(backup-class): gold ``` Targets can be pinned to a name or left open to a whole kind in a namespace, and the fields accept Kyverno variables, so the trigger can select its own targets. The patch dialects are the same ones you use at admission: a strategic merge with anchors, or a JSON 6902 patch. ## Where it runs, and why that is the whole answer This is the part interviewers are testing. A mutate-existing rule does **not** run inside the admission request. Kyverno's background controller performs it after the fact. Four consequences follow, and a good answer names them without prompting. **1. It is asynchronous and unreportable to the requester.** Nobody's `kubectl apply` fails because the patch failed. There is no admission response to carry an error. From the caller's side the operation simply succeeded; the target may or may not have been patched a moment later. **2. It needs permissions the admission path never needed.** Reviewing an object in an AdmissionReview is not the same as writing a different object through the API. The background controller must be granted get, list, update and watch on the target kind; Kyverno's controller roles are extended by adding aggregated ClusterRoles carrying the right label. The single most common reason a mutate-existing rule "does nothing" is that this permission was never granted, and the failure is visible only in the controller's logs and events. **3. It is a write against live objects, not a rewrite in flight.** You are editing something that is running. Some fields on a live object are immutable, and the API server will reject the patch. Annotations and labels are safe; a PVC's storage class is not. Choose targets and fields that can legally change after creation. **4. Timing is event-driven.** The rule fires when its trigger occurs. If you also want it to sweep everything the moment you install or edit the policy, that is what the `mutateExistingOnPolicyUpdate` setting is for: it re-applies the rule against matching targets on policy create or update. Without it, a PVC created before the policy and never touched again may simply never be visited. ## Operating it Because the change lands out-of-band, the operational discipline is different from an admission rule: - **Test the blast radius first.** A target selector that names a kind and a namespace but no name will patch every object of that kind in that namespace. Model it before you apply the policy. - **Watch the reports.** Rule outcomes land in Kyverno's policy reports and in Kubernetes events on the affected resources; that, plus the controller logs, is your only feedback channel. - **Expect drift complaints.** Objects that a person or a GitOps controller owns now differ from their source. The mutated object carries the `policies.kyverno.io/last-applied-patches` annotation naming the policy and rule, and that annotation is what turns a confusing diff into a five-second answer. - **Prefer add-if-not-present on the patch.** A sweep that overwrites a value some team deliberately set is far harder to defend than one that fills gaps, and it re-runs harmlessly. ## The distinction to state clearly Admission-time mutation and mutate-existing are the same rule vocabulary pointed at two different moments: one rewrites a request before it becomes an object, the other rewrites objects that already are. The first is synchronous, permission-free in the sense that the API server already authorised the request, and fails visibly. The second is asynchronous, needs its own RBAC, and fails quietly. Saying that out loud is most of the answer.
- The rule is installed, the policy is valid, and no PVC changes. What do you check first?Whether Kyverno's background controller has RBAC to get, list and update PersistentVolumeClaims — its roles are extended with aggregated ClusterRoles, and without that grant the patch simply never lands. Check the controller logs and events on the target namespace next. Because this runs outside admission, nothing surfaces to the person who triggered it, so silence is the expected symptom of a permission problem.
- Why can a mutate-existing rule not report a failure to the user the way an admission rule does?There is no admission request to respond to. The patch is applied afterwards by a controller reconciling in the background, and the client's call has already returned success. The only feedback surfaces are the policy report entry for the rule, Kubernetes events on the affected object, and the controller's logs — all of which someone has to actively look at.
- What does mutateExistingOnPolicyUpdate change?It makes the rule re-apply when the policy itself is created or updated, rather than waiting for a matching trigger event. That is how you reach objects that already exist and may never be touched again — the sweep you need when you introduce a new annotation to a cluster that has been running for a year. It also means editing the policy re-patches the estate, so the blast radius is worth modelling first.
- Are there fields you should refuse to patch on an existing object?Yes — anything the API server treats as immutable after creation, and anything whose change restarts or disrupts a workload. A PVC's storage class or size behaves very differently from an annotation. Sweeps are safest when confined to metadata that other controllers read; if the change would force a rollout, that is a scheduled migration, not a policy patch.
saying these in an interview costs you the question
- Thinks mutate rules can only ever touch the admitted object
- Forgets the background controller needs its own RBAC
- Expects a failed background patch to fail someone's kubectl apply
- Assumes installing the policy automatically sweeps existing objects
- Targets an immutable field on a live resource