A PersistentVolumeClaim has a backup annotation nobody wrote. How do you confirm Kyverno added it?
answer
- the object carries a receipt
- an annotation, not the kubectl one
- cross-check events and reports
- replay the policy offline
- last-applied-patches names the rule
basics
~20 sRead the object's policies.kyverno.io/last-applied-patches annotation, which records the patch and the policy and rule that applied it. Corroborate with events on the resource, the policy report entry, and a Kyverno CLI replay of the policy.
solid answer
~40 sStart on the object itself: Kyverno annotates a resource it mutated with `policies.kyverno.io/last-applied-patches`, and that value names the rule and the policy responsible along with the change it made. That converts "where did this come from" into "go ask the owner of this policy". Corroborate it three ways: `kubectl describe` shows Kyverno's events on the resource, the namespace's policy report carries a result entry for the rule, and the admission controller's logs cover the request. Then reproduce it offline — feed your original manifest and the policy to the Kyverno CLI's `apply` command and see the same field appear, which proves it deterministically without touching the cluster. Note that `kubectl.kubernetes.io/last-applied-configuration` will *not* show the field: that annotation records the manifest the client submitted, before any mutation.
go deeper
Know that a field can appear on a live object without anyone editing it, because an admission policy rewrote the request, and that the object usually carries an annotation saying so.
Explain the attribution surfaces and how they differ: an annotation on the object, events, policy report entries, and the client-side annotation that records only what you submitted.
Run the trace efficiently and prove it — reproduce the mutation offline against the original manifest, then reason about whether the rule filled a gap or overrode a deliberate value.
Treat traceability as a design requirement of every mutation you ship, and decide what evidence of past mutations your platform keeps once the object itself is gone.
## The situation You own a workload. You diff the live PersistentVolumeClaim against the manifest in your repository and there is an annotation you never typed — something like a backup class the platform's backup operator reads. Nothing failed, nothing warned you, and your `kubectl apply` returned success. This is the ordinary experience of being on the receiving end of a mutation policy, and being able to trace it in a couple of minutes is a genuinely useful skill. ## Step 1: the object carries its own receipt Kyverno records what it changed on the mutated object, in the annotation `policies.kyverno.io/last-applied-patches`. The value names the rule, the policy it belongs to, and the patch that was applied. That is the fastest possible answer: you now have a policy name, which means you have an owner to talk to and a rule to read. A trap worth knowing: `kubectl.kubernetes.io/last-applied-configuration` is *not* this. That annotation is written by client-side `kubectl apply` and records the configuration **you** submitted. The mutated field will be absent from it — which is exactly why the diff looked mysterious in the first place, and why seeing your own manifest echoed there is not evidence that nothing changed it. ## Step 2: corroborate off the object Three independent surfaces should agree: - **Events.** `kubectl describe` on the resource shows events Kyverno emits when a rule applies, naming the policy and rule. - **Policy reports.** Kyverno records rule results as report resources in the namespace (and cluster-scoped ones for cluster-scoped objects). A report entry gives you the same attribution in a form you can query across many objects at once — which is how you answer "how many of my PVCs did this touch", not just "this one". - **Admission controller logs.** The last resort when the object has since been replaced, or when you suspect a rule ran and produced no change. The reason to use more than one is that the annotation lives on the object and dies with it. If the object was recreated by a controller, or if a later mutation replaced the annotation, the report and the events are what survive. ## Step 3: reproduce it deterministically The convincing step is offline replay. The Kyverno CLI can apply a policy to a manifest and print the resulting resource: ``` kyverno apply policy.yaml --resource pvc.yaml ``` If your original manifest goes in and the annotated PVC comes out, you have proved the attribution without touching the cluster, and you now have a reproduction you can hand to the policy's owner or paste into a ticket. It is also the loop the policy author should have run before shipping, which is worth saying if you are the one being asked. ## Step 4: decide what you actually want Attribution ends the mystery; it does not end the conversation. Once you know which rule did it, the useful questions are: was this meant to be a default that yields to my value, or an override? Does my manifest set the field explicitly, and if so did the rule replace it or fill a gap? And is the annotation something my deployment tooling will now fight over on the next reconcile, because the live object no longer matches the source? A policy author reading this from the other chair should take one lesson from it: **the mutation must be traceable from the object outward**. A rule that silently rewrites a field and leaves no name attached to the change is the reason mutation policies get a bad reputation. Name your rules for what they do, keep the patch small enough that the recorded diff is readable, and prefer add-if-not-present when the field is a default rather than a mandate. ## What a weak answer looks like Guessing at admission controllers in general, proposing to read every policy in the cluster by hand, or reaching straight for controller logs before looking at the object. The object knows who touched it; start there.
- Why does kubectl.kubernetes.io/last-applied-configuration not show the mutated field?Because it records the manifest the client sent. Client-side `kubectl apply` writes that annotation from your local file as part of the request, so it captures your intent before any admission-time mutation rewrote the object. Comparing it to the live object is actually a decent way to *spot* a mutation, but it will never tell you who performed one.
- The annotation is gone because a controller recreated the object. Where else can you look?The policy report entries for that namespace, which record rule results independently of the object, and Kubernetes events on the resource while they are still within their retention window. Failing both, the Kyverno admission controller logs cover the request itself. This is the argument for treating reports as the durable evidence surface rather than the on-object annotation.
- As the policy author, what would you change so this trace is never needed?Make the rule self-describing: a rule name that states the change, a small patch so the recorded diff is readable, and an add-if-not-present anchor when the field is a default rather than a mandate. Announce the policy to the teams whose objects it touches before it ships, and give them the CLI reproduction so they can see what it will do to their own manifests.
The mutated object is a parcel with a sticker from whoever opened it in transit; the shipping label you printed yourself will never mention them.
saying these in an interview costs you the question
- Reads last-applied-configuration and concludes nothing mutated the object
- Goes to controller logs before reading the object's annotations
- Cannot name any attribution surface at all
- Assumes the mutation was a manual edit by another engineer
- Believes the object always records every historical mutation