Your Kyverno verifyImages rule rewrote a tag to a digest and GitOps now reports permanent drift — why?

## Verification has a side effect Most policy rules answer yes or no. A `verifyImages` rule does something extra: after it verifies an image it *changes the object it admitted*. The field `mutateDigest` controls this and defaults to true, so unless you turned it off, an image written as `registry.example.com/apps/api:1.4.2` is admitted as `registry.example.com/apps/api:1.4.2@sha256:...`, with the digest the engine resolved while doing the verification. The reason is straightforward: the engine verified a specific set of bytes, and pinning the admitted reference to that digest is how the decision stays attached to the thing it was made about. But the side effect is that the object in the cluster is no longer the object you applied — and that is exactly the invariant a GitOps controller is built to defend. ## Why the drift is permanent A GitOps controller reconciles by comparing the desired state — the manifests in the repository — against the live state of the resources it manages. Its whole model assumes that applying the desired state makes live equal to desired. Admission-time mutation breaks that assumption for one field: - Git says `image: registry.example.com/apps/api:1.4.2`. - The cluster holds `image: registry.example.com/apps/api:1.4.2@sha256:...`. - The diff is non-empty, so the app is reported out of sync. - Reapplying does not fix it, because the reapplied object goes through admission again and gets the digest appended again. If the controller is configured to self-heal, this becomes a loop: it detects drift, applies the tag form, admission mutates it back, it detects drift again. The cost is not just noise on a dashboard — a permanently out-of-sync application trains the team to ignore the sync status, and the next time drift means something real, nobody looks. ## The three fixes, and what each costs **Ignore the field.** Configure the GitOps controller to exclude the image field of the managed resources from diffing. Cheap and effective, but it is a targeted blindfold: from then on, that controller will not tell you if the image in the cluster differs from the image in Git for any reason, mutation or otherwise. Scope it as narrowly as the tool allows. **Put the digest in Git.** Have the release process resolve the tag to a digest and commit the pinned reference, so the manifest already says what the cluster will hold and the mutation is a no-op. This is the cleanest end state: desired and live agree, the reference in Git is unambiguous, and the change history records exactly which bytes were deployed. It costs you an automated step in the release path and a repository that now churns on every image bump — and the humans reading those pull requests can no longer tell at a glance what changed, which is a real reviewability cost. **Turn mutation off.** Setting `mutateDigest: false` makes the drift disappear because nothing is rewritten. What you give up is the link between the reference that was verified and the reference the node resolves when it pulls, which is the whole reason the mutation exists. ## Second-order effects worth naming The rewrite changes more than a diff. Anything reading the live spec now sees a two-part reference: dashboards and scripts that parse the image string, alerting that groups by image, a human running `kubectl get` to see what version is deployed. A digest is not human-legible, so "which release is this?" gets harder to answer from the cluster alone — the tag is still present in the mutated form, which helps, but tooling that split on `:` will find more than it expected. It also affects how a rollback reads. The object that was admitted carries a digest, so re-applying an old manifest by tag produces a fresh resolution rather than the bytes that ran last time. If reproducing exactly what ran is a requirement, that argues again for digests in Git. ## Answering it well The question is really testing whether you know that a decision engine here is also a mutating one, and whether you can reason about the interaction between two systems that each believe they own the object. Name the default, explain the diff mechanically, then give the three options with their costs and say which you would pick — usually digests in Git for the services where reproducibility matters, and a narrowly scoped diff exclusion everywhere else while you get there.