A Kyverno rule stopped blocking after its ConfigMap was renamed - how would you have caught it?

## Why this failure is quiet A Kyverno rule that reads an allow list from a ConfigMap has a runtime dependency nothing validated up front. The policy was accepted when it was created - its structure is fine. The ConfigMap is resolved per admission request. Rename it, move it to another namespace, or narrow the RBAC that lets Kyverno read it, and from that moment the rule cannot evaluate. It does not start blocking everything and it does not start allowing loudly. It produces an error for that rule and the request continues down the pipeline. Meanwhile the security dashboard everyone actually looks at counts violations. Violations go to zero. Zero violations is what success looks like. ## The signal that distinguishes the two Kyverno writes per-resource, per-rule outcomes into policy reports, and the result values are distinct on purpose: | result | what it means | | --- | --- | | `pass` | the rule evaluated and the resource satisfied it | | `fail` | the rule evaluated and the resource violated it | | `warn` | reported without being treated as a violation | | `error` | the rule could not be evaluated at all | | `skip` | the rule did not apply - typically an unmet precondition | Those last two are the ones that matter here, and they are the ones nobody alerts on. Practical monitoring: - **Alert on `error`.** Any sustained error rate for a policy is an outage of that control, whatever the admission outcome was. - **Alert on the absence of `pass`.** A policy that normally evaluates every Pod creation and suddenly evaluates none has stopped working, even if nothing errored - for example because a `match` block or a precondition was narrowed by a well-meaning edit. - **Watch `skip` as a ratio.** Skips are legitimate; a step change in the skip ratio is not. - **Read the events and controller logs.** Kyverno emits events for policy execution problems, and the controller logs name the entry that failed, which is what turns "a control is down" into "the allowed-placement ConfigMap is gone". The admission controller also exposes Prometheus metrics for policy results labelled by policy, rule and outcome, which is usually the cheapest place to hang the alerts. ## Prove it positively Monitoring tells you when something changed. A canary tells you the control still works. Keep a manifest that is deliberately non-compliant - a Pod targeting a zone that is definitely not on the list - and submit it on a schedule as a server-side dry run from a CI job or a small controller. If it is admitted, page. This is the single highest-value check for any guardrail whose enforcement depends on data it fetches, because it exercises the whole path: match, context lookup, precondition, condition, decision. ## Why CLI tests did not save you The Kyverno CLI can test a policy against fixture resources and lets you supply context values from a values file. That is exactly what makes it fast, and exactly why it is blind to this failure: the ConfigMap the test used came from a file, so the test stays green while the cluster's real ConfigMap does not exist. CLI tests verify the *logic* of the rule. They cannot verify its *dependencies*. Both need covering, in different places. ## Make the dependency explicit The durable fix is structural rather than observational: - Ship the ConfigMap and the policy in the same GitOps application so they are created, updated and pruned together, and a rename is one commit that touches both. - Keep the ConfigMap in a namespace owned by the platform team, not in the namespaces the policy governs, so an ordinary namespace admin cannot rename or delete it in the course of tidying up. - Guard it with a Kyverno policy of its own if it matters enough: a rule that refuses deletion or an update from an unexpected subject. - Record, in the policy's own repository, that the rule has an external dependency. The next person to read the YAML sees a `context` block; they do not see that a ConfigMap in another namespace is load-bearing. ## What to say in the interview The strong answer names the specific asymmetry: a control that fails by not firing produces the same telemetry as a control that is passing. Everything else - error alerts, canaries, GitOps coupling - follows from taking that seriously. The weak answer is "we would see it in the policy reports", which is true only if someone is looking at the column nobody looks at.