Kyverno background scanning is churning PolicyReport objects and loading etcd. How do you cut the cost?

## What background scanning actually does Admission only ever sees what someone submits. Objects that were created before a policy existed are never re-examined by it, and nobody resubmits a two-year-old Deployment. Background scanning closes that hole: a controller periodically lists the resources that policies match and evaluates them, writing the outcomes into `PolicyReport` objects (namespaced) and `ClusterPolicyReport` objects (for cluster-scoped resources). Nobody is blocked; the outcome is a record. That is exactly the population you care about for a rule like "this field is deprecated and disappears at the next API version". Nothing new is being created with it. The interesting instances are already stored, and only a scan will ever find them. A report is an ordinary API object with a `summary` of counts and a `results` list, one entry per rule per matched resource. Which is where the cost comes from. ## Where the load comes from Think of it as **matching resources x rules x rescans**, with four multipliers people miss: 1. **Match breadth.** A policy matching Pods in all namespaces matches every Pod in the cluster. Add ten such rules and every scan produces ten results per Pod. 2. **Replica fan-out.** One misconfigured Deployment with fifty replicas is fifty Pod-level results. Matching the controller instead gives you one — and one that a human can actually fix, since editing a Pod is pointless. 3. **Churn.** Namespaces full of short-lived objects — CI runs, Jobs, autoscaled workloads — generate a continuous stream of report creations and deletions on top of the scheduled passes. This is often the real source of a paged operator's write storm, not the interval. 4. **Rewrites.** Every rescan rewrites report content. Every write goes through the API server into etcd and fans out to every watcher of that resource type. Etcd cares about write volume and object size, not about whether the writes were interesting. A cluster whose reports are mostly unchanged from pass to pass still pays for the pass. ## Diagnosing it Before turning knobs, establish which of the multipliers is dominant: - Count report objects and their sizes, broken down by namespace. A handful of namespaces usually dominates. - Look at API-server request rate by resource and verb. If report writes dwarf everything else, you have confirmation. - Check etcd database size and how compaction and defragmentation are keeping up. - Look at the controller's own CPU and memory — a controller being throttled produces stale reports as well as load. The answer to "is it the interval or the match?" is almost always the match, and tuning the interval first is the classic wasted change. ## The levers, roughly in order of payoff **Narrow what matches.** Name specific kinds instead of wildcards. Use a namespace selector so the policy applies to the namespaces that matter. Exclude system namespaces you do not govern. Match the workload controller rather than the Pods it creates. **Turn background evaluation off where it makes no sense.** A policy has a background setting; when it is off, the rule decides only at admission and never appears in a scan. Any rule that reads the requesting user or the operation *must* be in that category, because a stored object carries no request — there is no user, no operation and no old object behind it. Rules guarding something meaningful only at creation time belong there too. **Filter at the door.** The Kyverno configuration carries resource filters that drop kinds and namespaces before the controllers process them at all. This is the blunt instrument for high-churn, low-value objects. **Lengthen the interval.** Real, but it buys less than the match changes and it costs freshness: the reports then describe the cluster as of the last pass, so anything driven off them lags by up to one interval. **Right-size the controller.** Give the reporting side its own resource budget so it is not competing with admission handling, whose latency is on the critical path of every request. **Stop using etcd as your results database.** If you want ninety days of history, trend lines, or a view across clusters, ship the findings to a store built for that and keep the in-cluster copy as a short-lived working set. ## The tradeoff to state out loud Every lever trades evidence for cost. A narrower match means you have no findings about what you excluded. A longer interval means your picture is older. Excluding a namespace means you cannot answer questions about it. That is fine as long as it is a decision rather than an accident — but say which population you are now blind to, because the next person to ask "does this rule hold across the cluster?" deserves an honest answer.