Before adopting an in-cluster policy engine, what do you require of its upgrade story and who owns it?

## Why the upgrade story is an adoption criterion An in-cluster policy engine is usually evaluated on what its rules can express. The chair that matters after adoption is different: someone on the platform team will be paged for this component and will move it through minor versions every quarter for years. From that chair the engine is infrastructure sitting in the path of API writes, and its properties as a running service outrank its properties as a language. The reason is structural. The engine is wired in by a webhook configuration, and if that configuration is set to fail closed, then while the engine's pods are unavailable, matching API requests are rejected. An engine that is down is therefore not merely "not enforcing" — depending on how it is wired, it can be the reason a deploy, a controller reconcile or an autoscaling event fails. That is why the upgrade is a change-managed event and not a routine bump. ## What to require before you install it **A pinned version and a known cadence.** You want to choose when to move, and you want to know roughly how often you will have to. An engine that ships breaking changes unpredictably makes the quarterly upgrade a research project each time. **Documented changes to the policy resources.** The engine's custom resource definitions are the API your rules are written against. An upgrade that changes their schema, tightens validation or deprecates a field is a change to every rule you own. You want release notes that state this plainly, and you want to know whether existing stored policies remain valid after the CRDs are updated. **A rehearsal that does not need a cluster.** This is where the criteria compose: because rules can be evaluated offline, you can run the entire rule suite against the **new** engine version before it goes anywhere near production, and see whether any verdict changed. An engine without offline evaluation cannot be rehearsed this way, so its upgrades are validated in a live cluster or not at all — that is the compounding cost of failing the first criterion. **A staged rollout.** Non-production cluster first, then production, with a soak in between long enough for the periodic background scan to run and for the normal deploy traffic of a working day to pass through. **A back-out an on-call engineer can execute.** Written down, tested, and short: how to relax or remove the webhook configuration so the API server stops depending on the engine, what that means (the cluster is unenforced until it is restored), and how to reconcile afterwards. The background report is what closes that loop: after enforcement is restored you scan to find whatever was created while the gate was off. ## The ownership question The more interesting half of the question is organisational, and it has a common failure shape: the security team authors and ships the rules, the platform team runs the engine, and the platform team's pager fires because of a rule they did not write and cannot immediately interpret. Adoption is the moment to settle this rather than discovering it during an incident. Concretely, write down: - **Who approves a new rule**, and against what bar — including whether it lands in a reporting-only state before it enforces anything. - **Who approves an engine upgrade**, and what evidence they expect (the rule suite re-run against the new version). - **Who may disable the engine during an incident**, without needing to find the rule's author at 3am. This authority has to sit with on-call. - **Who is accountable for the rules that exist**, because rules accumulate and the platform team should not inherit an unowned pile of them. ## The tradeoff a lead is expected to own This is where the expressiveness argument gets resolved honestly. A more expressive engine can encode rules the simpler one cannot, and that is a real benefit. Against it: an engine nobody on the on-call rotation can debug at 3am is a worse engine for that organisation regardless of what it can express, because the failure mode is not "a rule was hard to write" but "nobody could tell whether the engine or the workload was the problem". Where the split is severe, the usual resolution is not to pick expressiveness but to invest in the operability side — a runbook, a training session, dashboards for the engine's own health, and rules whose match blocks are narrow enough that the engine sees a fraction of the traffic. The related call is how many engines to run. Two engines double the on-call surface, the upgrade calendar and the number of places a denial could have come from. There are reasons to do it — a capability one has and the other lacks — but it should be a decision with a named owner and a plan to converge, not an accumulation. ## What a strong answer sounds like It names the team before it names the product, treats the upgrade as a rehearsed change with a back-out, ties the rehearsal to offline rule evaluation and the recovery to the background report, and states the decision rights between rule authors and engine operators explicitly.