skip to content

Why insist that a policy engine can evaluate one rule against a manifest file without a cluster?

level: juniorimportance: should knowfreq 58%

answer

  1. how long is one edit cycle
  2. cluster as the only test harness
  3. same rule text, second decision point
  4. raw file versus admitted object

basics

~20 s

A rule you can only exercise inside a live cluster can only be tested by deploying it. Offline evaluation runs the same rule text against a saved manifest in seconds, in a unit test and in CI.

solid answer

~50 s

Offline evaluation is the difference between a feedback loop measured in seconds and one measured in deploys. If the only way to know what a rule does is to install it in a cluster and try to create an object, then every edit costs a deploy, rules get written by trial and error in a shared cluster, and nobody dares refactor one. An engine that ships a CLI or library taking `policy + resource file -> verdict` lets me keep a test file next to each rule: this PVC naming an unapproved storage class must be denied, this one naming the encrypted class must pass. The same property buys a second decision point for free — the identical rule text can run in CI over rendered Helm or Kustomize output, so an unencrypted storage class is caught pre-merge instead of at `kubectl apply`. Gatekeeper's `gator` and the Kyverno CLI both exist for exactly this reason.

code

yaml · 12 lines
yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: orders-data
  namespace: orders
spec:
  storageClassName: gp2-unencrypted
  accessModes: [ReadWriteOnce]
  resources:
    requests:
      storage: 20Gi
# ... second fixture: same claim naming an approved encrypted class, expected to pass

go deeper

for a junior

Be ready to say plainly that a rule which can only run inside a cluster can only be tested by deploying it, and that engines ship command-line evaluators so you can run one rule against a saved manifest instead.

for a middle

Explain the mechanics: one policy artifact, two execution contexts — admission in-cluster and a file-based evaluation in CI over rendered chart output — and why maintaining a second copy of the rule for testing defeats the point.

for a senior

Show the limit as well as the benefit. Offline evaluation sees a raw file, not the object after defaulting and mutation, and it exercises no wiring, so you still keep one end-to-end check that the policy is selecting and reaching the objects you think it does.

for a principal

Frame it as an adoption criterion with a cost attached: an engine without offline evaluation buys expressiveness at the price of a slow, nervous change process for rules, and that cost is paid forever by whoever maintains them.

## The criterion When a platform team picks an in-cluster policy engine, one of the first questions to ask is not "what can it express?" but "can I run one of its rules against a file, on my laptop, with no cluster?" That single property changes how the whole apparatus feels to maintain. ## Why the feedback loop dominates A policy rule is code, and code you cannot run cheaply is code nobody improves. If the only execution environment is a live cluster, then the loop for every edit is: change the rule, apply it, try to create an object that should be denied, read the rejection message, repeat. Even with a throwaway cluster spun up per pull request that is minutes per iteration. The observable consequences are consistent: - Rules get written by trial and error in a shared cluster, which means someone else's `kubectl apply` gets blocked by a half-finished rule. - Nobody refactors an existing rule, because the cost of proving you did not change its behaviour is a deploy. - Edge cases go untested. The rule handles the manifest the author had open and nothing else. With offline evaluation the loop is a subsecond command or a unit test. That is what makes it reasonable to keep a small set of saved manifests beside each rule and re-run them on every change. ## What "offline" concretely means The engine must expose its evaluator as something you can call outside the cluster — a CLI or a library that takes the policy artifact plus one or more resource files and prints a verdict. Gatekeeper ships `gator`; Kyverno ships a CLI that applies policies to resource files; Kubewarden policies are WebAssembly modules that `kwctl` can run locally. The important detail when you evaluate an engine is whether the offline evaluator consumes **the same policy artifact the cluster runs**, not a translated or simplified copy. If you have to maintain a second representation of the rule for testing, you have two rules that will drift, and the test proves nothing about the one that is enforcing. ## The second decision point The same property is what lets one rule text serve two gates. Take the rule this leaf carries: dynamically provisioned volumes must use an approved, encrypted storage class. In-cluster it runs at admission and rejects a `PersistentVolumeClaim` naming `gp2-unencrypted`. Offline, the identical rule runs in CI against rendered chart output, so the author sees the failure in a pull request instead of discovering it when the deploy fails. Developers strongly prefer the earlier gate, and the platform team gets one rule to maintain rather than a policy plus a lookalike lint check that slowly disagrees with it. ## What offline evaluation does not prove This is the part candidates miss, and it is the honest limit of the criterion. Offline you are feeding the engine a **raw file**. At admission the engine is handed the object as the API server has it, which may already differ: - Fields may have been defaulted. A PVC with no `storageClassName` at all is a very different input from one naming a class, and your file-based test will not exercise it unless you write that case deliberately. - Mutating policies run before validating ones, so the object your validating rule sees may have been changed by something else in the cluster. - None of the wiring is exercised: whether the policy's match rules and namespace selectors actually select the object, whether the engine is running and reachable, whether RBAC lets it see what it needs. So offline evaluation tests rule *logic*, and you still want one end-to-end check in a real cluster confirming the rule is wired up and reachable. The two are complements, not substitutes. ## Applying the criterion in an evaluation When an engine is on the table, the concrete things to check are: does it ship an offline evaluator at all; does that evaluator take the exact artifact the cluster consumes; can it evaluate a single rule rather than requiring the whole policy set; does it report the same denial message the cluster would return; and can it be run from a plain CI job without cluster credentials. An engine that fails those tests is not unusable — it is just an engine whose rules will be maintained slowly and nervously, and that cost lands on whoever is on call for it.

  • What can an offline pass get wrong compared with what happens at admission?
    Offline you feed the engine a raw file; at admission it sees the object as the API server has it. Fields may have been defaulted — a claim with no `storageClassName` at all is a case your file-based test never sees unless you write it — and mutating policies run before validating ones, so another policy may have changed the object first. Offline tests rule logic; keep one live check that the rule is wired up.
  • Does an offline test suite remove the need to test the deployed policy?
    No. It proves the rule reaches the right verdict on a given object. It proves nothing about whether the policy's match rules and namespace selectors actually select that object, whether the engine is running and reachable, or whether its service account can see what it needs. One end-to-end smoke test per policy covers the wiring.
  • Why does it matter that the offline evaluator consumes the same artifact the cluster runs?
    If testing requires a translated or simplified copy of the rule, you now maintain two rules. They will drift, and the test suite will be green against a rule that is not the one enforcing. The property worth paying for is one artifact, two execution contexts.

An engine whose rules only run in-cluster is a spell-checker you can only invoke by publishing the book. It works, but nobody edits much.

saying these in an interview costs you the question

  • Says a rule can only be tested by applying it to a cluster
  • Assumes an offline pass guarantees the same admission verdict
  • Keeps a separate copy of the rule for CI and lets the two drift
  • Argues a per-pull-request throwaway cluster makes offline evaluation unnecessary
  • Tests only the exact manifest that prompted the rule

context

resolving links…