What must a policy rule's test suite assert beyond denying the obviously bad input?

## Why deny-only suites pass bad rules The instinct when testing a rule is to feed it the thing it is supposed to catch. For "managed database instances must keep at least seven days of backups", that is a template with a one-day retention window, and the test asserts a denial. Green. The problem is that a rule which denies *everything* also passes that suite. So does a rule that denies on the wrong grounds — matching the resource type and ignoring the property entirely. Deny-only fixtures test that the rule is *loud*, not that it is *right*. The missing half is allow fixtures: documents that are compliant and must come back with no findings. Over-blocking is the failure mode that kills a gate politically. A rule that misses a violation produces a risk somebody may never notice; a rule that blocks a compliant deploy produces an angry team at 5pm and a standing argument about whether the gate should be advisory. Allow fixtures are the only cheap defence against it. ## The three shapes every fixture set needs 1. **Violating** — the property is present and wrong (retention of one day). Must deny, with the expected rule id and message. 2. **Compliant** — the property is present and right (retention of fourteen days). Must produce no finding. 3. **Absent** — the property is not in the document at all. This is the case that separates a tested rule from an untested one, and it is where most rules quietly fail. A rule written as a numeric comparison against a field that does not exist frequently produces no result at all rather than a denial. No result is not a denial. The template sails through, and the resource is created with whatever the platform's default happens to be. Whether absence should deny is a judgment — often it should, because you cannot demonstrate compliance from a document that does not mention it — but it must be a decision the suite records, not an emergent property of the expression. A fourth shape is worth adding wherever the surface allows indirection: the value is supplied by a parameter, an intrinsic reference, or anything resolved after the document is reviewed. The rule sees a reference object where it expected a number. Comparing it numerically may throw, may silently produce nothing, or may compare against a type it does not understand. Decide what you want — commonly, deny with a message saying the value could not be determined at review time — and fixture it. ## Assert the output, not just the verdict The verdict is a boolean; the *message* is what a blocked engineer reads. Assert it. A test that pins the rule id and the denial text catches a rename that would orphan every reference to that rule, and it stops the message decaying into "policy violation" as the rule is edited over time. If the repository requires each rule to name its owning team, assert that field too, so it cannot go missing. ## Where fixtures should come from Hand-written fixtures encode what the rule author imagined. Real templates contain shapes nobody imagines: nested modules, resources declared through indirection, generated documents, the same logical resource expressed through a different resource type. The strongest suites are seeded from the estate — a corpus of documents already merged across the organisation — and each new rule is evaluated against that corpus before it merges, so the reviewer sees the true impact rather than a guess. Also: **every false positive that ever reached a team becomes a permanent fixture**. That is how a rule stops making the same mistake twice, and it is the cheapest trust-building move available to a rule owner. ## What a green suite does not prove It proves the rule matches the inputs you wrote down. It does not prove: - **Coverage of the resource types that matter.** The requirement is about databases; the rule may only match single instances while the estate's production data lives in cluster resources that carry their own retention property. Every document you fixtured is a document you thought of. - **That the rule's premise is still true.** A rule encoding a requirement that has since changed passes its own tests forever. - **Anything about the templates teams actually write.** Only evaluating the rule against real documents tells you that. The useful mental model: the suite is a contract with the rule's future editors, saying "these are the decisions we made on purpose". It is not a claim about the world.