Why keep a policy test fixture that your rule is expected to deny, and what does its sudden pass mean?
answer
- a green suite can mean nothing ran
- compliant fixtures cannot detect silence
- test the deny, not just the allow
- the fixture that must keep failing
- a sudden pass is the alarm
basics
~20 sA rule that stops matching anything denies nothing, and that silence looks exactly like success. A fixture the rule must deny turns the silence into a failing test: if it suddenly passes, enforcement has quietly stopped.
solid answer
~50 sPolicy rules fail open by construction. A rule contributes a denial only when its condition matches the document in front of it; if the document changes shape and nothing matches, the rule produces no denial and raises no error. Compliant fixtures cannot detect that, because a rule that matches nothing allows every compliant input too. So for each rule I keep at least one deliberately non-compliant input whose expected result is `deny`, and the test asserts the denial. That fixture is a canary: while it keeps failing the rule, the rule is alive. The day it passes, the most likely explanation is not that the input became compliant but that the rule stopped selecting it, usually after a platform or engine upgrade moved the field or version it reads. The wrong first move is editing the fixture until the suite is green.
go deeper
Be ready to say plainly that a rule matching nothing denies nothing, and that a suite of only-compliant fixtures cannot tell you that happened. Know what a must-deny fixture asserts.
Explain the mechanics: what changes to an input document make a rule stop matching, and why nothing errors when it does. Expect to be asked how you would triage a canary that went green.
Show the operating habit — a canary per rule, run against the real enforcement path and not just in CI, plus a second signal such as denial-rate monitoring after any upgrade.
Own the culture question: a control whose alarm is routinely silenced is worse than no control, because it is on the evidence list. Talk about who is allowed to edit a canary and what review that requires.
## The failure this fixture exists to catch A policy rule is not a program that runs to completion and reports on itself. It is a condition evaluated against a document, and it contributes a denial only when that condition matches. If the document changes shape — a field moves to a new path, a kind starts being served under a new group-version, the object your rule used to select is no longer selected — the condition matches nothing and the rule contributes nothing. Nothing throws. No log line announces that the rule went quiet. The pipeline is green, the requests succeed, and the guardrail is simply not there any more. That asymmetry is the whole problem. **A passing policy test suite is equally consistent with the rules working and with the rules matching nothing at all.** Positive fixtures — compliant inputs that must be allowed — cannot tell those two worlds apart, because a rule that has stopped matching allows everything, including every compliant fixture you own. The more thoroughly you test the happy path, the more confident and the more wrong you become. ## The canary: a fixture that must fail The fix is cheap. For every rule, keep at least one input that is unambiguously in violation of it — an object that carries the deprecated API version the rule bans, a workload missing the label the rule requires — and write the test so the run is green only when the engine returns a denial for it. The assertion is inverted from the usual one: success means the rule said no. Two practices make these worth having: - **Name them for what they are** (`must-deny/`, `canary-`, `expected-violation-`) with a comment saying that a pass is an alarm. Otherwise a future maintainer sees a fixture that fails the policy, assumes it is stale test data, and deletes or fixes it. - **Keep one canary per rule, not one per suite.** A single shared bad input tells you the engine is alive; it does not tell you that *this* rule is alive. You want the failure attributable to a rule. ## What a sudden pass means, and what it does not When a canary starts passing, work three hypotheses in this order: 1. **The rule stopped matching.** Something changed the shape of the input — a new API version for the kind, a renamed or relocated field, a different serialisation of the object, an engine upgrade that changed how the input document is assembled. This is by far the most common cause and the only one that is a live security gap. 2. **The fixture drifted.** Somebody regenerated it from a template, or a formatter rewrote it, and it is no longer actually in violation. Confirm by reading it against the rule's own words. 3. **The rule changed on purpose.** Someone narrowed its scope in the same change. Then the canary needs updating and the change needs review — but as a deliberate decision, recorded, not as test maintenance. The move that destroys the control is fixing the symptom: editing the fixture, loosening the assertion, or marking the test skipped so the build goes green. Every one of those converts a working alarm into a permanently silent one. ## Where the canary is weak Be honest about what it proves. A canary shows the rule denies *that* input. It is a liveness check, not a coverage check — it says nothing about the violations you never wrote a fixture for. It also has a trap of its own after an upgrade. If the canary is expressed in the old shape while real objects have moved to the new one, the canary keeps failing the rule, the suite stays green, and production is unguarded anyway. Every shape you actually accept needs its own canary: if the estate now carries a kind under two group-versions, you want a deliberately-bad fixture in both. So pair it with checks that cover different blind spots: replaying a corpus of the estate's real stored objects against the rules before an upgrade window, and watching the denial rate afterwards — a rule whose denial count drops to zero on upgrade day is telling you the same thing the canary would, a little later.
- Your canary went green the morning after a platform upgrade. What do you do first?Do not touch the fixture. Feed the engine a real object of that kind taken from the live estate and see whether it is denied. If it is not, the rule has stopped matching the current shape and the guardrail has been off since the upgrade — that is an incident with a start time, not a test failure. Only after confirming the rule's behaviour do you decide whether the rule, the fixture, or both need to move to the new shape.
- How is this different from ordinary positive test coverage of a rule?Positive coverage checks that compliant inputs are not blocked, which protects developers from false denials. It cannot detect a rule that matches nothing, because such a rule also lets every compliant fixture through. The must-deny fixture is the only one whose result changes when the rule goes silent, so it is the one that carries the liveness signal. You want both: the positive cases guard against over-blocking, the canary guards against under-blocking.
- Should the canary run in CI only, or also against the running system?Both, and they answer different questions. In CI it protects the rule source at merge time. Run periodically against the real enforcement path — submit the known-bad object in a dry-run or a throwaway scope and assert it is refused — and it also covers the deployment: the rule bundle that never loaded, the wrong version pinned, the scoping that excludes the namespace. The CI copy cannot see any of those.
It is the smoke-detector test button. Pressing it and hearing nothing is the point of pressing it; a detector that never chirps is not proof of clean air.
saying these in an interview costs you the question
- Believes a green policy suite proves the rules still enforce
- Fixes a canary that starts passing instead of investigating
- Assumes a rule that stops matching will raise an error
- Tests only compliant inputs against the rule
- Keeps one shared bad fixture and calls every rule covered