Your conftest gate has been green for months, but the rule never ran. How?
answer
- green can mean nothing ran
- default namespace is main only
- a renamed package silently unselects
- warnings exit zero without the flag
- canary fixture that must fail
basics
~20 sA conftest run with nothing to evaluate exits zero. The usual causes are a namespace never selected on the command line, a renamed policy package, a glob that missed the files, or an extension with no parser attached - all of which look identical to a clean pass.
solid answer
~50 sconftest only evaluates the namespaces you ask for: by default the `main` package, plus whatever `-n` names, or everything with `--all-namespaces`. If the rule lives in a package the invocation never selects - because someone renamed the package, or because that repository's command line was written before the rule existed - it loads and does nothing. The same green appears when the file glob missed the directory, when the file extension has no parser, or when the rule was authored as a warning and nobody passed `--fail-on-warn`. All of those exit zero and print a summary that reads like success. The fix is to stop trusting absence of failure as evidence: run a canary fixture that must fail in every pipeline, assert on the machine-readable output's counts rather than the exit code alone, standardise the invocation centrally instead of letting each repository hand-roll it, and keep `conftest verify` unit tests next to the rules.
go deeper
Remember that a policy run reporting no failures might have evaluated nothing at all, and that the rules which run are selected, not automatic.
Be able to list the concrete causes - unselected namespace, renamed package, missed glob, unparsed extension, warning-only rule - and explain why each exits zero rather than erroring.
Show how you make enforcement observable in a fleet: canary fixtures, asserting on evaluated counts, a centrally owned invocation, and separating rule unit tests from proof of wiring.
Own the difference between declared and accidental non-enforcement, and the inventory that lets you answer which repositories are covered before someone else asks.
## Why this is the platform team's problem When one engine runs across dozens of repositories with mixed file types, the invocation - which policies, which namespaces, which paths - is different in every repository, and it is the invocation that decides whether a rule is enforcement or decoration. The rule library looks complete. The dashboards are green. Nothing in either tells you that in eleven of those repositories the rule was never evaluated. ## Every way a run exits zero having done nothing **The namespace was never selected.** Policies are Rego packages; conftest calls the package the rule's namespace and evaluates only `main` by default. `-n docker` runs the `docker` namespace; `--all-namespaces` runs them all. A rule added under a new package that the repository's command line does not name is loaded and ignored. **The package was renamed.** Refactoring a policy repository into clearer packages is normal and harmless in the policy repository's own tests, which reference the rules directly. It is not harmless for every consumer whose command line pins the old namespace by name. **The files were never passed.** conftest evaluates the paths you give it. A glob that covers `k8s/` while the team has started putting manifests in `deploy/overlays/` produces exactly the same output as a clean repository. **The extension has no parser.** An unrecognised extension is not silently treated as YAML; forcing `--parser` is how you cover it. Until someone does, those files are outside the gate. **The finding was a warning.** Warnings print and count, but exit zero unless `--fail-on-warn` is set. A rule staged as a warning and never promoted is a rule that has been visible and ignored for months. **The exit code was swallowed.** A step that ends in `|| true`, a script that does not propagate the status, or a job configured to continue on error will happily discard a real failure. This one is not conftest's doing at all, and it is the most embarrassing to discover. ## Deliberate non-enforcement is fine; invisible non-enforcement is not Leaving a namespace out of one repository's invocation is a legitimate tool. It is how you roll a rule out in waves, and how you let a repository that genuinely has no Dockerfiles skip Dockerfile rules instead of paying for evaluation it does not need. The distinction that matters is whether the omission is **declared or accidental**. A declared omission has a name, an owner and a date; it appears in an inventory of which namespaces run where; someone can answer "which repositories are not enforcing this yet" without grepping thirty command lines. An accidental omission is indistinguishable from compliance right up until an incident. ## How to make it observable - **Canary fixtures.** Check in a file that is *supposed* to violate the rule and assert that the run reports a failure for it. If the canary stops failing, the gate stopped working. This catches every cause above at once, because all of them make the canary pass. - **Assert on counts, not just status.** The machine-readable output carries the number of tests, passes, warnings and failures. Zero tests evaluated is a red flag that a zero exit code cannot express. - **Own the invocation centrally.** A shared workflow or wrapper that every repository calls means the namespace selection is reviewed in one place. Hand-rolled command lines are where namespaces quietly go missing. - **Unit-test the rules themselves.** `conftest verify` runs the test rules that live beside your policies. That proves the rule logic is right; it does not prove the rule is wired into any pipeline, which is why you need both. - **Report on coverage, not results.** A quarterly answer to "which namespaces ran in which repositories" is a different artefact from "how many failures did we have", and it is the one that tells you whether the control exists. ## What to say when asked for evidence A year of green runs demonstrates that nothing failed. Demonstrating that the control was *applied* takes the canary, the coverage inventory, and the record of who was deliberately exempt and why. Building that habit early is much cheaper than reconstructing it from CI history under time pressure.
- How do you prove a rule ran, rather than merely passed?Keep a canary fixture that must produce a failure and assert on it, and read the run's machine-readable output for the number of tests evaluated. Zero evaluated tests is invisible in the exit code but obvious in the counts. Both checks fail loudly the moment a namespace, glob or parser stops covering what you thought it covered.
- One repository should genuinely skip a namespace. How do you keep that from becoming an invisible hole?Make the exemption a declared object, not an omission: an owner, a reason and a review date, listed in an inventory of which namespaces run where, with the invocation itself owned by a shared wrapper rather than hand-rolled per repository. Then "who is not enforcing this" is a query, not an archaeology exercise.
- The rule's own unit tests are green. Does that settle it?No. Policy unit tests prove the rule decides correctly when it is given input. They say nothing about whether any pipeline gives it that input - the namespace selection, the file globs and the parser coverage are all outside the rule. You need both the unit tests and evidence from the pipeline itself.
saying these in an interview costs you the question
- Treats a green pipeline as proof the control ran
- Does not know only main is evaluated by default
- Never tests that a violating file actually fails
- Lets each repository hand-roll the invocation
- Confuses passing rule unit tests with enforcement