Writing a Testable Rule

## The move A shadow run — also called a dry evaluation or a backtest of a rule — is the step between writing a rule and letting it decide anything. You take the candidate rule, feed it real inputs that already exist somewhere, and record the verdict it would have produced for each one. Nothing is enforced, nothing is warned, no developer sees a message. The run is a batch job over stored data, not a deployment. ## What it looks like concretely Take a rule that says every workload image must descend from an approved base-image lineage. The surface it reads is image metadata: the labels the build stamped on the image, the recorded base reference, the layer history. That metadata already exists for every image your pipelines produced. So the corpus is, for example, ninety days of built images — a few thousand records — and the run evaluates the rule once per record. The output is a table with one row per image: the image, the service and team that owns it, the verdict, and the message the rule would have printed. Two aggregates fall out of it immediately: how many images the rule would have denied, and how those denials distribute across services and teams. ## Why review is not enough Reading a rule tells you whether it expresses the intent you had. It cannot tell you what the rule does to the population it will meet, because that population is a fact about your estate, not about your rule. Estates are full of history: a service that still builds from a base image approved two years ago, a build that renames labels, a mirrored registry path that is the same base under another name. Every one of those is a change that would be denied by a rule you would have signed off on in review. The asymmetry matters. A rule that misses a bad change costs you a risk you already had. A rule that denies a good change costs you a person's afternoon, an escalation, and — repeated a few times — the credibility of the whole gate. So the measurement you want before shipping is specifically about the good changes. ## What comes out is a list, not a verdict The most common misreading is to treat the shadow denial count as an error count. It is not. A denial can mean the rule is right and the image genuinely descends from something unapproved, in which case you have found real work for a team to do. It can also mean the image complies in substance and the rule read the wrong thing. Separating the two is manual triage, and the shadow run's job is only to hand you the shortlist. The allowed rows deserve a look too. Everything the rule let through is where its misses live, and no aggregate in the report will point at them; you only find them by sampling. ## What it costs and what it risks Almost nothing. The engine is not in anyone's request path, so there is no latency question and no availability question — an evaluation that crashes on a malformed record costs you a rerun. The real cost is the triage labour on the denial list, which is why a corpus is chosen to be representative rather than exhaustive. ## What it cannot tell you A shadow run measures history. It contains only changes people actually made, in a world where this rule did not exist. It does not contain the change nobody attempted, the emergency rebuild that happens twice a year, or the practice teams will adopt once the rule is real. That is a limit to state out loud when you present the result, not a reason to skip the measurement. ## The decision it feeds At the end you can say something a lead can act on: this rule would have denied N changes over ninety days, of which M were legitimate, concentrated in these teams. That sentence is the difference between arguing about a guardrail and deciding about one — and sometimes the honest conclusion it supports is that the rule should not ship at all.