Diagnosing a Denial

## Why "just re-run it" is not diagnosis A gate refuses a change. The reflex is to tweak the job definition and push again, or to hit re-run in case it was a fluke. Both are guessing, and both are expensive: each attempt rebuilds inputs, occupies shared runner capacity, and takes minutes. Worse, if a retry goes green you have learned nothing about why the first attempt was refused, and the rule will refuse someone else next week. Diagnosis begins by moving the decision **off the enforcement path**: reproducing it somewhere you can poke at it, where being wrong costs nothing and where you can iterate in seconds. ## A decision is a function of three inputs Every policy engine, whatever the language, answers a question of the form *given this document, under these rules, with this reference data, is it allowed?* Reproducing a decision therefore means pinning three things: 1. **The input document** - the concrete JSON or YAML structure the engine was handed. For a pipeline gate that is the rendered CI job; for infrastructure it is a plan; for a cluster it is the submitted object. 2. **The rules** - a specific revision of the rule set, not "whatever is deployed now". Rules ship continuously; the version that refused you this morning may already have been replaced. 3. **External data the rule consults** - the approved-destination list, the ownership table, the exception list - as it stood at decision time. Pin all three and the answer is deterministic: the same trio always yields the same verdict. Miss one and you are debugging a different decision than the one that blocked you. ## The source you wrote is not the input the engine read This is the single most common time-waster. Engineers read the job definition in the repo, reason about what the rule *should* have done, and conclude the engine is broken. But between the file in the repo and the engine sits a rendering step: templates are included, matrix entries are expanded into separate concrete jobs, variables are interpolated, platform defaults are filled in, and fields you never typed appear while fields you did type may be nested somewhere else. Consider a rule that requires any build job holding production credentials to declare an egress allowlist. In the source there is one job block with a clear allowlist; after expansion there may be six jobs, and the rule reads each one on its own. So the artifact you want is the **evaluated input**, not the source. Practically that means your platform has to make it retrievable: the gate emits the document it evaluated (or a pointer to it) as part of the run, and the pipeline keeps it long enough for a human to fetch it. If your gate does not do that, the fastest permanent fix in the whole of policy operations is to make it do so - without the evaluated input, every denial becomes an archaeology exercise. ## The replay loop With the trio in hand, the loop is short: - **Replay.** Evaluate the recorded input against the pinned rules locally. Confirm you get the same verdict the gate got. If you do not, you have a different and more interesting problem: something in the trio differs. - **Reduce.** Delete parts of the input until the denial disappears. The last thing you removed is the field the rule is reacting to. This is bisection, and it works even when you cannot read the rule language well. - **Perturb.** Add the field the rule seems to want and confirm the denial clears. Now you have a hypothesis you can state in one sentence: *the rule refuses this job because field X is absent / has value Y*. - **Decide.** Only now is it worth arguing about whether the rule is right, whether the job should change, or whether the rule never anticipated this shape of input. ## What good self-service triage looks like The target is that a blocked engineer can run one command, on a laptop, with no access to the enforcement infrastructure and no production credentials, and get back which rule fired and on which field path. That means: the evaluated input is retrievable, the rule set is versioned and fetchable at that version, the engine runs standalone, and reference data has a snapshot. None of that is exotic - it is just deciding, before the first denial, that the decision has to be reproducible. ## The habits to carry Never debug a denial by editing and pushing. Never reason from the source file when the engine read a rendered document. Never assume the rules today are the rules that refused you. And when the local replay disagrees with the gate, treat the disagreement itself as the finding, not as a nuisance.