skip to content

A record says your audit-log retention rule passed six months ago. What must it capture to still mean anything?

level: middleimportance: nice to knowfreq 32%

answer

  1. a pass is a claim about a pair
  2. rules move; thresholds get raised
  3. which input did the engine actually see
  4. no rule matched is not compliant
  5. pin the rule set revision, not just the commit

basics

~20 s

The identity of what was judged and what judged it: the input the engine actually saw, the rule set version in force, and an explicit decision. Passed means little unless you know which rule passed it.

solid answer

~40 s

Passed is a claim about a pair, an input and a rule, so the record has to fix both. It must identify what the engine actually evaluated, not just the commit: which resources, from which rendered configuration or plan. It must name the version of the rule set in force at that moment, because rules move; if the minimum audit-log retention was raised from thirty days to ninety last quarter, a pass recorded under the old rule set is not evidence for the current control. And the decision should be positive rather than inferred from an empty violation list, because engines emit nothing both when a resource complies and when no rule matched it. Without those, a stored pass tells you only that something, somewhere, once had no complaints.

code

json · 6 lines
json
{
  "commit": "9f2c1ab",
  "pipeline_run": 41822,
  "result": "pass",
  "timestamp": "2026-02-11T09:14:03Z"
}

go deeper

for a junior

Know that a stored result has to say what it judged. Passed on its own is not information; the record should also identify the change it looked at and the rule that passed it.

for a middle

Be ready to explain rule drift: thresholds and selectors change, so a past pass is only evidence about the rule version in force then. Explain why an empty violation list conflates compliant with out of scope.

for a senior

Show that you would record the evaluated input's identity rather than just the commit, because the engine sees derived data. Be able to answer an auditor asking which rule revision produced a given pass.

for a principal

Decide how rule set revisions are named and frozen so past decisions stay interpretable, and set the rule for whether tightening a threshold triggers re-evaluation of what already exists or applies only forward.

## A pass is a claim about a pair Every gate decision is a statement of the form: *this input, judged by these rules, produced this outcome*. A record that keeps only the outcome has thrown away two thirds of the statement. Six months later, the person reading it — an auditor, or an engineer working backwards from an incident — has to answer a specific question, and the record either supports the answer or it does not. ### 1. Which rules were in force Rule sets are living code. Thresholds get raised, selectors get widened, a rule that used to warn starts to block, an exception baked into the rule expires. Take a minimum-retention rule on an audit log. Last year it required thirty days; this quarter it requires ninety. A stored `pass` from February is a true statement about the thirty-day rule and says nothing whatsoever about the ninety-day one. If you present old passes as evidence for the current control, you are backdating a guarantee that was never evaluated. So the record needs an identifier for the rule set as a whole — a version, a revision, a bundle identity — and that identifier has to be resolvable to the rule text later. This is why rule sets belong in version control with an immutable identity per release, and why a gate that pulls the latest rules from a mutable location makes its own history uninterpretable: it can tell you a decision was made, but not what the decision was made against. ### 2. What the engine actually saw The commit is not the input. Engines almost always evaluate something *derived* from the commit: a rendered manifest, an execution plan, a resolved configuration with variables and upstream state folded in. The same commit can render differently depending on tool version, input variables, and the state of things it references. Recording the commit alone leaves you unable to say which datastore definition was judged, or whether the resource in question was even present in the evaluated input. This matters most for the resources that are *missing* from an evaluation. If a plan-time evaluation covered eleven resources and the datastore was created by a twelfth path the gate never saw, the record still says pass, truthfully, about eleven other things. Only an identifier for the evaluated input — and ideally the list of resources it contained — makes that visible. ### 3. An explicit decision, not an empty list Most engines report violations. When they report nothing, two very different situations produce the same output: - the resource was in scope, the rule was evaluated against it, and it complied; and - no rule in the set selected that resource at all, so nothing was ever evaluated. That second case is the policy author's classic blind spot. A rule that does not match produces no result rather than an approval; nothing rejected the change, but nothing endorsed it either. If a record only stores an empty violation list, it cannot separate *checked and clean* from *never in scope*. Recording which rules were evaluated against which resources — even as counts — separates them, and it is the difference between a record that can answer a coverage question and one that cannot. ## Putting it together A record that carries these three things supports a real claim: *at this time, the rule set at revision R was evaluated against this input, rule X applied to this datastore, and the outcome was allow.* A record that carries only `{commit, pass}` supports the much weaker claim that a pipeline once emitted the word pass. There is a governance consequence too. Once the rule version is pinned in every decision, you can answer questions that are otherwise unanswerable: which changes were only ever judged by the old threshold, how long the estate ran under the looser rule, whether the tightening was applied retroactively to existing resources or only to new ones. Those are exactly the questions that follow a threshold change, and they are impossible if every historical record just says pass. ## The trap of confusing this with a re-run None of this is the same as being able to reproduce the decision. Reproducing means re-executing the old rules on the old input, which needs both preserved and needs the engine's behaviour to be stable. Pinning identities is weaker and much cheaper: it lets you *interpret* the record — to know what claim it makes — even when you cannot re-execute anything. Most of the time, interpretation is what is actually being asked for.

  • The retention minimum was raised from thirty days to ninety. What can you say about commits that passed before the change?
    Only that they met the thirty-day rule. They are not evidence for the ninety-day control, and presenting them as such backdates a guarantee nobody made. If the tighter rule matters for what already exists, you have to evaluate the current estate against it and say so, rather than reusing old passes.
  • Why is an empty violation list a weak way to record a pass?
    Because it conflates two outcomes: the resource was checked and complied, and no rule applied to it at all. A rule whose selector missed the resource type produces the same empty list as a clean evaluation. Recording which rules ran against which resources separates checked-and-clean from never-in-scope.
  • Does storing the commit hash identify the input the engine saw?
    Not on its own. The engine usually evaluates something derived from the commit — a rendered manifest, a plan, a resolved configuration — which also depends on tool versions, variables and upstream state. An identifier for the actual evaluated input, and ideally the resources it contained, is what keeps the record interpretable later.

saying these in an interview costs you the question

  • Stores only pass or fail alongside a commit hash
  • Assumes the rule set never changed between then and now
  • Treats an empty violation list as proof of compliance
  • Believes the commit alone identifies what was evaluated
  • Presents old passes as evidence for a newly tightened threshold

context