An auditor wants last quarter's policy evaluation re-run to reproduce the same verdict — what makes that possible?

## Why this question is asked It is the sharpest test of whether automated compliance evidence is real. Anybody can produce a record saying a check passed. Reproducing the evaluation in front of an auditor — feeding the archived input back through the archived rule and getting the identical verdict — demonstrates that the record was computed rather than asserted. The question is asked in senior interviews because the answer reveals whether the candidate designed for it up front, which is the only time it can be designed. ## Evaluation as a pure function Model the evaluation as `verdict = evaluate(rule, input, engine)`. Reproduction is simply calling the same function again with the same arguments. That framing tells you exactly what must be archived, and each argument has its own failure mode. **The input.** The document the rule was given: the description of the volume and its backup configuration as it was at that moment. If it was never captured, there is nothing to replay against, and the evidence record degrades into an assertion. Store it content-addressed and record its digest in the evidence record. **The rule.** Not "the backup-retention rule" but the exact text that ran. Rules get edited — thresholds move, exemptions get added, a helper is refactored. If the only reference is a branch name or a package name that always resolves to the current version, last quarter's logic is gone. Pin by digest and keep every version retrievable for as long as the evidence it produced is retained. **The engine and its configuration.** Behaviour can change between engine versions, and evaluation is often parameterised — which parts of the ruleset were loaded, which entry point was queried, how a missing field is treated. Record the engine version and the entry point. **Everything the rule reached for.** This is the one people miss. If the rule consults external data — an approved-exemption list, an asset inventory that defines which volumes count as production, a per-environment threshold table — that data is an argument too. It must be resolved before evaluation, folded into the archived input, and hashed with it. Otherwise you have archived two of three arguments. ## The three ways reproduction dies 1. **A moving rule reference.** The record names the rule but not its version, or names a version that is no longer retrievable. The replay uses today's logic, which may pass or fail for reasons unrelated to last quarter. 2. **An input that was never captured.** The pipeline evaluated a live description and kept only the verdict. There is nothing to re-run against, only a re-observation of today's world — which answers a different question. 3. **A live lookup inside evaluation.** The rule itself called out during evaluation. This is the subtle one, because such a rule looks fine and works fine; it just cannot be replayed, and the failure only surfaces at audit time. The fix is architectural: evaluation takes a document and returns a verdict, with all collection done before it and all reporting done after it. A fourth, smaller cause is non-determinism — a rule that reads the wall clock, samples, or emits messages whose ordering varies. Verdict-affecting non-determinism must be eliminated; parameterise the clock by passing an evaluation time in the input rather than reading it. ## What a successful replay actually proves Here is the distinction that separates a good answer from a great one. A matching replay proves the verdict follows from the archived input under the archived rule: the computation was honest and the record was not fabricated. It does **not** prove the archived input truthfully described the resource on the day. That is a separate claim, and it rests on the collection path: which credential read the resource, what API it read, and whether that read is itself recorded. Present these as two links in a chain — collection fidelity, then computational reproducibility — and be explicit that only the second is what replay tests. ## Designing for it before you need it In practice this means a small number of commitments made early: evaluation performs no I/O; inputs are archived content-addressed alongside every verdict; rule bundles are immutable and addressable by digest, retained as long as their evidence; the evidence record names the engine version and entry point; and there is a replay tool that takes an evidence record and re-derives the verdict without the original pipeline. Test that tool on old records periodically — a replay path that has never been exercised tends to be broken by the time somebody needs it, and a walkthrough is the worst place to discover a rule version was garbage-collected.