Machine-Generated Evidence

## What "evidence" means in this context An auditor testing a control is not asking whether you believe it held. They are asking to be shown an artifact, produced at the time, that a person who does not trust you can check for themselves. A green pipeline run is not that artifact. It is a UI state: the log rotates, the console link expires, the run identifier means nothing outside your tooling, and nothing in it states which control the run was supposed to prove. Machine-generated evidence is the deliberate act of turning a check's output into a durable, self-describing object. ## The control fixes the shape of the record Take a concrete control: every production volume must have at least 35 days of backup retention. That one sentence already determines the shape of the evidence. There is a **population** (production volumes), a **period** (the cadence at which the control is expected to operate), and an **assertion about each member of the population**. So the unit of evidence is one record per volume per period — not one record per pipeline run, and not one summary saying "all volumes compliant". ## The fields, and what each one buys you - **Control identifier.** Binds the record to the thing it proves. Without it, you have a technical scan result that somebody must later argue is relevant. With it, the record answers a specific control on its own. - **Resource identifier.** Names which member of the population this record is about. Use the stable machine identifier (a cloud resource ID or URI), not a display name someone can rename. - **Verdict plus the observed value.** `pass` alone hides the difference between a volume sitting exactly on the 35-day boundary and one retained for a year. The observed number lets you answer questions about margin and drift later without re-running anything. - **Rule version.** A digest of the exact policy bundle that produced the verdict. "Rule: backup-retention" is a name, and names get edited; a digest identifies one immutable text. - **Input digest, and the archived input.** The digest lets anyone confirm the stored input is the one that was evaluated. The archived input is what makes the digest usable — a hash of a document you threw away proves nothing. - **Evaluation timestamp.** When the observation was made, not when the file was written. Auditors test whether the control operated throughout the period, so the timing of each observation matters. - **Runner identity.** Which workload, job or service account produced the record. This is what lets the evidence store's own access records corroborate the claim. ## Why one record per resource Audit testing is sampling. The auditor picks items out of the population and asks you to show the evidence for those specific items. A single aggregate record forces you to re-derive per-item results at audit time from data that may no longer exist, and it hides the population: if three volumes were created and destroyed inside the period, an aggregate says nothing about them. Per-resource records also survive scope changes — when the population grows, you get more records rather than a redefined summary. ## The two questions every record has to survive 1. **Was it true when it was written?** That is a question about the input: where the observation came from, whether it was captured, and whether it faithfully described the resource at that moment. 2. **Has it been changed since?** That is a question about integrity: whether the stored object can be edited or deleted after the fact without anyone noticing. A record designed only to look good answers neither. The field list above exists because each field is what somebody reaches for when one of those two questions is asked in a walkthrough. ## Where the record goes Evidence belongs in a store the producing pipeline can append to and nothing routine can overwrite, with a retention period at least as long as the framework requires you to keep evidence for that control. If the same credential that writes the record can also delete it, the record's persistence is a matter of trust rather than a property of the system — and that is exactly the property an auditor is testing. ## The common failure The common failure is not a missing field; it is treating the CI system as the evidence store. Logs are retained for weeks, are mutable by anyone who can re-run a job, and are organised by pipeline rather than by control. The moment you decide the record is an artifact with its own schema, its own store and its own retention, most of the rest follows.