Why can't a release gate trust a scan report that the build job wrote about itself?
answer
- ask who produced the evidence
- the witness is the defendant
- same trust boundary as the artifact
- a skipped step still leaves a green file
- warn on it, never block on it
basics
~20 sThe build job is the thing being gated, so a report it wrote about itself is a claim, not a check. Anything that skips, misconfigures or compromises the build can also produce a passing file.
solid answer
~50 sEvidence is only worth what its producer is worth. A `results.json` in the build's own workspace, a step output the job sets, an environment variable the build script exports — all of these are produced inside the same trust boundary the gate is supposed to be evaluating. If someone can influence the build, they can influence the verdict, and you never have to reach for a malicious story: a scan step that was silently skipped, a cached file from last week, or a flag flipped to `--exit-code 0` all leave a green file behind and the gate happily passes. So a green result here proves only that a file existed and parsed. The fix is to change who produces the evidence: a scan the platform runs on the published artifact, or a signed statement the gate can attribute to a builder identity it pinned in advance. Self-asserted output is still fine as a warning or a dashboard signal — just never as the thing that blocks.
go deeper
Be ready to say, in one sentence, that a file the build wrote about itself is a claim rather than a check, and to name a mundane way it goes green without any scanning happening.
Expect to be pushed on the variants — step outputs, job outputs, exit codes, exported environment variables are all the same trust class — and to explain exactly what a passing gate on such input does and does not establish.
Show you would re-plumb the producer rather than harden the rule: platform-run scans on the published artifact, findings stored where the build cannot write, and self-asserted output demoted to warnings.
Own the framing that a gate whose input the gated party controls is a reporting liability, not just a weak control — it lets the organisation believe a control exists, which is worse than knowing it does not.
## The shape of the problem A pipeline gate almost never inspects the artifact directly. It reads *evidence about* the artifact: a scan report, an inventory, a test summary, a signed statement. The single most important question a gate author can ask about any input is **who produced this, and could they have produced it and still passed?** "Self-asserted" evidence is evidence produced by the same party the gate is judging. The canonical form is a file the build job writes into its own workspace and a later job reads. Two variants are exactly the same class and are missed far more often: - a **step output or job output** that the build sets (`scan_passed=true`) and a downstream job branches on; - an **environment variable or exit code** produced by a script that lives in the repository being gated. All three share one property: the entity that benefits from a pass also authors the pass. ## Why this is not just a malicious-attacker argument It is tempting to answer "because an attacker who owns the pipeline can forge it" and stop. That is true, but it makes the flaw sound exotic. The everyday failures are more common and just as damaging: - **The step didn't run.** A conditional, a matrix filter, a changed path filter or an early `continue-on-error` means the scanner never executed, but a stale artifact or an empty file is still on disk and the gate reads it as pass. - **The step ran and was told not to fail.** Many scanners exit zero unless asked otherwise; a report full of findings is a perfectly valid "green" file to a rule that only checks the file parsed. - **The report describes something else.** The build produced two images and the report is about the first one; nothing in the file binds it to what you are about to promote. - **A developer changed the pipeline.** The pipeline definition usually lives in the repository being gated, so the person the gate constrains can also edit the code that generates the gate's input. Each of these produces a green gate with no scanning behind it. A gate that fails this way is worse than no gate, because the organisation now believes the control exists. ## What a green gate on self-asserted evidence actually proves Only this: *a file was present, parsed, and matched the rule's shape.* It does not prove the check ran, that it ran against this artifact, or that its result was unmodified. If someone asks "what did that gate establish?", the honest answer is "that the build claimed something". ## What changes when the producer changes The repair is not a better rule; it is a different input. Two directions work: 1. **Move the producer out of the gated party's reach.** Have the platform scan the published artifact in a job the product team cannot edit, and have the gate read the finding record from a service the build has no write access to. Now producing a false pass requires compromising the platform, not the build. 2. **Make the evidence carry an attributable issuer.** A signed statement about the build lets the gate ask "who issued this?" and compare that against an identity the policy pinned in advance. The statement's contents become believable exactly to the degree the issuer is trusted — and crucially, the issuer is something the rule names, not something the evidence claims about itself. Both are the same move: put a boundary between the thing being judged and the thing doing the judging. ## Where self-asserted output still belongs Don't rip it out. Build-local reports are cheap, fast, and give developers feedback in the place they are already looking. Use them to **warn**, to annotate a merge request, to populate a dashboard. What they may not do is carry a blocking decision at a promotion boundary, and they may not be presented to an auditor as evidence that a control ran — a check whose input is written by the party it constrains evidences nothing. ## The one-line test Before any gate input is allowed to block, ask: *if the team I am gating wanted this to pass, what would they have to compromise?* If the answer is "a file in their own build", the input is self-asserted and belongs in the warn tier.
- Is a job step output any different from a file in the workspace?No — same trust class. A step output, a job output, an exported environment variable and a written report are all values the gated job chose. The gate reading them cannot distinguish "the scan passed" from "the job said the scan passed", which is the only thing that matters.
- The pipeline definition is protected by code review. Does that make its output trustworthy?It raises the bar without changing the class. Review constrains deliberate edits, not skipped steps, stale files, mis-scoped reports or a scanner exiting zero on findings. It also does nothing once the build environment itself is compromised. Review is a good control over the pipeline; it is not a substitute for independent evidence.
- So should we delete build-local scan reports?No. Keep them for fast developer feedback, annotations and dashboards, where being wrong costs nothing. The discipline is about outcome: self-asserted evidence may warn, and must not be the sole input to a block at promotion or be reported as a control that ran.
It is a defendant handing the court a note, written by the defendant, confirming they were somewhere else. The note may well be true — but it is testimony, not evidence, and it is not the sort of thing you convict or acquit on.
saying these in an interview costs you the question
- Says the pipeline is locked down, so its output is trustworthy
- Treats a green gate as proof the scanner actually ran
- Thinks committing the report to git makes it independent
- Believes a checksum the build computes about itself proves integrity
- Cannot name who produced a given gate input