A build gate finds no vulnerability scan report attached to an artifact — is that a pass?
answer
- absence is not a clean result
- three outcomes, not two
- an empty list denies nothing
- the report must name what it scanned
- scan status plus subject digest
basics
~20 sNo. A missing report is not a clean result, it is an absent one. A gate needs three outcomes — pass, fail, and no usable evidence — and the third must never be folded silently into the first.
solid answer
~50 sNo, and most gates get this wrong by accident. A rule usually says something like `deny when any finding in the report is critical`; if the report is missing, empty or unparseable there are no findings to iterate, the set of denials comes back empty, and the artifact sails through looking identical to a genuinely clean one. The fix is to make evidence existence its own assertion that runs before any finding is examined: the gate demands a report that parses, that records a completed scan, and that names the artifact it scanned. If that assertion fails, the decision is *unknown*, and unknown fails closed with a message saying no scan evidence was found for this artifact rather than reporting a policy violation. Zero findings is only credible when something vouches that the scan actually ran to completion over something.
go deeper
Be ready to say that a missing report is not a passing result, and that a gate needs a third outcome for 'no usable evidence'. Being able to name the three states is enough at this level.
Explain why the bug is silent: with no findings to iterate there are no denials, which the engine reports exactly like a clean artifact. Name the envelope fields that make absence detectable.
Show how you operate this: separate counters for violations and missing evidence, an assertion that the report's subject matches the artifact under test, and a failure message that points at the broken pipeline step.
Own the position that 'no usable evidence' is a platform-wide decision rather than something each pipeline re-argues, and be able to say who absorbs the cost when a team cannot yet produce the evidence the gate demands.
## The bug, and why it is silent A gate that reads a scanner report almost always encodes a rule of the shape *collect the findings whose severity is above some line; deny if that collection is non-empty*. Now delete the report. There are no findings to iterate, so the collection is empty, so no denial is produced. The engine's answer — an empty set of denials — is byte-for-byte the same answer it gives for an artifact that was scanned thoroughly and came back clean. Nothing errors. In most policy languages a rule body that never matches produces no result at all, and *no result* is not `false` and is certainly not a denial. The gate assembles its verdict out of denials, so the absence of denials reads as approval. This is why "we have a scanning gate" and "every deployed artifact was scanned" are two different claims. ## Three outcomes, not two Design the gate around **pass / fail / unknown**, and enumerate what lands in unknown: - the report file is absent — the step was skipped, ran on a different branch, wrote to a different path, or its name has a typo; - the file exists but is empty, truncated, or does not parse; - the file parses but records a scan that aborted, timed out, or was cancelled; - the report describes a *different* artifact than the one about to be admitted; - the report is well-formed but the fields the rule reads are not there. Unknown is not a softer fail. It is a different fact, with a different owner and a different fix, and it deserves its own message, its own metric and its own on-call story. ## Making unknown decidable A gate can only distinguish absence from cleanliness if the evidence carries an **envelope** around the findings. The minimum useful envelope is: - **subject** — the immutable identity of what was scanned, normally the artifact digest; - **tool identity and version** — who is making the statement; - **status** — did the scan complete, and what did it cover (packages, files, layers examined); - **timestamps** — when the scan started and finished. The rule then evaluates in two stages. First, assert the envelope: a report exists, it parses, its status is complete, and its subject equals the artifact under evaluation. Only then read findings. Structured that way, a missing field trips the presence assertion instead of quietly evaporating. ## Zero findings versus no scan Zero findings is produced by a clean artifact, and also by a scanner that crashed after writing an opening bracket, a scanner pointed at a path that matched nothing, and a scanner that silently skipped an ecosystem it does not support. All four are indistinguishable if all you look at is the finding list. A completion status plus a count of what was examined separates *scanned and found nothing* from *scanned nothing*. If a report claims zero findings and also claims it inspected zero packages, that is an unknown wearing a pass costume. ## Binding evidence to the artifact Evidence must be bound to a subject, or it can be laundered. If the gate does not compare the report's subject to the digest of the artifact it is admitting, then any report that parses will do — including yesterday's green report for a previous build. Binding is what stops one clean result from vouching for an entire release train. Note that this is a statement about the artifact, not about the world: how long a report stays *useful* as advisories accumulate is a separate decision from whether the report exists and describes the right thing. ## Say the right thing when it fails A pipeline message that says *policy violation* when the real problem is a skipped scan step sends the developer hunting through their own code for a bug that is not there. Distinguish the two in output and in metrics: violations are usually the change's fault, missing evidence is usually the pipeline's fault, and a rising count of the second is a systemic scanning outage that will otherwise hide inside a pile of ordinary findings. ## Scope This is about the *evidence* being absent, which is a property of the input. What a gate should do when the policy engine itself is unreachable is a separate design decision with its own tradeoffs, and it is not what this question asks.
- How does a rule tell 'the scan ran and found nothing' from 'the scan never ran'?By reading the envelope, not the finding list. Require the report to carry a completion status, the tool identity, and how much it examined — packages, files or layers. Zero findings alongside a completed scan of 412 packages is a result. Zero findings alongside a missing, aborted or zero-coverage status is an absence, and the gate should call that unknown.
- A report is present but describes yesterday's artifact — what should the gate do?Treat it as unknown. The gate compares the subject identity recorded in the report with the digest of the artifact it is about to admit; if they differ, the report is evidence about something else. Accepting it lets one previously green artifact launder every later one that happens to reuse the file.
- Why separate 'no evidence' failures from 'policy violation' failures in the pipeline output?Because the owner and the fix differ. A violation points at the change; missing evidence almost always points at a skipped or broken pipeline step. Mixing them buries a scanning outage inside ordinary findings and trains developers to read every red build as their own problem. Keep separate counters as well as separate messages.
A blank lab result is not a negative lab result. If the sample never reached the machine, the correct report is that nothing was tested, not that nothing was found.
saying these in an interview costs you the question
- Says zero findings means the artifact is clean
- Treats a missing report as an empty finding list
- Checks findings but never checks the report exists
- Accepts any report without checking what it scanned
- Assumes the pipeline already failed if the scan failed