A regenerated findings baseline let a real regression pass a green gate. How did that happen?

## Why the mechanism is fragile A new-findings gate answers a **relative** question: is anything here that was not in the reference set? Its correctness therefore depends entirely on the reference set being trustworthy. The reference set is a file in the same repository, editable by the same people the gate constrains, and updating it is a routine, encouraged operation. That combination is the fragility. ## What regeneration actually does Updating a baseline comes in two flavours, and conflating them is the root of the incident. - **Merge**: add specific, named findings to the known set, leaving everything else alone. Reviewable, because the diff is small and each added entry can be argued about. - **Regenerate**: scan now, write the entire result as the new baseline. Fast, tidy, and it silently absorbs anything present at that moment — including whatever the branch just introduced. Regenerate on a branch that added a critical and the critical becomes 'known'. The subsequent evaluation reports no new findings, which is exactly right given the reference it was handed. The gate did not fail; the reference moved. ## How it happens without malice Almost never as an attempt to smuggle a vulnerability. The realistic paths: - A developer is blocked by unrelated fingerprint churn — a reformat, a file move — and regenerating is the documented, one-command fix. It clears the churn and their own new finding at the same time. - A convenience step regenerates the baseline automatically when the gate fails, which converts the gate into a no-op by construction. This is the worst version because it is systematic rather than occasional. - A large refactor produces hundreds of unmatched fingerprints for problems that all pre-date it, and reconciling them by hand is genuinely infeasible, so someone regenerates and a real regression rides along in the noise. ## Detecting it - **Diff the baseline, not just the code.** An added entry is a security change and deserves the same scrutiny as an edit to the policy itself. Route baseline changes to a reviewer who is not the author of the change that needed them. - **Watch the size.** Track how many entries the baseline holds over time. Growth is a policy event even when every build was green, and it is the only signal that survives when the gate itself has been rendered silent. - **Compare against a trunk-derived reference.** Periodically scan the mainline independently and compare its findings against the committed baseline. Entries that appear on a branch and never existed on trunk before that branch are exactly the pattern you are hunting. - **Report the total, always.** The reporting lane exists partly for this: if the absolute count jumps while the gate stays green, something moved the reference. ## Preventing it - **Never regenerate inside the pipeline.** The gate must not be able to rewrite its own reference. If a build step can update the baseline, the gate is decorative. - **Regenerate only from a scan of the mainline**, never from a feature branch, so a branch's own additions can never enter the reference. - **Prefer merge over regenerate** for day-to-day updates, and make regeneration a deliberate, announced operation with a review. - **Make entries expire.** An entry with an owner and a date is one that someone must revisit; a permanent entry is one nobody ever looks at again. - **Separate the refactor case.** When a legitimate large change churns fingerprints, handle it as its own reviewed change that touches only the baseline, so the reconciliation is visible and is not mixed with functional edits. ## The honest framing This is the standing cost of baselining. Grandfathering is what made the gate survivable on a real codebase, and the price is a reference file that the gated team can move. You do not eliminate that; you make moving it visible, reviewed and rare. If an interviewer pushes on whether the mechanism is therefore worthless, the answer is that an unbaselined absolute gate is not stricter in practice — it is simply switched off faster, and a gate that is off catches nothing at all. A movable reference that is watched beats a perfect rule that nobody runs. ## What to say in the incident review Name the sequence plainly: the baseline was regenerated on a branch that carried a new finding, the finding entered the reference, and the gate correctly reported no new findings against a reference that now contained it. Then propose the process controls, and resist the temptation to describe it as the gate having failed — it did what it was configured to do, which is a more uncomfortable and more useful conclusion.