Should a scanning pipeline pin its ZAP add-on set or refresh it, and how would you decide?
answer
- the rules are add-ons, not the program
- the add-on set is the oracle
- reproducible verdict versus current coverage
- refresh as a reviewed, re-baselined change
- separate what blocks from what reports
basics
~20 sZAP's detection logic is not in the program — the rule packages and job types are separately versioned add-ons. Pinning the set makes two runs comparable and lets coverage age; refreshing buys coverage and makes a new finding ambiguous between tool change and code change.
solid answer
~40 sThe decision is unusual because the thing being versioned is the **oracle**. ZAP's active and passive rules ship as add-ons, and so do the job types a plan can name, so the add-on set is what your pipeline can detect. Pin it and the same commit scanned twice gives the same verdict, a new finding is attributable to the application, and coverage quietly ages. Refresh it and coverage arrives, but a gate can go red on an unchanged application because the tool changed, and every run needs marketplace egress. The usual resolution is to pin the running image, refresh on a schedule as a reviewed change that re-baselines, and separate the gate from the report — pinned set for the gate that blocks, unpinned run publishing informationally.
go deeper
Know that ZAP's rules arrive in add-ons rather than in the program, so a scan's findings depend on which add-ons your install carries. Recording the add-on listing beside a result is a habit worth starting early.
Explain why a scan can report something new against unchanged code: the rule packages are versioned separately and a refresh changes what the run can detect. Be able to state both costs rather than only the one you prefer.
Own the operational consequence. Keep the blocking path deterministic, run the refreshed set on a separate non-blocking schedule, and re-read suppressions whenever the set moves, because their keys can move under them.
Decide where the set is defined and who may change it. The tool choice was never the differentiator between teams; the add-on set is, and treating it as a governed artefact is what makes a portfolio-level coverage claim mean anything.
## What is actually being versioned Most pinning arguments are about a build reproducing byte-for-byte. This one is different: in ZAP the **detection logic is not in the program**. The active rule packages, the passive rule packages, the crawlers and the job types a plan may name all ship as separately versioned add-ons, each on its own release line. The add-on set is therefore your **test oracle**, and choosing whether it floats is a choice about whether two runs of the same pipeline against the same build are comparable at all. ## The two positions, stated honestly | | pinned set | refreshed set | |---|---|---| | same commit, scanned twice | same verdict | may differ | | a new finding means | the application changed | the application **or** the tool changed | | coverage of newly published weaknesses | ages | current | | outbound dependency per run | none | marketplace reachable | | failure mode | quiet blind spots | red gates nobody can attribute | Neither column is the right answer. A pinned set that nobody ever refreshes becomes a gate that reliably passes because it stopped looking; a set refreshed on every run turns the gate into a source of unattributable noise, and the predictable response — broadening what the gate ignores — costs more coverage than the refresh bought. ## The shape that usually survives contact 1. **Pin what blocks.** The image a blocking gate runs is a pinned artefact, add-on set included. A developer who cannot reproduce yesterday's verdict stops trusting the gate within a week. 2. **Refresh as a reviewed change.** Bumping the add-on set is a change with an author, a diff and a re-baseline — not a background event. The diff people actually need is which rule packages moved, because that predicts where new findings will appear. 3. **Re-baseline in the same change.** A refresh that adds rules produces findings that are new to you and old to the world. Triaging them in the same change is what keeps the next run's signal clean. 4. **Keep an unpinned run that does not block.** Run the refreshed set on a schedule, publish it as a report, and let it be the early warning for what the next pin will bring. 5. **Record the set with the result.** A scan result that does not say which add-ons produced it cannot be compared with another one. Capturing the add-on listing beside each run costs an argument. ## The suppression layer is where this goes wrong Whatever you use to hold findings at bay — an ignore list, per-rule verdicts, alert filters — its entries are keyed to specific rules. Two failure modes follow directly from add-on versioning: - an entry silences something a refresh **renamed or resplit**, so the finding returns as if it were new; - an entry silences something a refresh **broadened**, so it now suppresses more than the person who wrote it intended. Both argue for the same discipline: every suppression carries a reason and an owner, and a set refresh is the moment to re-read them rather than the moment to add more. The tell that this has gone wrong is a suppression list that only ever grows, which is what happens when refreshes arrive unannounced and the cheapest response to each new finding is another entry. ## The organisational question underneath If each team composes its own image, *"ZAP said it was clean"* means something different in each team, and no portfolio-level statement about coverage is possible. The lever is not the tool choice — everyone already picked the same tool — it is the **add-on set**. Standardising the set, versioning it as an artefact, and giving teams a supported path to add one is what turns many local verdicts into a comparable signal. ## What to say when asked to choose Ask what the result is used for first. A gate that blocks a merge is a **contract with developers** and needs determinism, so pin it. A programme-level coverage report is a **claim about the portfolio** and needs currency, so refresh it. Most arguments about pinning are really arguments about a single pipeline being asked to do both jobs at once, and splitting them dissolves the question.
- Why is this not just an ordinary dependency-pinning argument?Because the dependency is the oracle. Pinning a build library changes how your code is built; pinning ZAP's add-on set changes what your pipeline is able to notice. A regression in a pinned set does not show up as a failure — it shows up as continued success, which is the reverse of how a pinned build dependency fails.
- How would you make a refresh reviewable rather than a background event?Treat the add-on set as a versioned artefact with an owner. A refresh is a change that names which rule packages moved, runs against a known corpus so the delta in findings is visible before it reaches a gate, re-triages the new findings, and re-reads the existing suppressions. That converts "the scan is red today" into a decision somebody already made.
- What goes wrong if two teams pin different add-on sets?Their results stop being comparable while looking identical, because nothing in a scan's output names the set that produced it. Portfolio statements about coverage become unsupportable, and a finding present for one team and absent for another reads as an application difference when it is a tooling difference.
saying these in an interview costs you the question
- Treats pinning as free because a pinned scan keeps passing.
- Refreshes rule add-ons on every run and gates on the result.
- Assumes a new finding always means the application changed.
- Ignores that suppressions are keyed to rules that a refresh can move.
- Lets every team compose its own set and compares the verdicts anyway.