skip to content

You own the release gate for AI features across several product teams. Every additional finding class the gate blocks on buys real release delay, and no class ever reaches zero occurrences. How do you decide how many classes the gate blocks on?

level: principalimportance: should knowfreq 40%

answer

  1. would you truly hold the launch
  2. small set held firmly
  3. size to fix throughput
  4. overrides and escapes as feedback
  5. publish what went untested

basics

~20 s

Block only on classes where one instance is unacceptable and the harm cannot be undone. Size that set to what fix teams can actually clear inside a release cycle, or the gate gets routed around. Everything else takes a control or a named acceptance. Revisit the set from what escaped, not from opinion.

solid answer

~50 s

Two constraints set the size, and they pull opposite ways. From above: a class belongs in the blocking set only if the organisation would genuinely stop a launch for one instance — irreversible harm, another tenant's data, an agent action with external effect, a category leadership has committed against. If you would not actually hold the release, it is not a blocking class, and pretending otherwise teaches everyone the gate is negotiable. From below: the set has to be clearable. If it generates more work than fix teams absorb in a cycle, the gate becomes a queue, and teams schedule engagements late, scope them narrow, or seek exceptions. So: a small stable set sized against demonstrated fix throughput, everything else on controls and dated acceptances. Then make it empirical — escapes into production move a class up; overrides that keep recurring mean the class was never really blocking and should be honestly demoted.

go deeper

for a junior

Recognises that blocking everything is not viable and that some findings ship with controls or acceptance.

for a middle

Separates consequence from frequency, and can argue why a rare finding might block while a common one does not.

for a senior

Ties the blocking set to fix throughput and watches overrides and acceptance backlog as evidence the set is miscalibrated.

for a principal

Designs the gate as a feedback system across teams, defends a small firmly-held set against the pressure to expand after every incident, and makes untested surfaces visible next to the verdict.

### What is being sized The **blocking set** is the enumerated list of finding classes for which the gate refuses a release, as opposed to permitting a compensating control or a signed acceptance. It is a governance artefact, not a test artefact: the tools decide what is found, the blocking set decides what that finding does to a launch date. The strongest answers treat the gate as a system with feedback rather than as a policy document, because a blocking set is only as real as the organisation's willingness to pay for it. ### Derive from consequence, not from the tool catalogue The tempting shortcut is to build the class list out of what the scanners can currently detect, because those classes arrive with numbers attached. That inverts the logic twice. It makes the gate a function of tool coverage, so the set changes whenever a scanner adds probes rather than when the business's risk changes. And it silently declares every harm nobody can measure to be acceptable, which is the largest category. Derive the classes from what the organisation cannot survive one instance of, then ask separately, per class, how well it can be tested for — and record the untestable ones as low-confidence blocking classes, because a class nobody can measure is aspiration rather than enforcement, and saying so out loud is what keeps the gate honest. ### Size against demonstrated throughput Run the counterfactual before adopting a set: take the last several releases, replay the candidate blocking set over the findings those engagements produced, and count how many launches it would have stopped and for how long. Compare that to what fix teams actually cleared in the same window. A blocking set that generates a standing backlog is indistinguishable, from a product team's point of view, from an arbitrary release freeze, and the organisational response is predictable and fast: an exception process, engagements scheduled after code freeze, scope trimmed so the gate sees less surface. ### What it costs Every blocking class carries three costs. **Delay** — days to weeks per triggered block, paid by a team that did not choose the class. **Triage load** — every class needs a stable definition, a worked example, and someone who adjudicates edge cases, which is a recurring commitment of the programme's own hours, not a one-off drafting exercise. **Credibility** — the scarcest resource. A set that fires more often than the organisation will honour spends credibility on every override, and credibility does not recover when the set is later trimmed. This is why a small set held firmly outperforms a large set applied loosely, even when the large set is nominally stricter. ### Where the number misleads Three metrics get quoted about gates and all three deceive. **Block rate** — a low block rate is read as a healthy product, but it is equally consistent with a gate that is routed around; look at exception volume in the same period before believing it. **Zero blocks in a year** — presented as evidence the bar is met, it is more often evidence that the enforced set is not the stated set. **Mean time to fix** — it scores teams whose staffing you do not control, and it improves when the gate stops raising hard findings, so it can move in the right direction for the worst possible reason. The number that actually tells you the set is miscalibrated is not a rate at all: it is *where* the overrides concentrate. Overrides piling on one class mean the class is either too aggressive or badly defined, and the fix is redrawing its boundary rather than changing the threshold. ### What you would check Four signals, and only one of them justifies expanding the set. Overrides concentrated on a single class: redraw that class. A gate that has never blocked, alongside an exception path carrying releases: the stated set is not the enforced set, so restate it honestly. Acceptances accumulating with no expiries: the residual is growing invisibly and the gate is reporting a snapshot of a rising line. A production incident whose class the gate had put on the acceptable side: that, and essentially only that, is direct evidence the consequence judgement was wrong and the set should grow. Finally, publish what the gate does not cover. Every verdict is scoped to the surfaces an engagement reached; the honest artefact beside the pass/fail is the list of what went untested this cycle. Expect the errors to be asymmetric: blocking too much costs delay you can see and argue about, blocking too little costs harm you learn about later. That asymmetry argues for a small set held firmly — not for a large set that everyone has learned to negotiate.

  • How do you know the blocking set is too large?
    Overrides become routine, engagements get scheduled late or scoped narrow, and an exception path carries more releases than the gate does. Those are organisational symptoms, and they show up before any incident does.
  • What single signal justifies adding a blocking class?
    A production harm whose class the gate had put on the acceptable side. That is direct evidence the consequence judgement was wrong, as opposed to a near miss or an internal argument.

saying these in an interview costs you the question

  • Deriving blocking classes from what the tooling happens to detect, which makes coverage gaps look like clean results.
  • A blocking set nobody has ever enforced, alongside an exception process everyone uses.
  • Expanding the gate after every incident with no removal path, until teams schedule engagements to avoid it.
  • Presenting a gate verdict without saying which surfaces were never tested.
  • Blocking on a class the programme has no way to test for, and calling that enforcement.

context