How should blocked and not-run cases count against a pass-rate exit criterion?

## Why the denominator decides everything A pass-rate exit criterion is usually written as "at least *N* percent of cases pass". The sentence is silent about what the percentage is *of*, and that silence is where the criterion fails. Take an 11-person team finishing a cycle on a document e-signing flow. The plan holds **348** cases. At the end of the cycle: **274** passed, **17** failed, **39** are blocked, **18** were never started. Executed is therefore 291. - Pass rate over executed: 274 / 291 = **94.2 %** - Pass rate over planned: 274 / 348 = **78.7 %** The exit criterion said 95 %, so both numbers miss it — but they miss it by very different amounts, and only one of them is honest about the 57 cases nobody has an answer for. Worse, the executed-based figure has a perverse property: if the environment breaks further and 20 more cases become blocked, the executed-based rate goes *up*. Any criterion whose number improves when less work is done is measuring the wrong thing. ## Blocked and not-run are different problems They are often lumped together as "not executed", which loses the information that matters. **Blocked** means the case could not run for a reason outside its own subject: a dependency is down, prerequisite data does not exist, or an earlier defect stops you reaching the state the case starts from. Blocked cases are a readiness failure, and they cluster — one broken collaborator can take out a whole area at once, which is exactly why a raw count understates the damage. In the e-signing cycle, 34 of the 39 blocked cases sat behind a single unavailable callback, so the blockage was one problem wearing 34 hats. **Not-run** means the case was runnable and nobody ran it — time ran out, or someone consciously deprioritised it. That is a scope decision, and it should be an explicit one, because a not-run case in a high-risk area is a different conversation from a not-run case in an area last changed nine months ago. ## Not all unknowns weigh the same Counting cases treats every unknown as one unit, which is wrong in an obvious way: a blocked case covering the signing-limit boundary is not equivalent to a blocked case covering a tooltip. The workable practice is to tag cases by risk band and phrase the exit criterion per band — for example, no blocked or not-run cases in the top band at all, and a cap on the middle band — rather than a single global percentage. This keeps the criterion decidable while stopping it from being satisfied by unknowns in exactly the places that matter. The same logic applies to failures. A cycle with 17 failures may be in far better shape than one with 4, if the 17 are one defect surfacing seventeen times and the 4 are four independent defects in payment-critical paths. That is why the pass rate is never the whole exit story — it is one input beside the open-defect picture per severity band. ## What a re-test does to the arithmetic When a fix lands mid-cycle, the confirmation run turns some failures into passes, but it can also invalidate earlier passes on the same build if the build changed. Two disciplines keep the number meaningful. First, the pass rate is always **as of a named build** — a rate computed across three builds describes nothing. Second, a case that passed on an earlier build and has not been re-run on the current one is *not-run on the current build*, even though a naive report will show it green. In the e-signing cycle this is where an off-by-one at the signer-limit boundary nearly escaped: the boundary case had passed on the previous build, the fix for an unrelated defect changed the limit check, and the case was never re-executed. It showed as a pass on the summary and as not-run on the build. ## How to write the criterion A usable exit criterion on this axis names four things: the denominator, the build, the caps on unknowns, and the treatment of unknowns that remain. > On the release-candidate build, at least 95 % of *planned* cases pass; zero blocked or not-run cases in the top risk band; at most 5 % blocked or not-run overall; every remaining unknown listed by area with an accept-or-run decision recorded against it. The last clause is the one people skip and the one that does the work. A gap that has been named, argued over and accepted by someone accountable is a managed risk. The identical gap left inside a rounded percentage is a surprise waiting for production. ## What to say in the room If you are asked this question, the answer that lands is not a formula but a refusal to report one number: state the pass rate with its denominator and its build, state blocked and not-run separately with their causes, and state which unknowns sit in the areas that would hurt. Then say what you want to do about each one. That is the whole of it, and it is why the question gets asked — it separates people who report a percentage from people who report a position.