skip to content

A release with near-total code-execution coverage fails acceptance because an agreed behaviour was never built — why did that figure not warn anyone?

level: seniorimportance: should knowfreq 52%

answer

  1. Missing code cannot be unexecuted code
  2. Cutting work can raise that figure
  3. The agreed list is the other denominator
  4. Checks written from code confirm the misreading
  5. Ask it at sign-off, not after

basics

~20 s

Because that figure divides by the code that exists. Behaviour nobody implemented contributes no code, so it cannot appear as unexecuted code — cutting work can even raise it. Only the requirement-side denominator holds the missing behaviour.

solid answer

~50 s

The code-executed figure measures the code that was written, so an agreed behaviour that was never implemented is simply absent from its denominator — there is nothing to leave unrun, and dropping work can push the figure up rather than down. The requirement-confirmed figure divides by the agreed list instead, so the missing behaviour sits there with nothing confirming it and drags the figure down until it is built and checked or removed from the agreement deliberately. Acceptance failures of this shape usually take one of three forms: the behaviour was lost during a re-plan, it was built to a misreading and then confirmed by checks written from the implementation rather than from the agreed wording, or only one of its acceptance criteria was implemented. The fix is to ask the requirement-side question at sign-off, not to raise the code-side target.

code

pseudocode · 6 lines
pseudocode
// R2 was agreed in planning and never implemented
codeWritten        = [moduleA, moduleB]        // no module for R2
codeExercised      = [moduleA, moduleB]        // executedRatio = 1.00

agreedRequirements = [R1, R2, R3]
confirmed          = [R1, R3]                  // verifiedRatio  = 0.67

go deeper

for a junior

Recall the key asymmetry: a figure measured over the code that exists cannot report behaviour that was never coded. Being able to say that one sentence is enough at this level.

for a middle

Explain why cutting scope can raise the code-side figure, and what the requirement-side denominator does differently. Be able to give the three shapes: never built, built to a misreading, built in part.

for a senior

Show the diagnosis you would run and the change you would make — where checks are written from, what is asked before sign-off, and why raising an execution target is the wrong response to this failure.

for a principal

Own the trade-off between the cost of maintaining links from agreed behaviour to confirmations and the risk of discovering missing behaviour only at acceptance, and be able to say which products justify which level of that effort.

A high code-executed figure and a failed acceptance run are not in contradiction, because the two things measure different populations. The code-executed figure divides by the code that exists. Behaviour that was agreed and never built contributes no code, so it cannot lower that figure. In fact removing work from the plan tends to move it *up*: fewer branches exist, so the ones the suite does reach make up a larger share. A team can raise the number by not building things. The requirement-verified figure divides by the agreed list instead. A behaviour that nobody implemented sits in that denominator with nothing confirming it, so it pulls the figure down and keeps pulling until either the behaviour is built and checked or somebody removes it from the agreement on purpose. ## Three shapes of the same failure Acceptance rejecting a release that looked well covered almost always fits one of three shapes: 1. **Never built.** The requirement was agreed, then lost — dropped in a re-plan, buried in a ticket nobody groomed, or assumed by each side to be the other's. No code, no unexecuted code, no signal from the code-side figure at all. 2. **Built to a misreading.** The behaviour exists, but it does what the engineer understood rather than what was agreed. If the checks were written from the implementation, they exercise the misreading thoroughly and pass. Both figures look healthy and the product is still wrong, which is the case where writing checks from the agreed wording rather than from the code is the only defence. 3. **Built in part.** One acceptance criterion of three is implemented. The implemented part is exercised, the figure is high, and the requirement is only fractionally satisfied — which is why linking at criterion granularity rather than at whole-requirement granularity changes what the number can tell you. | The question you are really asking | The figure that answers it | | --- | --- | | Does code exist here that nothing exercises? | code-executed | | Has everything we agreed got something confirming it? | requirement-verified | | Was this behaviour built at all? | requirement-verified | | Is this old area safe to change? | code-executed | | Are the checks any good? | neither | ## When the requirement-side figure is the question worth asking Reach for it when the risk in front of you is about *the agreement*, not about the code: - **At a feature-complete or sign-off conversation**, where the claim being made is that what was asked for exists and has been confirmed. - **After a scope change mid-release**, when behaviour was added or cut and the plan has not caught up with what the suite confirms. - **When acceptance and the engineering team disagree about done**, because the disagreement is usually about the denominator: one side is counting code, the other is counting promises. - **Where evidence must be produced later** — an audit, a customer commitment, an incident review — and somebody will ask which confirmation covered which stated behaviour. Reach for the code-side figure instead when the risk is about the code you already have: an area about to be refactored, a long-lived component nobody has touched in a year, a decision about what to delete. ## What the requirement-side figure does not settle It is still an existence count. A requirement it marks verified might be confirmed by a single check that walks the easiest possible route, and a requirement it marks unverified might be covered perfectly well by a check nobody remembered to associate with it. It also inherits the quality of the agreed list: if the requirement was vague, "verified" means only that somebody interpreted it and wrote something. So the useful answer to the failed acceptance run is not "raise the target". It is: state which agreed behaviours had nothing confirming them, say whether that is because the behaviour is missing or because the confirmation was never associated with it, and change where the checks are written from — the agreed wording — so that the next release's figure means what people think it means. A candidate who responds by proposing a higher code-executed threshold has not understood which denominator failed.

  • How can cutting scope make the code-executed figure go up?
    The figure is exercised code divided by written code. Removing planned work removes code that would have been written, and often that code would have been the hardest to reach — failure paths, edge branches. Fewer hard branches in the denominator means the ones the suite does reach form a larger share. The number improves while the product does less.
  • Both figures were high and acceptance still rejected the release. What happened?
    Most often the behaviour was built to a misreading and the checks were written from the implementation rather than from the agreed wording. Every check confirms what the code does, so both inventories look full while the product does the wrong thing. The defence is writing checks from the agreed statement, and having someone other than the implementer read it.
  • Would a higher code-executed target have prevented this?
    No. The target applies to code that exists, and the failure was code that does not. Raising it adds work on the branches you already have while leaving the missing behaviour exactly as invisible as before. The change that helps is asking, before sign-off, which agreed behaviours have nothing confirming them.

saying these in an interview costs you the question

  • Blames the measurement instead of the behaviour never built
  • Claims unwritten code drags the executed figure down
  • Writes checks from the implementation and calls requirements confirmed
  • Proposes a higher execution target as the fix
  • Treats an acceptance rejection as an environment problem