skip to content

When traceability links are maintained by hand, what does a gap and orphan analysis over them actually measure?

level: seniorimportance: nice to knowfreq 24%

answer

  1. It reads the record, not the system
  2. Findings track how links are kept
  3. No findings is not the same as verified
  4. One error is loud, the other silent
  5. Sample by hand and quote the error rate

basics

~20 s

It measures the link set, not the system. With links entered by hand the findings track how well people keep those links current, so a clean result means nobody has recorded a gap, not that everything is verified.

solid answer

~50 s

The analysis reads a record of links. When people type those links in, the findings mostly measure link-keeping discipline, and the count moves when that discipline moves rather than when verification changes. Two errors follow with opposite costs. A **false gap** — a case exists but was never linked — sends somebody to write a case that already exists, and it self-corrects because that person discovers the truth. A **false verification** — a link surviving a case that was deleted or no longer checks the criterion — is silent, and nothing dispatches anyone. Records age towards the second, because links are entered when work finishes and rarely revisited when it changes. Validate before quoting: sample a handful of requirements and check them by hand, count links whose target no longer resolves, and compare link age against the last change to the statement.

code

pseudocode · 12 lines
pseudocode
sample = randomSample(requirementsInScope, 10)
mismatches = 0

for req in sample:
    claimed = req.linkedCases                 // what the record says
    actual  = findCasesByHand(req)            // what a person can find
    if claimed != actual:
        mismatches = mismatches + 1

recordErrorRate = mismatches / size(sample)

report(findings, qualifier = recordErrorRate)  // never report findings alone

go deeper

for a junior

Understand that these links are typed in by people and can be wrong in both directions, so a finding is something to check rather than a fact to act on immediately.

for a middle

Explain the two error directions and why they cost differently: a missing link wastes effort loudly, while a stale link makes a false claim that nobody is looking for.

for a senior

Demonstrate that you validate the instrument. Sample requirements by hand, resolve link targets, compare link age against the last change to the statement, and quote findings with the error rate you measured.

for a principal

Own what the organisation is allowed to claim from this record, how much upkeep it is worth funding, and what a decision needs when the evidence behind it is maintained by people rather than produced by the work.

## What the analysis is actually measuring Gap and orphan analysis reads a record of links, not a running system. When the links are produced by hand — somebody types the requirement identifier onto a case, or adds a row when they remember — the findings measure **how well those links are kept**, and only indirectly anything about whether the product is verified. The distinction sounds pedantic until you use the result to make a decision. A clean record supports one honest sentence: *nobody has recorded a gap.* It does not support *everything is verified*, and the gap between those two statements is exactly the error rate of the people maintaining the links. This is not an argument for skipping the analysis. It is an argument for knowing which of two different quantities you have measured, because they move for different reasons and they cost different amounts when wrong. ## Two errors with opposite costs | | False gap | False verification | |---|---|---| | Cause | A case exists but was never linked | A link survives a case that was deleted, renamed, or no longer checks the criterion | | What the report says | "Unverified" when it is verified | "Verified" when it is not | | Who notices | Whoever is sent to write the case, quickly | Nobody, until a defect arrives from that area | | Immediate cost | Wasted effort, a duplicate case, and eroded trust in the report | None visible, which is the problem | | Eventual cost | Small | The release decision was made on a claim that was not true | Both errors come from the same source, so they arrive together, and they push the finding count in opposite directions. A record maintained by an attentive team looks the same as one nobody has touched: neither shows findings. That symmetry is why a low gap count is not evidence of health. ## Why the finding count tracks discipline - The count rises when someone starts a link-hygiene push, and falls when they stop — neither of which is a change in how much of the product is verified. - A team reorganisation, a change in how requirements are identified, or a bulk rename can move the count more than a quarter of engineering work does. - Links are entered when the work is done and rarely revisited when the work is *changed*. So the record ages towards over-claiming: stale links accumulate, and stale links are the expensive error. - The one direction that self-corrects is the false gap, because someone is dispatched to fix it and discovers the case already exists. Nothing dispatches anyone at a false verification. ## Validating the record before you quote it 1. **Sample and check by hand.** Take a small random set of requirements, look for the cases yourself, and compare with what the record claims. Ten checks give you a usable error rate; the point is to have a number, not a perfect one. 2. **Resolve every link target.** Count the links that point at a case that no longer exists or no longer runs. Those are the ones already known to be wrong, and they are a floor under the true rate. 3. **Age the links against the statements.** A link that predates the last change to its requirement is a candidate for a stale claim, because nobody revisited it when the meaning moved. 4. **Track dismissals.** How often a finding is waved away tells you whether people believe the record. 5. **Report the analysis with its own error rate attached.** "No gaps found in a record whose sampled error rate was one in ten" is an honest sentence; "no gaps" alone is not. ## What you may conclude You may conclude that a finding is worth investigating, because a reported gap is cheap to check and frequently real. You may conclude that findings clustered in one area point at either a weakly tested area or a poorly maintained corner of the record, and that either one is worth someone's time. You may not conclude that a clean record means verified work, and you should be explicit about that when the record is used as evidence for anything, because the reader will otherwise supply the stronger reading themselves. The mature position treats a hand-maintained record as a **claim made by people**, subject to the error rate of people, and holds it to the same standard as any other measurement: state the method, state the uncertainty, and never let a number that measures bookkeeping be quoted as a number that measures the product.

  • Why does a hand-maintained record drift towards over-claiming rather than under-claiming?
    Links are added when a piece of work finishes and are rarely revisited when the work changes. Deletions, renames and reworded statements leave the link in place pointing at something that no longer verifies what it claims. Under-claiming self-corrects, because someone sent to close a false gap finds the case already exists; over-claiming has nobody looking for it.
  • Findings cluster in one area of the product. What are the two readings, and how do you tell them apart?
    Either that area is genuinely under-verified, or its corner of the record is poorly maintained. Check a few of the reported statements by hand: if the cases exist and only the links are missing, it is bookkeeping; if they do not exist, it is verification. Both are worth someone's time, but they go to different people.

saying these in an interview costs you the question

  • Reads a record with no findings as proof that work is verified
  • Quotes a finding count without any sense of the record's error rate
  • Assumes every reported gap is real before checking one
  • Treats a rise in findings as a fall in product quality
  • Never checks whether a link still points at a case that exists