skip to content

How do you review a requirement for testability, and what makes one untestable?

level: middleimportance: should knowfreq 57%

answer

  1. Ask what experiment would settle the rule
  2. Two properties, one about state, one about seeing
  3. Adjectives with no threshold
  4. Nothing tells you the answer is wrong
  5. The output is a rewritten requirement

basics

~20 s

Check two properties: controllability, whether you can put the system into the state the rule describes, and observability, whether you can see the promised outcome. A requirement is untestable when it names no measurable outcome, no reachable precondition, or no way to observe the result.

solid answer

~50 s

I read the requirement asking one question of every rule: what experiment would settle whether this is true? That decomposes into controllability — can I create the precondition, the data, the timing, the participant state the rule assumes? — and observability — when the rule fires, is the outcome visible somewhere I can check it? Untestable requirements fail one of the two: unquantified adjectives with no threshold, an outcome that happens only inside a component with no exposed signal, a state that no supported operation can reach, or a rule whose "correct" answer nobody can name, so there is no oracle. Ambiguity is the neighbouring defect class: undefined terms, missing negative cases, silent boundaries and two rules that contradict each other. The output of the review is written changes to the requirement, plus concrete examples, not a hallway conversation.

code

pseudocode · 14 lines
pseudocode
# before: no threshold, no precondition, nothing to observe
RULE R-11: the eligibility banner shows the participant's current status

# after: controllable precondition, observable outcome, stated oracle
RULE R-11a:
  GIVEN participant P was withdrawn at time T
  WHEN the visit form is opened at any time later than T plus 5 seconds
  THEN the banner reads "withdrawn"
  AND the form records status_decided_at, which is not older than 5 seconds

RULE R-11b:
  GIVEN the banner reads "withdrawn"
  WHEN the user attempts to open the adverse-event section
  THEN the section stays locked and the attempt is recorded

go deeper

for a junior

Learn to spot the obvious offenders: adjectives with no number, rules that only describe the valid input, and boundaries left unstated. Bring those questions to a refinement conversation rather than waiting to hit them later.

for a middle

Explain the mechanics cleanly: controllability, observability and the oracle, plus the ambiguity checklist you apply. Be able to rewrite a vague rule into one with a precondition, a threshold and a visible outcome on the spot.

for a senior

Show the judgement calls — when to ask for a product change so an outcome becomes observable, how you get a sponsor to commit to a number, and how you keep the review producing written changes rather than warm agreement.

for a principal

Own testability as a design constraint: argue for observability being built in rather than retrofitted, decide how much review a programme funds, and explain how you keep the practice alive when delivery pressure rises.

## The one question behind the whole review For each rule in a requirement, ask: **what experiment would settle whether this rule holds?** If you cannot describe the experiment, the requirement is not testable yet, and that is a defect in the requirement — logged, not tolerated. The question decomposes into two properties that are worth naming explicitly in an interview. **Controllability** — can I put the system into the state the rule assumes? That covers data (does a participant in this state exist, or can I create one?), timing (can I make the clock, or a scheduled refresh, land where the rule needs it?), configuration, and role or permission. A rule about behaviour in a state that no supported operation can reach is untestable in practice even when it is perfectly clear. **Observability** — when the rule fires, can I see the outcome? A rule whose entire effect happens inside a component with no exposed signal — no visible field, no record, no event, no log entry — cannot be confirmed or refuted. Very often the fix is not a test technique but a small product change: expose the value, stamp the record, emit the event. A third, quieter property is the **oracle**: the thing that tells you the observed behaviour is wrong. Sometimes the requirement names it (a stated threshold, a formula, a reference table). Sometimes nobody can say what the right answer is, which means the requirement has deferred the hard decision and the review has to force it. ## What untestable looks like in practice - **Unquantified adjectives.** "Loads quickly", "handles large files", "behaves appropriately". No threshold, no condition, no measurement point. - **Undefined terms.** A word carrying business meaning that two people in the room define differently. - **Missing negative cases.** The rule says what happens when the input is valid and is silent about everything else. - **Silent boundaries.** "Participants over 18" — is 18 included? At what moment is age computed? - **Unreachable preconditions.** The state exists in the domain but no operation produces it. - **Invisible outcomes.** The effect is real but nothing exposes it. - **Contradictions.** Two rules in the same document give different answers to the same case. ## A worked review A clinical-trial data capture form has 27 fields and an eligibility banner. Three requirements come to review: > R-11: *The eligibility banner must show the participant's current status.* Controllability is fine — a participant's status can be changed. Observability is the problem: status is served from a cached projection, so the banner may show *enrolled* seconds after a withdrawal, and nothing on the screen says how old the value is. A stale-cache read here is not just a display bug: the adverse-event section unlocks off that banner. The review produces two changes — a freshness rule stated in seconds, and a requirement that the record carry the timestamp of the status it was decided from, which is what makes the rule observable at all. > R-12: *Data entry must be fast enough for site staff.* No threshold, no operation named, no measurement point. Rewritten with the sponsor: the visit form saves and returns confirmation within 2.4 seconds at the 95th percentile with 40 concurrent sites entering data. > R-13: *Adverse events must be reported promptly.* "Promptly" is an adjective; worse, nobody in the room can state the oracle. The review forces the decision: a serious adverse event must be transmitted within 24 hours of entry, and a queued transmission older than that must appear on a monitoring view — a rule with both a threshold and a place to look. ## Ambiguity review is the sibling activity Testability asks *can this be checked?*; ambiguity asks *does everyone read it the same way?* In practice you do both in one pass, using a small checklist: undefined term, vague quantifier, missing negative case, unstated boundary, ambiguous pronoun, conflicting rule, unstated assumption about state or ordering. A checklist matters because free-form reading finds whatever the reviewer happens to notice, and different reviewers notice different things. ## Running the review so it produces something Read the item beforehand and arrive with a written list of questions — the session is for decisions, not for first contact. Log each finding as a defect against the requirement so the effort is visible in the same place as code defects. Convert each resolved ambiguity into a concrete example with an expected outcome, because an example is unambiguous in a way that prose is not, and those examples become the acceptance criteria the developer builds against. Time-box the session and stop when the list is exhausted rather than filling the slot. The most common failure is a review that ends in agreement but changes nothing: no rewritten rule, no logged finding, no example. If the requirement is unchanged afterwards, the review's only output was a shared feeling.

  • A rule's outcome happens entirely inside one component and nothing exposes it. What do you ask for?
    A change to the product, not a cleverer test. Ask for the smallest durable signal that makes the outcome visible: a stored field, a record stamped with the value the decision used, or an event carrying the decision and its inputs. That signal serves support and incident diagnosis too, which is usually how you justify it. Avoid solving it by reaching into internals from a test, which couples the check to a structure that will change.
  • The business owner refuses to put a number on a performance rule. How do you unblock the review?
    Stop asking for a number and ask for a decision instead: what response would make a site coordinator complain, and what would make them abandon the entry? That produces a boundary you can write down. Offer a measured starting value from current behaviour and mark it provisional with a review date. A provisional threshold is checkable and can be revised; an adjective can be neither met nor missed.
  • How do you keep a testability review from turning into a design meeting?
    Keep the question on what must be true, not how to build it. When a solution debate starts, record it as an open design item with an owner and move to the next rule. The reviewer's job is to leave with rules that can be confirmed or refuted; if the room needs a design decision to make a rule checkable, that is a finding to hand over, not an agenda item to absorb.

saying these in an interview costs you the question

  • Treats vague adjectives as acceptable until testing starts
  • Says untestable requirements are fine, just test manually
  • Never asks whether the outcome is observable anywhere
  • Reviews by reading through with no checklist
  • Ends the review with agreement but no rewritten rule
  • Cannot say what would prove the behaviour wrong

context

resolving links…