skip to content

A passing run cannot show whether a behaviour is guaranteed or incidental. How do you decide which one you are asserting on?

level: seniorimportance: nice to knowfreq 26%

answer

  1. One run is one data point
  2. What would have to change?
  3. Look for a written promise
  4. Currently faster is not promised

basics

~20 s

Green proves only that the behaviour held once. Check the contract, the interface's stated promises and the state machine, and ask whether anything forbids the opposite. If nothing does, the behaviour is incidental and the case should not assert on it.

solid answer

~50 s

A passing run is evidence about one build on one day, not a statement about what the system promises. The check is a documentary one: does the specification, the interface description or the state machine forbid the behaviour you observed from changing? If it does, the behaviour is a **guarantee** and asserting on it is legitimate. If nothing forbids it, if the value appeared first only because that path is currently faster, or the collection came back in that order only because that is how the store happened to return it, then the behaviour is **incidental**, and a case built on it is a trap that fires on some later, entirely correct change. Where the behaviour genuinely matters, do not assert it quietly: get it written into the contract, then assert it as a promise the owners have agreed to keep.

code

pseudocode · 11 lines
pseudocode
# incidental: nothing forbids the store returning these in another order
assert results[0].id == "A-17"

# guaranteed: the interface documents a sort by creation time
assert is_sorted_by(results, "createdAt")

# incidental: the audit path is merely faster today
assert audit_row_exists(orderId) and not receipt_exists(orderId)

# guaranteed: the contract says a receipt follows capture
assert not (receipt_exists(orderId) and not capture_recorded(orderId))

go deeper

for a junior

Be ready to say where your assertion's expectation came from. If the only reason you expect a value is that you saw it once while running the code, say so: that is an observation, not a rule the system has agreed to keep.

for a middle

Explain the difference in mechanics. A guarantee is enforced by something, a stated contract, a state machine or a constraint, while an incidental behaviour is whatever the current implementation happens to do first or fastest. Show how you check which one you have.

for a senior

Show the production judgment: a case you found asserting an implementation detail, how a correct change broke it, and how you decided whether to fix the case or promote the behaviour into the contract instead.

for a principal

Own the standard that keeps a suite honest: every assertion traceable to a stated promise, gaps raised as contract decisions rather than encoded in silence, and a review habit that asks what promises this before it asks whether it passes.

## Green is a record of observations, not a statement of promises A passing run establishes exactly one thing: on this build, on this machine, at this moment, the system did what the case expected. It says nothing about whether the system was **obliged** to. Two very different situations produce that identical green result: - The behaviour is **guaranteed**: something enforces it, and changing it would be a defect the owners would fix. - The behaviour is **incidental**: the current implementation happens to do it, nothing forbids the alternative, and a correct change may flip it tomorrow. Asserting on an incidental behaviour builds a trap. The trap does not fire during the change that introduced it; it fires months later, on a change that is entirely correct, and it presents itself as a regression. The cost is not just the wasted investigation. It is the pressure to "fix" the product back to the behaviour the case expected, which is how an accidental implementation detail quietly becomes a permanent constraint nobody chose. ## Telling them apart | | Guaranteed | Incidental | | --- | --- | --- | | Enforced by | A stated contract, a state machine, a constraint, a validation rule | Nothing; it is a consequence of the current implementation | | Evidence for it | A clause you can point at | Runs you have watched | | If it changed | The owners would call it a defect | The owners would call it fine | | A failing assertion means | The product is wrong | The case was wrong | | So the case should | Assert it | Not mention it | The right-hand column is uncomfortable because incidental behaviours are often extremely stable. Stability is not the test. The test is what happens when someone asks the owners to change it. ## The warrant test Before an assertion goes in, answer three questions about it: 1. **What promises this?** Name the source out loud: the interface description, the specification, an explicit invariant, a state machine with no transition that could produce otherwise. "I ran it and saw it" is not a source. 2. **What would have to change for this to become false?** If the answer is an implementation choice nobody has committed to, such as which of two paths is currently faster, or how a store happens to return unsorted results, the behaviour is incidental. 3. **Would the owners accept a change that breaks it?** If they would shrug and merge it, your case is asserting something they never agreed to keep. ## Behaviours that get asserted without a warrant - the order a collection comes back in when the request did not ask for an order - which of two independent deferred effects becomes visible first - the format of a generated identifier, or the fact that identifiers happen to increase - a default value the current implementation supplies where the contract states none - the exact wording of an error description where only the error's category is specified - how many attempts the product's own redelivery path makes before giving up - completion "within about two seconds", where no deadline was ever stated Every one of these is a real, reproducible behaviour. None of them is promised. ## When the warrant is missing but the behaviour matters Sometimes the honest answer is that the behaviour genuinely should be guaranteed and simply is not written down. That is a valuable finding, and the response is to raise it rather than to encode it silently. Ask the owners whether they intend to promise it. If they do, get it stated, then assert it as a promise with a name. If they will not, leave the case asserting only what is promised, and record the assumption you deliberately chose not to encode. Silently asserting a behaviour is the one option that helps nobody: it neither secures the promise nor leaves the design free. ## Record the warrant beside the assertion The practice that makes this survive a team is cheap. Put the warrant next to the claim: a short note naming the clause or the state-machine edge the assertion rests on, and a case name that says which promise is under test rather than which values appeared. A reviewer can then argue with the warrant instead of arguing with the value, and the next person to see the case fail has the information needed to decide the real question, which is whether the promise moved or the product broke. Assertions with no recorded warrant are precisely the ones that get "repaired" by loosening them, until the suite is green, unfalsifiable, and worth nothing.

  • Where do you record the warrant so the next reader does not have to rediscover it?
    Beside the assertion. A short note naming the clause or the state-machine edge the claim rests on, plus a case name that says which promise is under test rather than which values appeared. A reviewer can then challenge the warrant instead of the value, and someone seeing the case fail knows whether the promise moved or the product broke. Assertions with no recorded warrant are the ones that get loosened when they finally break.
  • The behaviour you want to assert on is real and stable, but nowhere in the contract. What do you do?
    Treat the gap as the finding. Ask the owners whether they intend to promise it; if they do, get it stated and then assert it. If they will not, leave the case asserting only what is promised and record the assumption you chose not to encode. The wrong move is to assert it silently, which neither secures the promise nor leaves the design free to change.

saying these in an interview costs you the question

  • Treats a hundred green runs as proof of a guarantee
  • Asserts on the order a collection happened to come back in
  • Says a behaviour is guaranteed because the code currently does it
  • Loosens a broken assertion instead of asking what was promised
  • Cannot say which document or clause an assertion rests on