skip to content

Why hand a drafting model the acceptance criteria rather than the implementation when drafting a test?

level: middleimportance: must knowfreq 58%

answer

  1. Where do the expected values come from?
  2. A test needs a source outside the code
  3. Code in, code out: assertions mirror branches
  4. Criteria give behaviour, the interface gives shape

basics

~20 s

Acceptance criteria describe the behaviour a test must prove; the implementation describes what the code happens to do today. A model given the code writes assertions that mirror it, so the case stays green even when the behaviour is wrong.

solid answer

~50 s

A test is only useful if it can **disagree with the code**, which means its expected values have to come from somewhere other than the code. Hand a drafting model the implementation and it has exactly one source of truth, so the draft restates branches, thresholds and message strings as they stand. Those cases pass on day one, survive a real defect that the code expresses consistently, and break on every harmless refactor. Hand it the requirement and its acceptance criteria and the expected values come from outside the system, so a mismatch is news. You still supply the *contract* - operation names, parameter and response shapes, the route and the identity the case runs as - because a draft has to call something real. The split to remember: **shape may come from the code, values may not.**

code

pseudocode · 20 lines
pseudocode
# context bundle A - the implementation is pasted in
given_to_model:
    body_of discountedTotal(subtotal, coupon):
        if coupon.kind == PERCENT: total = subtotal * (1 - coupon.rate)
        if total < 0: total = 0

draft_returned:
    assert discountedTotal(subtotal=100, rate=0.10) == 90
    # the same arithmetic, restated; it cannot disagree with the body

# context bundle B - the criterion is pasted in, the body is not
given_to_model:
    criterion: "a percentage coupon reduces the order total by its
                percentage, applied before shipping, and never below zero"
    interface: discountedTotal(subtotal, coupon) -> Money

draft_returned:
    assert discountedTotal(subtotal=100, rate=0.10) == 90
    assert discountedTotal(subtotal=10,  rate=1.50) == 0
    assert shipping_is_added_after_discount() == true

go deeper

for a junior

Be ready to say what a test is for: proving the behaviour somebody asked for, not describing the code. Know that expected values belong to the requirement, and that a case written from the code alone can never disagree with it.

for a middle

Explain the mechanism. With only the body in its context a drafting model has one source of truth, so it restates branches and constants. Then show the split you supply instead: criteria for behaviour, the interface for shape, and why that division works.

for a senior

An interviewer expects the failure mode from real suites: change-detector cases that break on every restructure and stay green through genuine defects. Talk about how you notice them and how you repair the context that produced them, rather than patching one case.

for a principal

Own the policy angle: where acceptance criteria live so that drafting can reach them, who is accountable when they are too vague to draft from, and whether a suite full of code-mirroring cases is a tooling problem or a requirements problem.

## The problem a test is supposed to solve A test is only worth running if it can disagree with the code. That is the entire mechanism: you write down, from a source independent of the implementation, what the system is supposed to do; the run compares the system against that record; a mismatch is news. Remove the independence and the machinery still runs, still turns green and still reports coverage - and can no longer tell you anything you did not already know. **Every expected value in a test is a claim about intent, and intent does not live in the code.** This is why the context handed to a drafting model matters more than the wording of the request. A model can only assert from what it was given. Give it the implementation and the only statement of intent in the room is the code, so the expected values are derived from the code. Give it the requirement and the acceptance criteria and the expected values are derived from the requirement. ## What implementation-as-context actually produces The drafts are not obviously bad, which is the whole difficulty. They compile, they run, they pass, and they look thorough. What you get: - **Restated branches.** Every condition in the body becomes a case, including the ones that exist only because of how the code was factored. That is coverage of the code, not of the behaviour. - **Constants copied forward.** A retention window is asserted as fourteen days because the body says `RETENTION_DAYS = 14`, not because anyone required fourteen. If the requirement said thirty and the code is wrong, the case now protects the defect. - **Tautological assertions.** The strongest form: the expected value is computed the same way the system computes it, so the assertion is true by construction. - **Error text pinned.** Message strings become expectations, so a copy edit breaks the suite. - **Silence about what is absent.** A behaviour the code never implemented cannot appear as a case, because it was not in the context. Missing functionality is invisible to a drafter that only saw the code. The combined result is a **change detector**: a suite that fails on every harmless refactor and passes through real defects, because a real defect is usually the code doing something consistent and wrong. ## What criteria supply that code cannot | Context supplied | Source of expected values | Fails when | Blind to | |---|---|---|---| | The implementation | The code itself | The code is restructured | Any defect the code expresses consistently | | The requirement and its acceptance criteria | The specification | The behaviour departs from what was asked | Anything the criteria do not state | The right-hand column is the honest cost. Criteria-derived cases are only as good as the criteria, and a vague criterion drafts a vague case. That is a real limit, and it is a better limit than the alternative because it is *visible*: you can read the criteria and see the gap. You cannot read a code-mirroring suite and see that it has never once disagreed with anything. ## What you still hand over "Do not give it the implementation" is not "give it nothing about the system". A draft has to name real operations and real shapes or it is fiction. Supply the **contract** and withhold the **semantics**: 1. The operation or entry point being exercised - its name, its parameters, and the shape of what comes back. 2. The error and status vocabulary the interface uses. 3. The route to the screen or endpoint, and the identity the case runs as. 4. The house conventions a case in this codebase has to follow. 5. The acceptance criteria, written as concrete examples wherever you can. The distinction that keeps this straight is the one above: shape may come from the code, values may not. Knowing that `discountedTotal` returns an amount and a status is shape. Knowing that the amount is ninety for this input is a value, and a value copied out of the body is not a test. ## Sharpen the criteria before drafting, not after Criteria written as three sentences of prose draft badly, and the reflex when the draft comes back thin is to paste the code in, which undoes everything above. The alternative is to turn the criteria into examples first: a small table of concrete inputs and the outcome each must produce, including at least one boundary and one rejection. That work has to happen either way - a person writing the case by hand does it in their head. Handing over vague criteria simply relocates the guessing into the draft, where it arrives with the confident formatting of a finished test. ## The one legitimate exception Characterisation is the case where mirroring the implementation is the point: code with no written specification, about to be restructured, where you want a net that catches any change at all. Draft those from the code deliberately, name them so nobody later mistakes them for statements of intent, and treat them as scaffolding with an expiry date. The defect is never that the implementation was used as context. It is that it was used *by default*, and the result was then read as though it proved a requirement.

  • If you withhold the implementation, how does the draft know what to call?
    You supply the contract without the behaviour: operation names, parameter and response shapes, the error vocabulary, the route, and the identity the case runs as. That is enough for a draft to compile and drive the system, while every expected value still comes from the criteria. Signature in, semantics out.
  • The acceptance criteria for a feature are three vague sentences. What do you do before drafting?
    Turn them into examples first - concrete inputs paired with the outcome each must produce, including a boundary and a rejection. That work is needed whether or not a machine drafts the case. Handing over vague criteria only moves the guessing into the draft, where it comes back looking finished.
  • Is there ever a case where showing the implementation is the right call?
    Yes, for characterisation: code with no specification that is about to be restructured, where a net that catches any change at all is exactly what you want. Draft those from the body deliberately, name them so nobody reads them as statements of intent, and retire them once the specification exists.

Asking the code what the code should do is like proofreading a translation against itself: everything agrees, and nothing has been checked.

saying these in an interview costs you the question

  • Pastes the function body in and treats criteria as optional
  • Believes a draft that passes immediately has proved something
  • Cannot say where a test's expected value should come from
  • Reads an assertion copied out of the code as a specification
  • Assumes more context always yields a better draft