skip to content

How do you check that an agent's change did what you asked, and not just something reasonable?

level: middleimportance: must knowfreq 58%

answer

  1. Reasonable is not the same as asked
  2. List outcomes before opening the change
  3. Walk the change in both directions
  4. Extra behaviour traces back to nothing
  5. Ambiguity was resolved silently, as code

basics

~20 s

Write the request out as observable outcomes before reading the change, then walk it both ways: every outcome present, and every behaviour change traceable to an outcome. The backwards walk is the half reviewers skip.

solid answer

~50 s

Grade the change against the request, not against whether the code is good. Write the request down as a short list of observable outcomes before you open the diff — read the diff first and it supplies the expectations you were meant to bring to it. Then walk it twice: forwards, so every outcome appears in the change, and backwards, so every behaviour in the change traces to an outcome. The backwards walk is the one people skip, and the one that finds unrequested behaviour — which survives review precisely because nothing about it looks like a defect. Where a run had to choose, as in a stock-count change that silently corrects the quantity it was only asked to flag, the request had a hole in it, and that hole is a finding in its own right.

code

pseudocode · 12 lines
pseudocode
# the request, in full:
#   "when a counted quantity differs from the system, record the
#    discrepancy and flag the bin for recheck"

function submitCount(bin, sku, countedQty):
    systemQty = stock.levelFor(bin, sku)
    if countedQty == systemQty:
        return OK

    discrepancies.record(bin, sku, systemQty, countedQty)
    stock.setLevel(bin, sku, countedQty)
    return OK

go deeper

for a junior

Know that a change is graded against the request, not against whether the code reads nicely. Read what was asked before you read what was written, and keep the two side by side while you read.

for a middle

Explain the backwards walk: every behaviour in the change has to trace to something that was asked for. Be able to say why unrequested behaviour survives a review that a missing requirement would not.

for a senior

Show that you treat divergence as evidence about the request as well as about the change. Where the run had to choose, the request had a hole, and that hole is still open for the next run.

for a principal

Own the call on wanted-but-unasked behaviour: it becomes its own change with its own review rather than shipping inside a review aimed at something else, and the request absorbs what the review discovered.

## Two ways a generated change can be wrong A change can fail to do what you asked. It can also do something you never asked for. Reviewers are trained, by years of reading colleagues' work, to hunt the first: a missing requirement announces itself because you go looking for it. **The second is close to invisible, because nothing in it looks like a defect.** Extra behaviour is usually sensible, usually well written, and usually consistent with the code around it. It fails one test only: nobody chose it. Call it **intent divergence** — the change is locally reasonable and is not globally the change that was requested. On an unattended run it has a specific cause. Wherever the request was ambiguous, the run resolved the ambiguity by choosing, because it had no channel to ask. **The choice arrives as code and never as a question.** A person in the same position can send you a message; an unattended run has no such channel, and a run told to finish a feature finishes it. ## The stock-count case The request: *when a counted quantity differs from the system, record the discrepancy and flag the bin for recheck.* The change that comes back, across nine files, records every discrepancy faithfully — and also sets the system quantity to the counted one, so the books agree again immediately. No bin is ever flagged. Read as code, this is decent work: short, the equal case handled early, the names sensible. Read against the request it is two defects at once, an invented behaviour and a dropped requirement, and the invented one is the worse of the two, because **the correction destroys the very signal the feature existed to produce.** A discrepancy that is silently reconciled is a discrepancy nobody will ever investigate. | the request asked for | the change delivers | status | |---|---|---| | record the discrepancy | records it | asked for | | flag the bin for recheck | nothing | missing | | — | sets the system quantity to the count | invented | ## Write the outcomes down first, then walk both ways 1. **Before opening the change, write the request out as a short list of observable outcomes.** A handful of lines, each something you could point at in the running system. 2. **Walk forwards:** every outcome on the list appears in the change. This is the direction everybody already does. 3. **Walk backwards:** every behaviour change in the diff traces to an outcome on the list. This direction finds divergence, and this is the one that gets skipped. 4. **Collect what traces to nothing** and sort it: incidental consequence, or behaviour nobody chose. Step 1 is ordered first for a reason. Read the change first and it hands you a coherent story, after which the request reads like confirmation of that story rather than a specification against it — a plausible addition then registers as a requirement you had forgotten. For the same reason, walk the change with the request open rather than from memory; memory of a request is quietly reshaped by the change you have just read. ## What to do when you find it The reflex is to ask whether the extra behaviour is *good*. That is the wrong axis, because nearly all of it is defensible in isolation. The question is whether it was *wanted*, and by whom. Three honest outcomes: - **Not wanted** — it comes out, and the request gains a sentence saying so. - **Wanted, but separate** — it comes out of this change and goes in as its own, with its own review. Bundled here, it ships on the strength of a review aimed at something else. - **Wanted and inseparable** — rare, and worth saying out loud, because it means the request was wrong about what the feature is. The most valuable finding is usually not in the code at all: **divergence is evidence about your request.** If the run had to choose, the request had a hole in it, and the hole is still there for the next run and for the next person who reads the feature. ## What this is not Not every line that fails to trace back is scope creep. A change that renames a field must update its callers; a change that adds a branch may need a helper to hold it. Those are consequences of what you asked for, they can each be named, and a review that files them as divergence will bury the real finding in noise. Unrequested behaviour is also not the same thing as unrequested structure — an interface with one implementation, a setting nobody sets. Both arrive uninvited, but one changes what the system does while the other changes what the code costs to read, and they are judged on different grounds.

  • The extra behaviour is genuinely better than what you asked for. Do you keep it?
    Not in this change. Whether it is better is a separate decision from whether it has been reviewed, and bundling it means it ships on the strength of a review aimed at something else. Take it out and raise it as its own change; the second review is cheap once the behaviour is isolated.
  • How do you tell incidental edits from scope creep in a large change?
    Ask what forced the edit. A rename that updates its callers, wiring for a new file, a block moved to keep one function readable — each is a consequence of something you asked for, and you can name the cause. Scope creep has no such cause: it changes what the system does and traces back to nothing in the request.
  • Where should the finding go once the divergence has been removed?
    Into the request. The run chose because the request did not say, so the missing sentence is the durable fix. Removing the code alone leaves the ambiguity in place for the next attempt, and for whoever reads the feature next.

saying these in an interview costs you the question

  • If the extra behaviour is an improvement, scope creep does not matter
  • Read the diff first; the request is obvious from it
  • Only missing requirements count as divergence, not additions
  • An unattended run would have stopped to ask if the request were ambiguous
  • Unrequested behaviour is harmless because the existing tests still pass