A run reported every case green after repairing forty locators - how should its output surface those repairs?

middleshould knowfreq 38%

answer

A clean pass and a repaired pass differ
Where does a reviewer actually look
Show the recorded anchor beside the substitute
Reporting without consequence decays into decoration

basics

~20 s

Give a repaired case its own outcome, distinct from a clean pass; put the substituted anchors on the run summary a reviewer actually reads; and set a volume above which the run stops being green at all.

solid answer

~50 s

Three changes, in increasing order of cost. Status: a repaired case is not a pass, so give the result model a third outcome - passed-with-substitution - and render it differently wherever case outcomes are listed. Placement: put the count and the substituted anchors in the run summary and on the check that gates a merge, not only in a file under the artefacts directory; a record nobody opens is not a report. Consequence: pick a number of substitutions, or a share of cases, above which the run is no longer green, so unattended drift eventually stops the line. Each entry should show the anchor the case recorded beside the element that was driven instead, because that pair is the product diff somebody has to accept or reject. Without the third change, the first two decay into decoration.

Three changes, in increasing order of cost. **Status**: a repaired case is not a pass, so give the result model a third outcome - passed-with-substitution - and render it differently wherever case outcomes are listed. **Placement**: put the count and the substituted anchors in the run summary and on the check that gates a merge, not only in a file under the artefacts directory; a record nobody opens is not a report. **Consequence**: pick a number of substitutions, or a share of cases, above which the run is no longer green, so unattended drift eventually stops the line. Each entry should show the anchor the case recorded beside the element that was driven instead, because that pair is the product diff somebody has to accept or reject. Without the third change, the first two decay into decoration.

code

json · 18 lines

json

{
  "run": "nightly-2451",
  "summary": {
    "passed": 812,
    "passedWithSubstitution": 40,
    "failed": 0,
    "verdict": "needs-review"
  },
  "substitutions": [
    {
      "case": "checkout/apply-discount",
      "step": "activate the apply control",
      "recordedAnchor": { "kind": "announcedName", "value": "Apply code" },
      "drivenInstead": { "kind": "position", "value": "third interactive element in the form" },
      "accepted": false
    }
  ]
}

go deeper

for a junior

Know that a case which needed a substitution should not look the same as one that passed cleanly, and be able to say where you would expect to see that difference.

for a middle

Explain the three parts: a distinct outcome for a substituted case, the recorded-versus-driven anchor pair placed where reviewers look, and a volume that changes the run's verdict.

for a senior

Show that you would attach a consequence rather than more detail, and describe how each substitution gets explicitly accepted or rejected by a named person before it becomes normal.

for a principal

Own what a green run is allowed to mean across the organisation, and be ready to justify blocking on substitution volume against the delivery cost of doing so.

Why a green run is the wrong place to stop

A locator-repairing suite that substituted forty elements and reported no failures has, in effect, made forty small edits to what the cases mean and then told nobody. The information exists - the mechanism knows exactly which anchor it could not match and which element it drove instead - but it lands somewhere with no reader: a verbose log, a file under the artefacts directory, a counter on a page nobody opens after a green run.

The default human behaviour around a green run is to not look, and that behaviour is correct almost all of the time. Any scheme that depends on somebody voluntarily opening a passing run's detail has already failed. So the fix is not more recording. It is to change what a passing run looks like when substitutions happened.

Three changes, in increasing cost

Status. A repaired case is not a clean pass, so stop calling it one. Give the result model a third outcome - passed with substitution - and render it distinctly everywhere case outcomes appear. This costs almost nothing and removes the possibility of mistaking one for the other at a glance.
Placement. Put the substitution count and the affected cases in the run summary and on whatever check gates a merge, not only in the detail view. A reader who never opens the run should still learn that forty anchors moved.
Consequence. Choose a point - a number of substitutions, or a share of cases - above which the run is not green at all. Without this, the first two decay into decoration within a few sprints, because a signal that costs nobody anything gets tuned out.

Where the repair is recorded	Who sees it	Effect on behaviour
A log line in the run's output stream	nobody, after a green run	none
A file among the run's artefacts	whoever is already debugging	none
A distinct case status in the results list	anyone scanning outcomes	noticed
A count on the check that gates a merge	the author of the change	acted on
A threshold that withholds the green verdict	the whole team	forced

What each entry has to contain

A count is not triageable. Forty substitutions tells a reader that something changed repeatedly and gives them nothing to accept or reject. The unit that can actually be triaged is a pair:

The recorded anchor - what the case says the control is: the name it should announce, the attribute it should carry, the text it should read.
The element that was driven instead - what the runner settled for, described the same way, so the two can be read side by side.

That pair is a product diff. Read together, they say: the case expects a control announced as "Apply code"; the run reached it by position instead. A reviewer can look at that and reach one of exactly two conclusions - an intended rename that should be written into the case, or a defect that should be filed. Both are progress. Neither is available from a number.

Two further fields make the entry work in practice:

A reviewed flag, so an accepted substitution stops re-alarming on every run while a new one still does. Without it the report either screams constantly or is silenced wholesale.
The case and step, so the entry has an owner. An unattributed substitution belongs to everybody, which means nobody.

The failure mode this is guarding against

Teams that add reporting without consequence end up worse off than teams that added nothing, because they now believe they have visibility. The report exists, it is accurate, it is complete, and it has never once changed a decision. Six months later the suite is substituting hundreds of anchors a night, the interface has drifted well away from what the cases describe, and every run is green.

The test for whether the reporting is real is simple and uncomfortable: name the last substitution somebody looked at, and say what happened next. If the answer is that nobody has opened it, the reporting is decoration regardless of how detailed it is. Attach the consequence - a status that is not green, a check that blocks, an explicit acceptance before the next run treats a substitution as normal - and the report starts being read, because now something depends on it.

## Why a green run is the wrong place to stop A locator-repairing suite that substituted forty elements and reported no failures has, in effect, made forty small edits to what the cases mean and then told nobody. The information exists - the mechanism knows exactly which anchor it could not match and which element it drove instead - but it lands somewhere with no reader: a verbose log, a file under the artefacts directory, a counter on a page nobody opens after a green run. The default human behaviour around a green run is to not look, and that behaviour is correct almost all of the time. Any scheme that depends on somebody voluntarily opening a passing run's detail has already failed. So the fix is not more recording. It is to change what a passing run *looks like* when substitutions happened. ## Three changes, in increasing cost 1. **Status.** A repaired case is not a clean pass, so stop calling it one. Give the result model a third outcome - *passed with substitution* - and render it distinctly everywhere case outcomes appear. This costs almost nothing and removes the possibility of mistaking one for the other at a glance. 2. **Placement.** Put the substitution count and the affected cases in the run summary and on whatever check gates a merge, not only in the detail view. A reader who never opens the run should still learn that forty anchors moved. 3. **Consequence.** Choose a point - a number of substitutions, or a share of cases - above which the run is not green at all. Without this, the first two decay into decoration within a few sprints, because a signal that costs nobody anything gets tuned out. | Where the repair is recorded | Who sees it | Effect on behaviour | | --- | --- | --- | | A log line in the run's output stream | nobody, after a green run | none | | A file among the run's artefacts | whoever is already debugging | none | | A distinct case status in the results list | anyone scanning outcomes | noticed | | A count on the check that gates a merge | the author of the change | acted on | | A threshold that withholds the green verdict | the whole team | forced | ## What each entry has to contain A count is not triageable. *Forty substitutions* tells a reader that something changed repeatedly and gives them nothing to accept or reject. The unit that can actually be triaged is a **pair**: - **The recorded anchor** - what the case says the control is: the name it should announce, the attribute it should carry, the text it should read. - **The element that was driven instead** - what the runner settled for, described the same way, so the two can be read side by side. That pair is a product diff. Read together, they say: *the case expects a control announced as "Apply code"; the run reached it by position instead.* A reviewer can look at that and reach one of exactly two conclusions - an intended rename that should be written into the case, or a defect that should be filed. Both are progress. Neither is available from a number. Two further fields make the entry work in practice: - **A reviewed flag**, so an accepted substitution stops re-alarming on every run while a new one still does. Without it the report either screams constantly or is silenced wholesale. - **The case and step**, so the entry has an owner. An unattributed substitution belongs to everybody, which means nobody. ## The failure mode this is guarding against Teams that add reporting without consequence end up worse off than teams that added nothing, because they now believe they have visibility. The report exists, it is accurate, it is complete, and it has never once changed a decision. Six months later the suite is substituting hundreds of anchors a night, the interface has drifted well away from what the cases describe, and every run is green. The test for whether the reporting is real is simple and uncomfortable: **name the last substitution somebody looked at, and say what happened next.** If the answer is that nobody has opened it, the reporting is decoration regardless of how detailed it is. Attach the consequence - a status that is not green, a check that blocks, an explicit acceptance before the next run treats a substitution as normal - and the report starts being read, because now something depends on it.

Why show the recorded anchor beside the substitute rather than just a repair count?
Because the pair is the product diff. A count says something changed forty times and offers nothing to act on. The pair says the case expects a control announced one way and the run reached it by position instead, which a reviewer can resolve in exactly two ways: accept an intended rename into the case, or file a defect. Only the pair can be triaged.
Your run summary already lists substitutions and nobody reads it. What changes?
Attach a consequence. Put the count on the check that gates a merge, render a substituted case as something other than green wherever outcomes are listed, and require an explicit acceptance of each new entry before the next run treats it as normal. Reporting that costs nobody anything is tuned out by default; a report that blocks something gets read.

saying these in an interview costs you the question

Records substitutions only in a log nobody opens
Reports a repair count with no anchors and no owner
Renders a repaired case identically to a clean pass
Adds reporting but attaches no consequence to it
Assumes somebody will read the artefacts after a green run

A run reported every case green after repairing forty locators - how should its output surface those repairs?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Why a green run is the wrong place to stop

Three changes, in increasing cost

What each entry has to contain

The failure mode this is guarding against

▾common follow-ups2 questions

▾red flags5 flags

Why a green run is the wrong place to stop

Three changes, in increasing cost

What each entry has to contain

The failure mode this is guarding against

answer

go deeper