Which steps must a self-repairing suite refuse to heal and fail outright on, whatever the substitute's confidence?

seniorshould knowfreq 42%

answer

Some substitutions change what the case means
Identity, irreversibility, contract
Consequence decides it, not likelihood
A discovered list is one escape behind

basics

~20 s

Steps where the control's identity is part of what you are testing: the exact action a case exists to prove is reachable, irreversible or money-spending operations, and any check of a control's announced name, enabled state or permission gating.

solid answer

~50 s

Draw the list from the product's own contracts, not from incidents. A step belongs on it when substituting a different element changes what the case means, or does something the run cannot take back. Three families cover most of it. Identity steps: the case exists to prove a specific control is present and reachable, so any substitute makes the assertion vacuous. Irreversible actions: paying, deleting, sending, confirming once - driving a similar-looking element here has consequences outside the suite. Contract checks: an announced name, an enabled state, a permission-gated control, where the lookup failing is the observation. Everything on the list fails on a missed anchor however well a candidate scores. Write the list as the case is written, review it when the contract changes, and never let it grow only by post-mortem - a discovered list is always one escape behind.

Draw the list from the product's own contracts, not from incidents. A step belongs on it when substituting a different element changes what the case *means*, or does something the run cannot take back. Three families cover most of it. **Identity steps**: the case exists to prove a specific control is present and reachable, so any substitute makes the assertion vacuous. **Irreversible actions**: paying, deleting, sending, confirming once - driving a similar-looking element here has consequences outside the suite. **Contract checks**: an announced name, an enabled state, a permission-gated control, where the lookup failing *is* the observation. Everything on the list fails on a missed anchor however well a candidate scores. Write the list as the case is written, review it when the contract changes, and never let it grow only by post-mortem - a discovered list is always one escape behind.

code

pseudocode · 9 lines

pseudocode

step "open the account menu":
    anchor      = announced_name("Account")
    repairable  = true          # navigation, reversible, not the assertion

step "confirm the payment":
    anchor      = announced_name("Pay now")
    repairable  = false         # identity + irreversible
    reason      = "spends money; the control is what we are asserting"
    on_miss     = fail("no control announces Pay now")

go deeper

for a junior

Know that some steps must never be substituted: ones that spend money, delete data, or exist to prove a particular control is there. Be able to give one example of each.

for a middle

Explain why substitution changes meaning on those steps - the assertion becomes about whatever was found rather than about the control named in the case - and say where the decision is recorded.

for a senior

Show that you would derive the list from the product's contracts as the case is written, defend a default of repairable with argued exceptions, and reject a list assembled from incident reports.

for a principal

Own the boundary: decide which suites may adapt at all, who reviews exceptions, and what evidence would move a whole area off adaptive lookup rather than adding one more exception.

Why some substitutions change what a case means

A locator-repairing suite silently swaps one element for another when the recorded anchor stops matching. On most steps that swap is harmless: the case was going to click through a navigation menu on the way to the thing it actually verifies, and any control that gets it there serves the purpose equally well.

On some steps the swap destroys the case. The distinguishing question is not how likely is the substitution to be wrong - that is a scoring question, and no threshold answers it. The question is what happens if the substitution is wrong, and on a subset of steps the answer is that the case now asserts something other than what it was written to assert, or that the run has done something to the system it cannot take back.

Those steps need a hard rule, not a confidence bar: they fail on a missed anchor, and no candidate is ever driven in place of the recorded one.

The three families

Identity steps. The case exists to prove that a specific control is present, reachable and in the expected state. "The refund action is available to a support agent." "The delete option is absent for a read-only member." Substituting any other element makes the assertion vacuous - it now says that something was found, which was never in doubt.
Irreversible actions. Paying, deleting, sending, publishing, confirming a one-time operation. Here a wrong substitution is not merely a wrong verdict; the run has spent money, mailed a customer or destroyed a record. The blast radius is outside the suite.
Contract checks. Steps whose subject is a property of the control rather than the flow through it: the name it announces, whether it is enabled, whether a permission gate hides it. The lookup failing is the observation, so absorbing the failure removes the only evidence.

Step	Substitution is	Correct policy
Open a menu on the way to the page under test	harmless, keeps the suite alive	repairable
Dismiss an incidental banner before the flow starts	harmless	repairable
Confirm a one-time payment	financially consequential	never repairable
Assert a permission-gated action is absent	makes the assertion vacuous	never repairable
Check the control announces the expected name	erases the observation	never repairable

Deciding the list rather than discovering it

The failure mode here is procedural, not technical. Teams assemble this list from post-mortems: a defect escapes, somebody traces it to a substitution, and that step is added. The list grows one entry per incident, which means every entry has already been paid for by the defect that created it, and the list is permanently one escape behind. Repair counts do not help either - frequency measures interface churn, not consequence, and the step that hides the expensive defect may be substituted exactly once.

The alternative is to decide it at authoring time:

The author sets it per step, while writing the case. That is the only moment when the reason the step exists is still explicit in somebody's head. A step that is the assertion, or that cannot be undone, is marked non-repairable there and then.
Review it like an assertion. Whoever reviews the case checks the flag the same way they check what the case proves. An irreversible action carrying a repairable flag is a review comment, not a runtime problem.
Re-derive it when the contract changes. A step becomes an identity step the moment somebody adds a permission rule or a legal confirmation to it, whether or not the case text changed.
Record the reason, not just the flag. Irreversible and this is what we are asserting are different justifications with different review triggers later.

Keeping the list small enough to mean something

Default to repairable and require a stated reason for each exception. The inverse - mark everything non-repairable to be safe - is not caution, it is switching off adaptive lookup while continuing to pay for it, and a team that has actually decided that should say so plainly and remove the mechanism.

The healthy shape is a suite where the great majority of steps are ordinary navigation, arrangement and cleanup that may be substituted freely, and a small, argued set of steps that never are. If the exception list keeps growing, that is information: either the interface is churning so fast that the cases no longer describe it, or the suite has been written so that its assertions are spread across many steps instead of concentrated in a few. Both are worth acting on, and neither is fixed by another exception.

## Why some substitutions change what a case means A locator-repairing suite silently swaps one element for another when the recorded anchor stops matching. On most steps that swap is harmless: the case was going to click through a navigation menu on the way to the thing it actually verifies, and any control that gets it there serves the purpose equally well. On some steps the swap destroys the case. The distinguishing question is not *how likely is the substitution to be wrong* - that is a scoring question, and no threshold answers it. The question is **what happens if the substitution is wrong**, and on a subset of steps the answer is that the case now asserts something other than what it was written to assert, or that the run has done something to the system it cannot take back. Those steps need a hard rule, not a confidence bar: they fail on a missed anchor, and no candidate is ever driven in place of the recorded one. ## The three families 1. **Identity steps.** The case exists to prove that a *specific* control is present, reachable and in the expected state. "The refund action is available to a support agent." "The delete option is absent for a read-only member." Substituting any other element makes the assertion vacuous - it now says that *something* was found, which was never in doubt. 2. **Irreversible actions.** Paying, deleting, sending, publishing, confirming a one-time operation. Here a wrong substitution is not merely a wrong verdict; the run has spent money, mailed a customer or destroyed a record. The blast radius is outside the suite. 3. **Contract checks.** Steps whose subject is a property of the control rather than the flow through it: the name it announces, whether it is enabled, whether a permission gate hides it. The lookup failing *is* the observation, so absorbing the failure removes the only evidence. | Step | Substitution is | Correct policy | | --- | --- | --- | | Open a menu on the way to the page under test | harmless, keeps the suite alive | repairable | | Dismiss an incidental banner before the flow starts | harmless | repairable | | Confirm a one-time payment | financially consequential | never repairable | | Assert a permission-gated action is absent | makes the assertion vacuous | never repairable | | Check the control announces the expected name | erases the observation | never repairable | ## Deciding the list rather than discovering it The failure mode here is procedural, not technical. Teams assemble this list from post-mortems: a defect escapes, somebody traces it to a substitution, and that step is added. The list grows one entry per incident, which means **every entry has already been paid for by the defect that created it**, and the list is permanently one escape behind. Repair counts do not help either - frequency measures interface churn, not consequence, and the step that hides the expensive defect may be substituted exactly once. The alternative is to decide it at authoring time: 1. **The author sets it per step, while writing the case.** That is the only moment when the reason the step exists is still explicit in somebody's head. A step that is the assertion, or that cannot be undone, is marked non-repairable there and then. 2. **Review it like an assertion.** Whoever reviews the case checks the flag the same way they check what the case proves. An irreversible action carrying a repairable flag is a review comment, not a runtime problem. 3. **Re-derive it when the contract changes.** A step becomes an identity step the moment somebody adds a permission rule or a legal confirmation to it, whether or not the case text changed. 4. **Record the reason, not just the flag.** *Irreversible* and *this is what we are asserting* are different justifications with different review triggers later. ## Keeping the list small enough to mean something Default to repairable and require a stated reason for each exception. The inverse - mark everything non-repairable to be safe - is not caution, it is switching off adaptive lookup while continuing to pay for it, and a team that has actually decided that should say so plainly and remove the mechanism. The healthy shape is a suite where the great majority of steps are ordinary navigation, arrangement and cleanup that may be substituted freely, and a small, argued set of steps that never are. If the exception list keeps growing, that is information: either the interface is churning so fast that the cases no longer describe it, or the suite has been written so that its assertions are spread across many steps instead of concentrated in a few. Both are worth acting on, and neither is fixed by another exception.

Who decides whether a step may be substituted, and when?
The author, at the moment the case is written, because that is the only time the reason the step exists is still explicit. Deciding later means deciding from failure data, which contains only the steps that already escaped. A reviewer then checks the flag the way they check the assertion: an irreversible action marked repairable is a review comment, not a runtime surprise.
How do you keep the never-repair list from swallowing the whole suite?
Default to repairable and require a stated reason for each exception, rather than the reverse. Most steps are navigation, arrangement and cleanup where an equivalent control serves equally well. Reserve the list for steps whose meaning depends on the exact element. If it keeps growing, that is a signal in itself - either the interface is churning badly, or you have chosen a plain suite and should say so.

saying these in an interview costs you the question

Builds the exclusion list only after a defect escapes
Says a high enough acceptance threshold makes exclusions unnecessary
Marks every step non-repairable and calls that safe
Lets an irreversible action be driven by a substituted control
Treats it as a global runner setting rather than a per-step decision

Which steps must a self-repairing suite refuse to heal and fail outright on, whatever the substitute's confidence?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Why some substitutions change what a case means

The three families

Deciding the list rather than discovering it

Keeping the list small enough to mean something

▾common follow-ups2 questions

▾red flags5 flags

Why some substitutions change what a case means

The three families

Deciding the list rather than discovering it

Keeping the list small enough to mean something

answer

go deeper