What confidence threshold lets a self-healing locator repair silently, and who owns that number?

A self-healing mechanism produces a **similarity score** for its best substitute. The **confidence threshold** is the line that turns that score into a decision: above it the run continues on the substitute, below it the step fails. Choosing that line is not a tuning detail. It decides how much unreviewed change a suite is willing to absorb and still report green. ## What the number is, and what it is not The score is a weighted sum of per-feature similarities. It is **not** a probability that the substitution is correct, and it is not comparable across mechanisms, across applications, or even across two screens of the same application whose elements carry different amounts of identifying information. A screen where every control has a distinct identifier produces confident, well-separated scores; a screen of near-identical rows produces a cluster of high scores that mean very little. The same 0.82 therefore means different things in those two places, which is the first reason a number copied from somebody else is indefensible. ## The two errors you are trading Every threshold sits between two failure modes, and moving it trades one for the other. | Setting | What happens | What it costs | |---|---|---| | **Too low** | A wrong element is accepted and the case proceeds against it | The run reports green while the case no longer tests what it claims; the failure surfaces later, from a user | | **Too high** | A substitution a person would call obvious is rejected | The suite fails on changes that broke nothing, and the team learns to re-run rather than read | The two are not symmetric. A false repair is silent and durable; a false alarm is loud and self-correcting. That asymmetry is the argument for erring high — but *err high* is still not a number. ## Calibrating rather than guessing The only defensible way to place the line is to measure your own application. 1. Run for a period — a few weeks, or a few hundred proposals — with the mechanism recording every repair it *would* make: its score, the runner-up's score, the features that agreed, and a captured image of the screen. 2. Have someone who knows the product label each proposal *same control* or *different control*. This is the expensive step and nothing substitutes for it. 3. Plot the two labels against score. You get two overlapping distributions, and the overlap is the part of the problem that no threshold solves. 4. Choose the cut from the shape of that overlap and the cost asymmetry above, not from a round number. 5. Write down the sample size, the date and the application version you calibrated on. A threshold with no provenance cannot be argued about later. Recalibrate when the screens change materially. A redesign, or a change in how identifiers are assigned, shifts the whole distribution, and last year's line is describing a product that no longer exists. ## Bands beat a single line A single cut forces every uncertain match into one of two outcomes, and the uncertain ones are exactly the interesting ones. Most mature setups use three bands, each with its own `min_score`: - **High** — substitute and log. The run passes normally and the repair sits in the record. - **Middle** — substitute, but mark the run repaired-and-unreviewed so it reads differently from a clean pass, and put the repair in front of a person before the next release. - **Low** — fail the step. The mechanism has not identified the element and should say so plainly. Add a `required_margin` over the runner-up as a second condition on the high band: a 0.93 that beats a 0.91 is not a confident identification, however high it looks. ## Whose number it is The threshold does not belong to whoever wrote the failing case, and it is not the mechanism's to decide. It belongs to the group that owns the suite and acts on its results, for the same reason a coverage gate or a release checklist does: it encodes a risk decision on behalf of everyone who reads a green run. In practice that means it lives in version control beside the suite rather than in someone's local settings, it carries a note naming the calibration behind it, changes to it are reviewed like code, and a named person is asked when the healing rate moves. The failure mode worth naming out loud is **threshold drift under pressure**. The suite goes red the day before a release, someone lowers the number until it goes green, and nobody records why. That single edit converts a safety net into a silencer, and because it is one line in one file it is invisible in every report the team looks at afterwards. A threshold that can be changed without evidence is not really a threshold; it is a slider that always ends up at whichever setting makes the pipeline quiet.