A product feature's generated answers are wrong in 4% of attempts and will not reach zero. How do you make the ship call?

## The comparison is never against zero The question usually arrives as "can we accept 4%", which quietly compares the feature with a perfect version that was never on offer. The real options are the feature at 4%, the process it replaces, and no feature at all. The process it replaces has an error rate of its own and it is almost never measured: people mistype, skip steps, work from stale copies and get tired. If the existing path is wrong in 7% of attempts and takes four minutes, then a feature wrong in 4% of attempts and taking ten seconds is an improvement, and the fact that its errors are new and attributable to a system is a matter of perception rather than of expected loss. Perception is nonetheless a real cost. Errors a system makes are noticed, aggregated and remembered in a way individual human errors are not, and people forgive their own mistakes more readily than a product's. The comparison is honest only when it also accounts for who gets blamed and what the errors spend from trust in the rest of the product. ## The two defensible positions **Ship it, contained.** The residual is measured, the expensive classes are gated so the remaining errors are cheap and detectable, users are told plainly what the feature is and is not, and a rate-based reversal condition exists with an owner and a fallback. The argument: expected value is positive against the real alternative, the downside is bounded by containment, and holding indefinitely for a number that will never reach zero is a decision never to ship at all. **Do not ship this shape.** Narrow it to the input range where it is reliably right and route the rest to the previous path; or demote it from acting to suggesting; or hold until the containment work exists. The argument: the residual is invisible to the person using it, the team has no way to discover afterwards that it was wrong, or the expensive class cannot be gated without removing the point of the feature. Both positions accept the same 4%. What separates them is containment and detectability, not the number. ## What picks between them 1. **Is the figure real?** Judged on inputs resembling what people will actually send, with the judging rule written down, over enough attempts that the range is narrow enough to decide on. A 4% measured over a hundred convenient inputs is not a 4%. 2. **What does the wrong 4% cost?** Split by consequence. If the irreversible slice is not zero after gating, that alone settles it. 3. **Can the user tell?** A feature whose errors are visible in the moment is bounded by the user's attention. One whose errors are plausible, confident and invisible transfers the whole risk to whoever reads the output later. 4. **Can we tell, afterwards?** If no recorded signal moves when the feature degrades, you are not shipping a 4% feature; you are shipping an unmeasured feature that was 4% once. 5. **What is the exposure plan?** Starting narrow and expanding on a reading of that signal turns one large bet into several small ones — but only if a breach genuinely holds the next expansion. 6. **Who carries the residual?** A cost borne by the person who chose to use the feature differs from one borne by a customer downstream who did not. ## The cost of being wrong in each direction | Direction | What it costs | |---|---| | Shipped and should not have | errors reach people; trust is spent on the whole product rather than this feature; withdrawal after adoption is far more expensive than never launching; sometimes an obligation to correct something outside the team | | Held and should have shipped | the worse existing process continues at its own unmeasured rate; the work stales and is quietly defunded; the team learns nothing, because real input only exists in production | The asymmetry between those two rows is what the decision actually turns on, and it is specific to the product. Where the downside is bounded, reversible and visible, the cost of holding usually dominates — the only route to a better rate runs through exposure. Where the downside is unbounded or silent, the cost of shipping dominates, because you will not find out in time to act. ## What gets written down - The measured rate, the number of judged attempts, and the spread across repeats. - The rule that judged right from wrong, and the input mix it was measured on. - The consequence classes and the tolerance accepted for each — particularly the irreversible one at zero, and how that zero is enforced. - The reversal condition: its signal, its levels, its window, and the person who reads it. - The date or event at which the decision is revisited, so an acceptance does not become permanent through silence. - **Who accepted the residual, by name.** That is the person who owns the product outcome, not the engineer who measured the rate. Stating a number is reporting; accepting it is deciding, and blurring the two is what turns an acceptance into a private judgement nobody can find again.