Under 1% of impressions are manipulated — do you ship the hardened ranker, and what would change that call?

## Why this is a principal question Everything technical about the trade-off can be settled below this level: the hardened checkpoint costs clean accuracy on all traffic, the benefit is conditional on manipulation that stays inside a stated edit set, and neither number means much without its columns. What remains is a judgement somebody has to own and could be asked to defend: **you are proposing to make the product measurably worse for everyone in order to make it harder to abuse for a few.** That is a decision with owners outside the model team. ## Step one: one unit, honestly constructed Expected harm per impression, for each candidate: - **Cost side:** the clean loss, multiplied by 1.0. Unconditional, permanent, on every impression, and it recurs unchanged after every retrain. - **Benefit side:** the harm avoided per prevented manipulation, multiplied by the share of impressions that are manipulated **and** inside the trained-for edit set. The '<1% manipulated' figure is the wrong multiplier and it is the most common error in this room. It is an upper bound: some of that 1% is manipulation the hardening does not cover — different edits, a larger budget, or abuse that never touches the ranking features at all. The qualifying share is smaller, sometimes by a lot, and if nobody has measured it you should say the decision is not yet decidable rather than guess. ## Step two: the two harms are not the same currency A straight accuracy-point subtraction assumes one undeserved top slot and one ordinary mis-ranked page cost the same. They rarely do. An undeserved slot can carry trust, integrity and regulatory weight far beyond its impression count; equally, on some surfaces it is worth very little and the honest answer is that the hardening is not worth its bill. Getting this ratio is not the model team's call — it belongs to whoever owns marketplace integrity — and refusing to invent it is part of doing this job well. ## Step three: who absorbs the loss The clean loss is almost never uniform. It concentrates where forcing a constant answer over a region bites hardest: thin categories, rare queries, new or small sellers. So the real question is not 'is 1.8 points acceptable' but 'is 6 points acceptable for this segment, in exchange for a defence that mostly protects the segments where the money is.' That is a distributional decision and it needs a named owner, not an average. ## Step four: the claim you can honestly make Be explicit about what shipping does and does not let anyone say. - **Can say:** manipulation inside a specific, written-down edit set is measurably harder against this checkpoint than the previous one. - **Cannot say:** the ranker is robust; manipulation is solved; a seller who works outside that edit set gains nothing. None of those follow, and a leader who repeats them in public has been mis-briefed by you. ## Step five: make the decision reversible and dated The cost is standing and the threat is not, so a single go/no-go is the wrong artefact. Whichever way it goes, record: - **the trigger that flips it** — a measured rise in in-budget manipulation above a stated share, or a change in what a manipulated slot is worth; - **the instrument that would detect it** — you cannot use a threshold you have no way to observe, so if the qualifying-manipulation share is not currently measurable, funding that measurement may be the actual decision; - **a re-measure date**, because a defence trained against last quarter's edit set drifts as sellers move, while its accuracy bill does not drift at all. ## The shape of a strong answer 'On what is in front of me, no — the bill is unconditional and the multiplier on the benefit is unmeasured and smaller than the 1% quoted. I would not spend a permanent product regression on an unmeasured conditional. What I want funded is the measurement of in-budget manipulation and a per-slice view of who pays the 1.8; with those two, this is a five-minute decision either way, and I will commit to a trigger and a review date rather than to a permanent answer.' A candidate who says either 'ship it, robustness is good' or 'skip it, 1% is nothing' without constructing the unit has not answered the question.