How do you choose an embedding similarity threshold for auto-closing incidents?

## Why the number cannot be borrowed Similarity scores have no absolute meaning. Every embedding space has its own baseline: the level unrelated pairs reach because the vectors crowd into a narrow region and share a large common direction. In one model that baseline may sit near 0.35, in another near 0.80. A cutoff of 0.85 is aggressive in the first space and barely above noise in the second. The corpus matters too — a homogeneous corpus of incident tickets, all written in the same register with the same product nouns, compresses scores further than a diverse corpus would. So the first thing to say is that a threshold is a property of the triple (model, corpus, task). Copying one across any of those three is guessing. This is the single most common mistake in production semantic systems, and it is why an interviewer asks the question at all. ## Turning it into a measurement The procedure is unglamorous and works. Take historical incidents where the duplicate relationship is already known — linked tickets, merged records, post-incident reviews — and build two sets of pairs: true duplicates and true distinct pairs, the latter sampled to look like what the system will actually see rather than uniformly at random, because uniformly-random negatives are far too easy. Score every pair with the exact production pipeline, including the query and document roles the model expects. You now have two distributions. Read the operating characteristics off them: for each candidate cutoff, what fraction of things above it are real duplicates (precision) and what fraction of real duplicates are above it (recall). If the two distributions overlap so heavily that no cutoff gives acceptable precision, the answer is not a cleverer threshold — it is a better representation or a second-stage check. ## Letting the costs choose the point This is where judgment enters and where there is no single right answer. The default habit is to maximize F1, which implicitly declares a false positive and a false negative equally bad. For auto-closing incidents they are not remotely equal. Wrongly closing a live incident suppresses a real signal, delays response, and destroys trust in the automation; wrongly leaving a duplicate open costs an engineer a few minutes of triage. The asymmetry might be a hundred to one, which puts the operating point far out on the high-precision end, accepting low recall deliberately. The honest framing is to state the cost ratio explicitly with the people who own the consequence, then pick the point that minimizes expected cost under that ratio. If nobody will state the ratio, that itself is the finding — the automation is not ready to be autonomous. ## Designs that beat a bare threshold **Margin over the runner-up.** Require that the best match exceeds the second-best by some gap. This is far more robust than an absolute score, because it is a within-query comparison: it survives a shift in the whole score distribution, which an absolute cutoff does not. **An abstain band.** Above a high line, act automatically; below a low line, do nothing; in between, surface a suggestion to a human. This converts a binary threshold problem into a coverage problem, and coverage can be tuned over time as confidence grows. **Calibration.** Fit a monotonic mapping from raw score to estimated probability of a true duplicate on held-out labelled pairs, using an approach such as isotonic regression or logistic (Platt) scaling. Then the decision rule is stated in probability terms, which is the only form in which a cost ratio can be applied honestly. The mapping is as model-and-corpus specific as the threshold it replaces. **A cheap second check.** A high-precision confirmation step — a stricter comparison, a structural check on the affected service and time window, or a model judgment on the pair — lets you run a lower similarity gate without the precision loss. ## Operating it over time Pin the threshold, or the calibration mapping, to the model version and the pipeline configuration, and store them together. A model upgrade invalidates the number completely; so does a change to how the two sides are encoded, and so, more gradually, does a corpus that shifts as your product changes. Monitor the inputs, not just the outputs: track the score distribution of the daily match candidates, and alert on drift in its mean or spread. Track the auto-close rate and, most importantly, sample outcomes — audit a fraction of automated closes for correctness, because that is your only direct measurement of production precision. Keep a reversal path so that a wrong auto-close is recoverable, which is what makes a slightly-too-aggressive threshold survivable rather than catastrophic. ## The short answer to give No universal number; derive it on your own labelled pairs with your own model; let an explicitly stated cost asymmetry choose the operating point; prefer margins and abstain bands to a bare cutoff; version the threshold with the model and audit the outcomes.