How do you decide whether semantic search should replace an existing keyword search?

## Frame it as a migration, not an upgrade The question contains a trap: "replace". A keyword search that has been in production for years has absorbed a great deal of accumulated tuning — synonym lists, boosts, curated results for head queries, spelling correction, business rules. That tuning is invisible and undocumented and it is why the incumbent performs better than anyone expects on the queries that matter most commercially. Replacing it wholesale discards all of it at once. The disciplined framing is: what does the new capability add, what does it cost, and what does it break. ## Establish the baseline before anything else Most teams proposing semantic search cannot state how the current system performs. So step one is instrumentation, not modelling: - Sample real queries with real frequencies from logs. Traffic is heavily head-weighted, and evaluating on a hand-written query list systematically over-represents the tail where semantic search looks best. - Build a judged set with known-good answers, either from human labels or from strong implicit signals such as the result users ultimately converged on. - Record the incumbent's offline quality on that set, plus its production behaviour: zero-result rate, reformulation rate, latency percentiles, and success on the commercially important query classes. Without this, any comparison later is a story rather than a measurement. ## Segment or you will ship a regression This is the central discipline. Query populations have wildly different characteristics and semantic search affects them in opposite directions: - **Navigational and head queries** — a brand, a category, a known product. The incumbent is heavily tuned here and semantic retrieval frequently makes them worse by returning plausible alternatives. - **Identifier queries** — SKUs, reference numbers, error codes. Reliably weaker under dense retrieval. - **Long natural-language queries** — descriptions of intent that contain none of the corpus's vocabulary. This is where semantic search wins decisively, often turning a zero-result page into a good one. - **Misspellings and morphological variants** — mixed, and depend on whether the incumbent already has spelling correction. - **The long tail** — high aggregate volume, individually rare, historically poorly served, and usually the strongest business case for the change. Read the experiment per segment and require each one to be evaluated on its own. A flat overall number with a large loss in the highest-intent segment is a worse outcome than a modest overall gain. ## Offline first, online decides Offline evaluation on a judged set is cheap and fast and is the right place to iterate. It is not the decision, because it cannot capture presentation, latency, user behaviour or the interaction with the rest of the page. Move to an online comparison for the decision: either a conventional A/B, or interleaving, where results from both systems are mixed into one list and user preference between them is measured directly — interleaving needs far less traffic to reach a conclusion, which matters when a search team has weeks rather than quarters. Set guardrail metrics before you start and agree what would stop the launch: a regression threshold on identifier and navigational segments, latency percentiles, zero-result rate, and reformulation or abandonment rate. Deciding those afterwards is how teams talk themselves into shipping a regression. ## Count the operating cost honestly The recurring costs are the part a principal is expected to raise unprompted: - **An ingestion path that must not drift.** Every document has to be embedded, re-embedded on change, and reconciled if the pipeline fails midway. Silent divergence between the source of truth and the index is the most common production defect in vector search. - **Rebuilds on model change.** Upgrading the embedding model invalidates the entire index. That means capacity for a full re-embed, a dual-index or shadow strategy, and a cutover plan — for a large corpus this is a project, not a deploy. - **A second failure domain.** A new store, new capacity planning, new latency characteristics, and a new dependency in the request path. - **Serving cost.** Embedding every query, plus vector search, plus any reranking tier. Against that, quantify the upside in the same currency: the tail and natural-language queries currently returning nothing, converted at your normal rate. ## The usual honest conclusion As of 2026 the mature answer is almost always **augment rather than replace**: keep the keyword index and add semantic retrieval as a second leg, so every existing tuned win is preserved and the new capability covers what keywords structurally cannot. That also de-risks the rollout, because the incumbent remains available as a fallback and as a comparison in production. Full replacement is defensible mainly for a genuinely new surface with no incumbent tuning to lose, or a corpus and query mix where lexical matching was never the right model in the first place. Stage the rollout regardless: internal traffic, then a small percentage, then per-segment expansion, with a fast rollback path and per-segment dashboards left in place afterwards — because model and corpus drift will move these numbers long after launch.