How do you tune the dense-versus-lexical weight in hybrid retrieval on a mixed jargon corpus?

## What the weight actually is In a weighted hybrid retriever, the fused score is a convex combination of the two legs — conventionally written `alpha * dense + (1 - alpha) * lexical`, so alpha=1 is pure vector search and alpha=0 is pure lexical. The first practical warning: **implementations disagree about which end is which**, so read the documentation of the system you are in rather than assuming. A weight can also be applied in rank space, as a per-list multiplier inside reciprocal rank fusion, which avoids putting the two raw scores on one axis at all. ## The evaluation set decides the answer A sweep is only as good as the queries you sweep against. Requirements: - **Distributional fidelity.** If 20% of production traffic is exact-identifier lookups, roughly 20% of the eval set should be. A set hand-written by engineers skews heavily toward prose questions and will systematically recommend too much dense weight. - **Real relevance labels.** Graded judgements over a pooled candidate set, ideally pooled from *both* legs so you are not scoring against labels only one retriever could have found. - **Held out.** Tune on one split, confirm on another. A weight chosen and reported on the same queries is a fitted parameter reported as a result. - **Enough of it.** A 50-query set gives confidence intervals wider than the differences you are trying to resolve. ## Running the sweep Step alpha across [0, 1] — steps of 0.1 are plenty for the first pass — and record recall@k for the retrieval stage plus an order-sensitive metric such as nDCG@k. Always include both endpoints; they are the baselines the whole exercise is justified against. Then, critically, break the metric out by query slice: natural-language questions, exact identifiers, mixed jargon phrases, misspellings. ## Reading a broad flat optimum The characteristic result on a mixed jargon-and-prose corpus is a curve that rises off both endpoints and then sits flat across a wide middle band, with all values inside that band statistically indistinguishable. That shape carries three messages: 1. **Both legs contribute.** If either endpoint matched the plateau, the corresponding leg would be redundant. 2. **The exact value is not critical.** Anywhere in the plateau is fine, so choose the middle for margin against drift rather than chasing the peak. 3. **Aggregate tuning has hit its ceiling.** The flatness exists because moving weight helps one slice while hurting another. The per-slice curves are *not* flat — the identifier slice typically improves monotonically toward the lexical end and the paraphrase slice toward the dense end — and the aggregate is their average. Further gains come from something other than a single global constant. The most common failure here is treating the argmax as a discovery. With a small eval set the peak wanders between reruns and even between re-labellings; deploying it is overfitting to sampling noise, and it makes the system fragile because the deployed value sits on an edge of the plateau rather than in its interior. ## When the curve is not flat A genuinely sharp peak usually means something structural: one leg is misconfigured (wrong analyzer, wrong similarity metric, unnormalized vectors), the eval set is dominated by a single query shape, or the score scales are so mismatched that alpha is really acting as a scale correction rather than a preference. Investigate rather than accept it. Likewise, if the optimum is at an endpoint, the honest conclusion is that hybrid is not paying for itself on this corpus and you should drop a leg and its operational cost. ## Beyond one global constant When the per-slice curves clearly diverge, the next move is to stop asking for one number. Options, in ascending order of complexity: classify the query shape and apply a per-shape weight; move the weighting into rank space so the parameter is a source preference rather than a score correction; or drop weighting altogether and use unweighted rank fusion, which has no alpha to tune and is far more robust to a model swap. Each of these buys robustness at the cost of another moving part, and on many systems the honest answer is that a mid-plateau constant is good enough. ## Keeping it alive A tuned weight is not permanent. It is coupled to the embedding model, the chunking strategy, the analyzer configuration and the corpus mix. Any of those changing invalidates the sweep, so wire the sweep into the evaluation pipeline as a job you can re-run rather than a number someone once put in a config file with no note about how it was chosen.