Search & Information Retrieval Concepts

Index-time expansion makes queries simple and fast but requires reindexing to change and distorts term statistics. Query-time expansion is editable immediately and keeps statistics honest, at the cost of larger queries and hard multi-word cases. Most teams expand at query time.

Size the addressable query segment first, not the technology. Hybrid pays when a meaningful share of real traffic is descriptive or vocabulary-mismatched; it does not when traffic is navigational, the corpus is small, or the latency and re-embedding budget is tight.

Positional data usually dominates, because it scales with token occurrences rather than documents; the term dictionary grows with distinct terms. Cut positions on fields that never take phrase queries, drop unqueried fields entirely, and reconsider n-gram fields before anything else.

Version the ranking configuration like code, change one signal at a time, gate it offline on a fixed query set, then expose a small traffic slice behind a runtime kill switch. Review results per segment: an average win often hides a tail regression.

Learned sparse models expand a query into tens or hundreds of weighted terms, many of them common with long postings lists, and their flatter weights defeat the dynamic-pruning tricks that make keyword scoring fast. More lists to traverse, far less skipping.

The classic probabilistic IDF is a log-odds ratio that falls below zero once a term appears in more than half the collection, meaning a matching document is penalised for containing a very common word. Implementations add one inside the logarithm or floor the value.