Search & Information Retrieval Concepts

Filter when a non-matching document must never be shown: permissions, region, availability. Boost when the signal is only a preference — recent or in-stock items should rank higher, but a strong match elsewhere may still outrank them.

Precision@k is the fraction of the top k results that are relevant. Recall@k is the fraction of all relevant documents that appear in the top k. Raising k can only raise recall, and usually lowers precision.

Text analysis runs in three stages: character filtering (strip markup, map characters), tokenization (cut the stream into tokens), then token filtering (lowercase, fold accents, drop stopwords, stem, add synonyms). The surviving tokens become the index terms.

Reciprocal rank fusion gives each document 1/(k + rank) from every list it appears in, summed. The constant k controls how sharply top ranks dominate: small k makes rank one overwhelming, large k flattens the curve so agreement across lists matters more.

A posting carries the document ID plus, optionally, the term frequency in that document for scoring, the token positions for phrase and proximity matching, character offsets for highlighting, and sometimes per-position payloads.

k1 controls term-frequency saturation: how quickly extra occurrences of a term stop adding score. b controls document-length normalisation: how strongly a document longer than the collection average is penalised, from none at zero to full at one.

Term frequency measures how often a term occurs in one document, a proxy for how much that document is about the term. Inverse document frequency measures how rare the term is across the whole collection, a proxy for how discriminating it is.

Because a boost multiplies that field's score contribution, and field contributions are not on a comparable scale. A ten-times title boost lets a weak, incidental title match outscore a document that matches the query strongly everywhere else.

NDCG sums each result's graded relevance discounted by a logarithm of its position, then divides that discounted cumulative gain by the gain of the best possible ordering. The division normalises every query onto a 0-to-1 scale so scores can be averaged.

Both sides must reduce text to the same term forms. The index stores analyzed terms, and a query can only match a term spelled identically after its own analysis, so mismatched pipelines silently return zero hits instead of an error.

Map each leg onto a comparable range before weighting: min-max or z-score over a fixed candidate window, or a corpus-calibrated bound. Per-query min-max is the usual choice and the usual bug, because it forces a 1.0 onto every query's best hit however bad it is.

Check first whether the click drop is real harm or good abandonment, then look for the usual gaps: an unrepresentative judgment query sample, judges guessing intent differently from users, position and presentation bias in clicks, and non-relevance regressions such as latency.

Boolean retrieval treats a query as a logical predicate and returns an unordered set of documents that satisfy it. Ranked retrieval scores every candidate for degree of relevance and returns them ordered, so partial matches still surface.

Nearest-neighbour search ranks by distance and has no match predicate, so it returns the k closest vectors however far away they are. Keyword retrieval intersects postings lists first, so query terms that appear nowhere yield an empty result set.

Sorted order lets the engine enumerate ranges — prefixes, wildcards, fuzzy neighbourhoods — compress shared prefixes between adjacent terms, and hold only a sparse in-memory index over on-disk blocks. A hash table gives exact lookup and nothing else.

A raw dot product grows with vector magnitude, so long documents win simply for containing more terms. Cosine divides by both vector lengths, comparing the direction of the vectors — the mix of terms — rather than their size.

Multiply the relevance score by a decay factor between zero and one that falls with age, instead of adding a freshness bonus. A bounded multiplier preserves relevance ordering among documents of similar age and caps how much recency can distort ranking.

Mean reciprocal rank averages 1 divided by the position of the first relevant result, so it only sees one document per query. Mean average precision averages precision measured at every relevant position, rewarding a ranking that surfaces all relevant documents high.

Stemming strips affixes with language rules, fast and dictionary-free, producing stems that may not be real words. Lemmatization uses a dictionary and word context to return the true base form. Search engines usually choose stemming for speed and recall.

Stopword removal shrinks postings for very common words and speeds queries, but it destroys phrases and titles built from them. Modern engines usually keep stopwords because ranking already gives near-zero weight to ubiquitous terms and compression makes them cheap.

A keyword engine treats a filter as one more postings list to intersect, so it gets cheaper and stays exact as selectivity rises. An approximate vector index must discard matches after searching, or traverse a structure built over every vector, so recall falls instead.

Postings are sorted ascending, so storing gaps instead of absolute IDs turns large numbers into small ones. Variable-byte encoding then spends one byte on small gaps and more only when needed, using seven payload bits per byte plus a continuation flag.

An AND query drives iteration from the rarest term and asks the other lists to advance to each candidate document. Skip pointers store absolute document IDs and byte offsets at intervals, so advancing jumps over whole blocks instead of decoding every posting.

BM25 bounds each term's contribution with a saturation curve and folds a tunable length correction into that curve, so a long document cannot win by repetition or bulk. Raw TF-IDF grows linearly with term frequency and normalises length only crudely.

Saturate the raw counts, cap popularity's share of the total score, and correct for position bias — clicks measure where a document ranked as much as how good it was. Add exploration and signal ageing so new documents can be discovered.

Labelled training data, a feature pipeline that produces identical values at training and serving time, and a latency budget for re-scoring the top candidates. Hand-tuned boosts need none of that, but they also cannot learn interactions between signals.

Interleaving blends both rankings into one result list per query, so every user compares both systems at once. That paired within-user design removes between-user variance, which is what makes an A/B test need large samples.

Pooling judges only the documents that the contributing systems retrieved, and everything unjudged is scored as non-relevant. A new system that surfaces relevant documents no contributor ever returned is therefore penalised for finding them.

Edge n-grams index every prefix of every token so a prefix search becomes a plain term lookup. The cost is a multiplied index and term dictionary, distorted term statistics, and a full reindex whenever the gram length range changes.