What do stopwords cost in a search index, and why do modern engines usually keep them?

## What a stopword is A stopword is a term deemed to carry so little discriminating information that it is dropped from the token stream during analysis. Standard English lists run to a few dozen or a few hundred function words: articles, prepositions, conjunctions, auxiliary verbs, pronouns. Removal happens as a token filter, after tokenization, and — critically — the removal at index time is *permanent* for those documents. The word is not stored anywhere; it is simply gone. ## Why removal was standard Natural language word frequencies follow a Zipf-like law: a very small number of word types account for a very large fraction of all tokens. That means the postings lists for stopwords are enormous — often close to "every document" — while conveying almost nothing about which document you want. In early systems where disk was scarce and postings were traversed with little skipping machinery, deleting those lists produced a big win in both index size and query latency, with little apparent quality loss. ## What removal breaks The damage falls into three buckets. **Phrases and titles.** Entire meaningful strings consist of stopwords: *The Who*, *Take That*, *Let It Be*, *The The*, *To Be or Not to Be*. After removal these produce zero terms, so the query has nothing to match on and the user gets an empty result or an unrelated one. Short queries like *vitamin a* or *the who band* lose their most discriminating token. **Meaning-bearing function words.** *Not*, *without*, *no* invert meaning, and many stock lists include them. `flights not to Paris` and `flights to Paris` become the same query. **Phrase and proximity queries generally.** Phrase matching relies on token positions. A well-behaved removal filter leaves a position gap where the token was, so `king of denmark` indexed as `king` (pos 1) and `denmark` (pos 3) can still be matched by a query analyzed the same way. But this only works if both sides remove identically — and it also means a document containing `king in denmark` matches the phrase `king of denmark`, a precision loss that surprises people. **Irreversibility.** Because index-time removal deletes information, changing your mind requires a full reindex. There is no query-time knob that brings the words back. ## Why modern engines usually keep them Three things changed. First, **ranking got better at ignoring them by itself**. Inverse document frequency is defined so that a term appearing in nearly every document receives a weight near zero. A term present in 98% of documents contributes almost nothing to a BM25-style score whether or not you deleted it from the vocabulary. The historical fear that stopwords would swamp relevance was really a fear about naïve term-frequency ranking. Second, **storage and traversal got cheap**. Postings lists are delta-encoded and compressed, and skip structures let a conjunctive query jump over long stretches of a common term's list rather than reading it. A high-frequency term costs far less per query than it once did. Third, **dynamic top-k pruning replaced static deletion**. Techniques in the WAND and block-max family let a query computing the top *k* results skip whole blocks of a postings list whose maximum possible contribution cannot change the result set. A term with negligible weight is effectively skipped *at query time*, without ever having been removed at index time — which gives you the performance benefit while keeping the word available for phrase matching. This is the key modern argument, and it is the one interviewers are hoping to hear. ## When removal is still right - **Domain stopwords.** In a corpus of legal filings, words like *court*, *plaintiff* or *hereby* appear everywhere and carry as little signal as *the*. In a set of resumes, *experience* and *responsibilities* qualify. A tuned, corpus-specific list can be worth more than any generic English list, though IDF again handles much of it automatically. - **Very large corpora with hard latency budgets**, where index size or tail latency is the binding constraint and phrase quality on stopword-only queries is not a product requirement. - **Fields where phrases will never be searched** — tag soup, aggregation-only text. ## Middle grounds Rather than an all-or-nothing choice, teams commonly: keep stopwords in a phrase-capable field while removing them in a bag-of-words field used for recall; require a specified fraction of query terms to match so a stopword-heavy query is not over-constrained; or treat high-frequency terms as optional refinements that only influence ranking among documents already matched by the rare terms. Whatever you choose, remember the same list must apply on both sides, and any change on the index side means a reindex.