When would you add a shingle token filter to an Elasticsearch analysis chain, and what does it cost?

## What shingles are A shingle is a word n-gram — the token-level analogue of a character n-gram. The `shingle` token filter takes the stream `the`, `quick`, `brown`, `fox` and, with a shingle size of 2, emits `the quick`, `quick brown`, `brown fox`. Because these are ordinary tokens, they land in the inverted index as ordinary terms and can be matched, counted, and scored like any other term. ## Why you would want them **Adjacency as a term.** A phrase query proves adjacency by reading positions from the postings, which costs more than a term lookup. Indexing bigrams turns "these two words were adjacent" into a single term match. On a field where phrase-like matching drives relevance, matching a shingled subfield is a cheap way to reward word order. **Relevance boosting.** The common pattern is a `multi_match` across the plain field and a shingled subfield with a higher boost on the shingled one. Documents containing the exact word pair score above documents that merely contain both words somewhere. This gives a phrase-ish ranking signal without turning the whole query into a phrase query. **Bigram statistics.** Features that need to know how often two words co-occur — suggesters, query analysis, ranking experiments — read those counts straight from the shingled field's term statistics. ## The parameters `min_shingle_size` and `max_shingle_size` both default to 2, giving bigrams only. Raising `max_shingle_size` to 3 adds trigrams, multiplying terms again. `index.max_shingle_diff`, an index-level setting with a default of 3, caps the difference between the two sizes, for the same reason the n-gram cap exists. `output_unigrams` defaults to `true`, meaning the original single tokens are emitted alongside the shingles. On a dedicated subfield that is usually wrong — you already have the unigrams in the main field, and keeping them doubles the subfield's work and muddies its scoring. Set it to `false` there. The related `output_unigrams_if_no_shingles` emits unigrams only when no shingle could be produced, which matters for single-word inputs that would otherwise analyze to nothing. `token_separator` (default a single space) joins the words in a shingle, and `filler_token` (default `_`) is inserted where a previous filter removed a token. That second one is the surprise: run `stop` before `shingle` and `the cat sat` yields shingles containing `_`, because the removed stopword still occupies a position. Whether that is desirable depends on what you are matching — it does preserve the fact that *something* was there, but it means the shingles no longer read as natural word pairs. ## The cost Term count is the whole story. A corpus has far more distinct word pairs than distinct words, and the distribution is much flatter, so a shingled field enlarges the term dictionary disproportionately rather than just adding postings to existing terms. Consequences: bigger index on disk, more memory for the terms index, slower merges, and longer shard recovery. Trigrams make all of that dramatically worse. There is also a relevance cost if you are careless. Scoring a field that mixes unigrams and shingles conflates two different signals, and boosting a shingled field too heavily makes exact word-order matches dominate results that users expected to be more forgiving. ## How to deploy it sensibly Put shingles on their own subfield with their own analyzer, keep `min_shingle_size` and `max_shingle_size` at 2 unless you have measured a reason to go further, set `output_unigrams: false`, and decide deliberately whether stopwords are removed before shingling. Then measure: index size before and after, and relevance before and after on a real query set. Shingles are one of the few relevance levers where the storage cost is large enough to be a capacity decision, not just a mapping choice. ## When not to use them If you simply need exact phrase matching, a phrase query on a normally-analyzed field already does that correctly and cheaply enough for most workloads. If you need substring matching inside words, that is character n-grams, a different filter entirely. Shingles earn their place when word adjacency needs to be a *scoring* signal blended with other signals, or when you need bigram counts as data.