Analyzers and Tokenization

## Why this API exists Almost every "why doesn't my search match?" bug in Elasticsearch is an analysis bug: the term you are searching for is not the term that was indexed. `_analyze` is the microscope. It runs the analysis chain on text you supply and prints the resulting token stream, so you can compare what the index holds against what the query produces instead of guessing. ## The three ways to call it **1. By analyzer name, cluster-scoped.** ``` POST /_analyze { "analyzer": "standard", "text": "The Quick Brown Fox" } ``` This needs no index and is how you compare built-ins side by side. **2. By field, index-scoped.** ``` POST /my-index/_analyze { "field": "title", "text": "The Quick Brown Fox" } ``` Elasticsearch resolves the analyzer the way it would at index time — the field mapping's `analyzer`, otherwise the index default, otherwise `standard`. This is the form to reach for during debugging, because it tests the actual configuration rather than your recollection of it. The endpoint must be index-scoped; `field` on the cluster-level endpoint has no index to resolve against. A `normalizer` can be tested the same way on a keyword field. **3. By assembled parts.** You can pass `tokenizer` plus `filter` and `char_filter` arrays to prototype a chain before you commit it to index settings. ## Reading the output Each token comes back with: - `token` — the indexed term. - `start_offset` / `end_offset` — character offsets into the original text; these drive highlighting. - `type` — the tokenizer's classification, for example `<ALPHANUM>` or `<NUM>`. - `position` — the ordinal used for phrase and proximity matching. Gaps here explain why a phrase query fails after stopword removal, and `positionLength` shows tokens that span several positions. ## explain: true Adding `"explain": true` changes the shape of the response: instead of a flat token list you get the intermediate state after the character filters, after the tokenizer, and after each token filter, each labelled by name. This is how you localize the damage — for example seeing that the tokenizer produced `foxes` correctly and a stemmer collapsed it later. You can narrow the verbose output with the `attributes` parameter, which limits which token attributes are shown. ## What _analyze does not tell you It is a simulation of the analyzer, not a dump of the index. It cannot tell you what terms are actually stored — if the mapping changed after the documents were written, `_analyze` shows the *new* behaviour while the index still holds the *old* terms. To inspect what is really stored, use the `_termvectors` API for a specific document, or a `terms` aggregation on a keyword field. This distinction catches people out constantly: the analyzer looks right, the search still fails, and the answer is that the documents predate the mapping change and need reindexing. ## The debugging routine A productive sequence when a query misses: 1. Run `_analyze` with `field` on the *indexed* text. Note the tokens. 2. Run `_analyze` with `field` on the *query* text. Note those tokens. 3. Compare. If the two token sets do not intersect, the problem is analysis, not query type. 4. If they do intersect and there are still no hits, look elsewhere: the query might be a `term` query, which does not analyze its input at all and therefore must be given an indexed term verbatim; or you might be querying a multi-field such as `title.keyword` instead of `title`; or the document may not be searchable yet because the index has not refreshed. ## Practical notes `_analyze` accepts `text` as either a string or an array of strings — an array is analyzed as if the values were separate values of the same field, which is useful for checking position gaps between array entries. The API is cheap and read-only, so it is safe to run against production. And because it accepts arbitrary analyzer definitions inline, it is the right place to iterate on a chain before writing it into index settings, where changes are far more expensive to apply.