Text Analysis & Tokenization

## Why analysis exists An inverted index maps a **term** to the list of documents containing it. Lookup in that structure is an exact match on a byte string: the engine hashes or binary-searches the term dictionary for the term you hand it. Nothing about that structure is fuzzy. So every bit of tolerance users expect from search — case-insensitivity, matching *running* against *run*, ignoring the accent in *café* — has to be manufactured **before** anything is stored, by normalising both the document text and the query text into the same reduced vocabulary. That normalisation process is text analysis, and it runs in three ordered stages. ## Stage 1: character filtering The first stage sees the raw character stream and rewrites it before any token boundaries exist. Typical jobs: stripping HTML or XML markup so `<b>shoe</b>` does not index `b` as a word; character mapping so `&` becomes `and`, or so curly quotes become straight ones; Unicode normalisation, which matters because the same visible character can have several encodings (an *é* can be one code point or *e* plus a combining accent, and those are different byte strings — different terms). Anything that must happen before tokenization belongs here, because once the text is cut into tokens you can no longer merge or split across a boundary. ## Stage 2: tokenization The tokenizer cuts the character stream into tokens. For English and most European languages this is roughly word-boundary splitting: whitespace and punctuation separate tokens and are themselves discarded. Real tokenizers are more careful than `split(" ")` — they have rules for hyphens, apostrophes, URLs, email addresses and numbers, because those choices are visible to users. Whether `wi-fi` becomes one token or two decides whether a search for `wifi` can ever match. Tokenization is also where languages differ most sharply. Chinese, Japanese and Thai are written without spaces, so a tokenizer must either segment using a dictionary and statistical model, or fall back to overlapping character bigrams. German forms long compounds (*Fußballmannschaft*), so a decompounder may be needed for a search on *Mannschaft* to hit it. There is no language-neutral tokenizer, which is why engines ship per-language ones. The tokenizer also attaches metadata to each token that later stages and later queries depend on: a **position** (used by phrase and proximity matching) and character **offsets** into the original text (used by highlighting). Losing or shifting these is a real bug class. ## Stage 3: token filtering Token filters form a chain, each taking a token stream and emitting one. Order matters, because each filter sees only what the previous one produced. The common filters: - **Lowercasing** — so `Shoe` and `shoe` collapse to one term. - **Accent/diacritic folding** — `café` to `cafe`, so users typing without accents still match. - **Stopword removal** — dropping very common function words such as *the*, *of*, *and*. - **Stemming** — reducing inflected forms to a common stem, so *running*, *runs* and *ran* can meet. - **Synonym expansion** — injecting extra tokens at the same position, so *tv* also indexes or queries as *television*. - **N-gram generation** — emitting substrings or prefixes of a token to support partial matching. - **Length limits and trimming** — discarding absurdly long tokens produced by junk data. Ordering bugs are common: applying a synonym list *after* stemming means your synonym entries must be written in stemmed form, or they never fire. ## What actually lands in the index A useful mental exercise is to run a sentence through the pipeline by hand. `The Quick-Brown Foxes!` through a standard chain — tokenize, lowercase, remove English stopwords, stem — indexes roughly `quick`, `brown`, `fox`. The literal string is gone; only those three terms are searchable in that field. This is why a query that looks obviously correct can return nothing: it is being compared to `fox`, not to `Foxes`. ## Full-text fields versus exact-match fields Not every field wants this treatment. Identifiers, status codes, tags and anything used for exact filtering, sorting or grouping should be indexed **unanalyzed** — as a single term equal to the whole input value — because you want `IN_PROGRESS` to be one term, not `in` plus `progress`. Most engines therefore distinguish an analyzed full-text field type from a keyword-style exact field type, and let you index one source value both ways under different field names, choosing at query time which behaviour you want. Getting this choice wrong is the second-most common cause of "my filter matches nothing" after analyzer asymmetry. ## The takeaway Analysis is a lossy, deliberate transformation. Everything the index knows about a document's text is what the pipeline emitted, so when debugging matching problems the first move is always to ask what terms this text actually produced.