Unicode Normalization

### The problem You want a *folding key*: a version of a Python `str` you can compare, index or put in a URL slug, in which "Renee" finds "Renée" and "Fabrege" finds "Fabrège". The two approaches people reach for first both fail, and they fail for the same underlying reason. The first is a hand-written table: `s.replace("é", "e").replace("è", "e")...`, or the same thing via `str.maketrans` and `str.translate`. It works only for the accented letters you thought of, and Latin script alone has hundreds of them across a dozen diacritics. The second is `s.encode("ascii", "ignore").decode()`. On `"Renée Fabrège"` that returns `b'Rene Fabrge'` — the accented letters are not *stripped*, they are **deleted**, because the codec sees one un-encodable code point and the `ignore` error handler drops it whole. The same call silently deletes every CJK character, every emoji and every Cyrillic letter in the string too, so a non-Latin title collapses to an empty key. ### Two representations of the same letter Unicode can spell "é" in two ways. **Precomposed** (NFC) is one code point, U+00E9 LATIN SMALL LETTER E WITH ACUTE. **Decomposed** (NFD) is two: `e` followed by U+0301 COMBINING ACUTE ACCENT. `unicodedata.normalize` converts between them. Text arriving from a form, a database or an HTTP request is usually precomposed, and that is exactly why filtering does nothing: there is no separate accent character to remove — the accent is *inside* the single code point. ### The recipe ```python import unicodedata def fold(s: str) -> str: d = unicodedata.normalize("NFD", s) return unicodedata.normalize("NFC", "".join( ch for ch in d if not unicodedata.combining(ch) )) ``` Three steps, each load-bearing: 1. **`normalize("NFD", s)`** applies canonical decomposition, splitting every precomposed letter into a base character plus its combining marks. `len("café")` is 4; `len(normalize("NFD", "café"))` is 5. 2. **Filter the marks.** `unicodedata.combining(ch)` returns the canonical combining class as an integer — non-zero exactly for characters that stack onto a preceding base character, zero for everything else. Testing `unicodedata.category(ch) == "Mn"` (nonspacing mark) is the usual equivalent; `combining` is the more precise test for "this reorders/stacks during normalization", and `Mn` is the more precise test for "this is a nonspacing mark". For accent stripping either is fine, and they agree on Latin diacritics. 3. **`normalize("NFC", ...)`** recomposes what is left. After removing the marks there is usually nothing to recompose, but it matters when the string mixes scripts: NFC is the form you want to store and compare, and returning a half-decomposed string means the next `==` against a precomposed value fails for the very reason you started here. ### What it does not do `NFD` only undoes *canonical* decompositions, and plenty of letters have none: `unicodedata.decomposition("ø")` is the empty string, and so is `đ`, `ł`, `ß`, `þ` and `æ`. NFD leaves them as they are, so the folded key still contains non-ASCII. That is a feature — it stops you silently mangling text — but it means "strip accents" is not the same as "make ASCII". If you truly need ASCII you must add an explicit mapping (`str.maketrans` over the letters you care about), and you are then making a **language-dependent** choice: German expects `ö → oe`, Swedish treats `ö` as a distinct letter that sorts after `z`, and Turkish distinguishes dotted and dotless `i`. A general transliteration library exists for this, but it is a transliteration policy, not a Unicode operation. `NFKD` decomposes more — compatibility mappings turn `fi` into `fi`, `²` into `2`, and full-width forms into ASCII — which is often desirable in a search key and disastrous in stored text, since it is not round-trippable. It still does nothing for `ø`. ### Using it correctly The folded value is a **key beside the original, never a replacement for it**. Store and display the user's real text; compute the key for lookup, uniqueness or a URL slug, and index that column. For a case-insensitive key, combine with `str.casefold()` (not `str.lower()`), which handles `ß → ss` and Greek final sigma. A defensible order is: `NFD` → drop marks → `casefold()` → `NFC`, because casefolding some characters can itself change the normalization form, so normalizing last leaves the key in a stable shape. Cost is a full pass over the string plus a database lookup per character in the worst case, so fold once when the record is written rather than on every comparison in a query loop. Note also that a folded key is **lossy and many-to-one**: "Renee", "Renée" and "Renèe" collapse together, which is what you want for search and what you must not do for authentication or authorization identifiers, where collapsing distinct inputs onto one value is how impersonation happens.