Why do two Python str values that render identically sometimes compare unequal, and how does unicodedata.normalize fix it?

## The model: str is a sequence of code points A Python `str` is a sequence of Unicode **code points**, and `==` on two `str` objects is a length-plus-element comparison over that sequence. Nothing in that comparison knows about fonts, glyphs, or how many marks end up stacked over one letter on screen. That is the whole source of the surprise: *rendering* is a many-to-one function of code points, so two different sequences can produce the same picture. Unicode makes this unavoidable rather than accidental. To stay round-trippable with the older national character sets it replaced, Unicode encodes many accented letters twice: once as a **precomposed** character (U+00E9 LATIN SMALL LETTER E WITH ACUTE) and once as a **base plus combining mark** (U+0065 LATIN SMALL LETTER E, then U+0301 COMBINING ACUTE ACCENT). The standard calls these two sequences **canonically equivalent**: they mean the same character and conforming software must treat them as the same text. Python's `==` does not, because `==` is defined on code points, so you get `len(a) == 4` and `len(b) == 5` for the same visible word. ## The four normalization forms `unicodedata.normalize(form, s)` returns a new `str` rewritten into a chosen normal form. There are four, and they are two independent choices crossed: * **Composed vs decomposed.** `'NFC'` pushes combining marks back into precomposed characters where one exists; `'NFD'` splits every precomposed character into base plus marks. Both preserve canonical equivalence — the text still means the same thing. * **Canonical vs compatibility.** `'NFC'`/`'NFD'` only apply canonical mappings. `'NFKC'`/`'NFKD'` additionally apply **compatibility** mappings, which are *lossy* in a formatting sense: a fullwidth letter becomes an ASCII letter, a ligature becomes its component letters, a superscript becomes a plain digit. That extra folding is powerful and dangerous, and it is a separate subject from simply making equal-looking text compare equal. For "these two strings should be the same string", `'NFC'` is the default answer. It is the form the web platform and most interchange formats assume, it is the shortest of the four for typical text, and it changes nothing for pure ASCII, so applying it costs you nothing on the common path. ## Where to put the call Normalization is a **boundary** operation, not a comparison-time operation. If you normalize inside your equality helper, then every dict key, every database row, every index entry and every log line still holds whichever spelling arrived, and the moment some other code path compares them directly — a `set` membership test, a `dict` lookup, a uniqueness constraint in storage — the mismatch is back. Normalize once where untrusted or foreign text enters (request parsing, file reading, decoding), keep the normalized form as the only form you store, and let ordinary `==` work from then on. `unicodedata.is_normalized(form, s)` answers "is this already in that form?" without allocating the converted copy, which matters when the answer is nearly always yes. It is also useful as a *policy* check rather than a fix: some systems prefer to reject a non-canonical input at the edge instead of silently rewriting it, so that the sender learns about the problem. ## Related pieces of the module `unicodedata` also exposes the character database itself: `unicodedata.name(ch)` gives the official character name, `unicodedata.category(ch)` gives the general category (`'Lu'`, `'Nd'`, `'Mn'` for a non-spacing combining mark, and so on), and `unicodedata.combining(ch)` returns the canonical combining class, which is nonzero exactly for the marks that reordering rules apply to. Those are the tools for *inspecting* why two strings differ, and printing `[unicodedata.name(c) for c in s]` for both sides is usually the fastest way to end an argument about a mystery mismatch. `unicodedata.unidata_version` reports which Unicode data release the interpreter was built against, which is why a normalization result can in principle differ between old and new interpreters for very recently added characters — stable, long-standing characters do not move. ## What this is not Normalization is not sanitization, not case handling and not a lookalike filter. Two characters can be visually confusable and still be entirely distinct after any normal form — a Cyrillic small letter that looks exactly like a Latin `a` is a different letter, canonically and compatibly, and NFC will happily leave both alone. Normalization only collapses the spellings Unicode itself declares equivalent. Treating it as "make this text safe" is the mistake that produces the next bug.