When is unicodedata.normalize with NFKC the right choice, and what does it destroy?

## Canonical versus compatibility equivalence Unicode defines two kinds of "the same". **Canonical equivalence** says two sequences encode the same character: a precomposed é and `e` plus a combining acute are the same letter, encoded differently. **Compatibility equivalence** is weaker and says two characters mean the same thing while differing in form: the ligature fi and the two letters `fi`, the superscript ² and the digit `2`, full-width Ａ and ASCII `A`, a no-break space and an ordinary space. `unicodedata.normalize` exposes both. NFD and NFC apply canonical decomposition (and, for NFC, recomposition) and are **lossless** — you can round-trip between them forever. NFKD and NFKC apply *compatibility* decomposition first, and are **lossy**: after NFKC there is nothing left in the string that says the author wrote a superscript, a ligature or a no-break space. ```python import unicodedata print(unicodedata.normalize('NFKC', 'file')) # file print(unicodedata.normalize('NFKC', 'x²')) # x2 print(unicodedata.normalize('NFKC', 'Ａ１')) # A1 print(unicodedata.normalize('NFKC', '\u00a0') == ' ') # True ``` ## What NFKC is for The K forms exist to build **keys**, not text. Three uses justify them: **Search and matching.** A user typing `file` should find a document typeset with the fi ligature, and a user typing an ASCII product code should find a record entered with full-width digits. Folding both sides through NFKC makes those collide by design. **Deduplication.** Two records that differ only by a no-break space or by full-width punctuation are the same record for the purpose of merging, and the folded key says so. **Identifier normalization.** Python itself does this: since Python 3.0 the parser normalizes identifiers to NFKC, so a name written with a ligature and the same name written with plain letters bind the identical object. That is a deliberate application of exactly this rule — the identifier is a key, not display text. ```python fi = 1 print(fi) # 1 -- the parser folded the identifier to NFKC ``` ## What it destroys Because compatibility decomposition is one-way, applying it to text you keep is data loss: - Mathematical and scientific notation flattens: exponents, subscripts and many symbol variants become ordinary characters, and a formula rendered from the folded text is wrong. - Deliberate spacing disappears: a no-break space inserted to keep a number with its unit becomes an ordinary space that can be broken across a line. - Typography flattens: ligatures and specialised forms become plain letters, which changes text a publisher chose deliberately. - Some scripts are affected far more than Latin, so a change that looks harmless on English test data can be destructive on real multilingual input. The rule that follows: **store the canonical form, derive the compatibility form.** Keep NFC text as the value of record, compute the NFKC key alongside it for indexing and lookup, and never write the key back over the value. ## Combining with case folding A case-insensitive search key needs both operations, and the order matters because case folding can produce output that is no longer normalized. Fold between two normalization passes: ```python import unicodedata def search_key(s: str) -> str: return unicodedata.normalize('NFKC', unicodedata.normalize('NFKC', s).casefold()) ``` ## The limits worth naming NFKC is not a sanitizer. It does not remove zero-width characters, it does not merge characters from different scripts that merely look alike, and it does not strip control characters. Treating it as a cleanup pass for untrusted input gives false confidence: whatever policy you need about confusable or invisible characters has to be written explicitly, typically by inspecting `unicodedata.category` and rejecting or stripping the categories you disallow. It is also not free, and the folded result depends on the interpreter's Unicode database — 16.0.0 in CPython 3.14, reported by `unicodedata.unidata_version`. Keys persisted under an older database are not guaranteed identical to keys computed today for rare characters, which is a real consideration for a long-lived index and an argument for being able to rebuild it. The one-sentence interview answer: NFC for what you keep, NFKC for what you look things up by, and never the reverse.