In a C# chat app, two nicknames both display as "José" yet == says they differ; why, and how does string.Normalize fix it?
answer
- one letter, two encodings
- a combining accent mark
- ordinal compares code units
- the parameterless overload's form
basics
~20 sUnicode stores é either as one precomposed code point (U+00E9) or as e plus U+0301 COMBINING ACUTE ACCENT; == compares UTF-16 code units ordinally, so they differ. Normalizing both with string.Normalize(), which uses FormC, makes them equal.
solid answer
~40 sThe two strings are **canonically equivalent** but not identical: one holds U+00E9, the other `e` followed by U+0301, so their lengths are 4 and 5. C# `==` and the default comparer of `HashSet<string>` and `Dictionary<string, TValue>` are ordinal, so both forms can be registered as separate nicknames. `string.Normalize()` (FormC by default) rewrites either into the precomposed form, after which `==` is true; `IsNormalized()` checks without rewriting. Normalize identifiers once at the boundary and normalize the probe before every lookup. Avoid FormKC on stored text — it folds fi into `fi` and ² into `2` — and remember normalization neither folds case nor unifies lookalike letters from other scripts.
code
csharp · 14 linesusing System.Text;
string typed = "José"; // precomposed é, U+00E9
string pasted = "José"; // e followed by U+0301 COMBINING ACUTE ACCENT
Console.WriteLine($"{typed.Length} {pasted.Length}");
Console.WriteLine(typed == pasted);
Console.WriteLine(typed.Normalize() == pasted.Normalize());
Console.WriteLine(pasted.IsNormalized());
Console.WriteLine(typed.Normalize(NormalizationForm.FormD).Length);
var nicknames = new HashSet<string> { typed };
Console.WriteLine(nicknames.Contains(pasted));
Console.WriteLine(nicknames.Contains(pasted.Normalize()));go deeper
Recall that one accented letter can be stored two ways, that == compares the stored code units, and that string.Normalize exists to make the two forms match.
Explain precomposed versus decomposed forms, name the four NormalizationForm values with FormC as the default, and show the HashSet<string> lookup missing until both sides are normalized.
Normalize at the boundary and before every lookup, validate lengths after normalizing, keep FormKC for derived keys, and confirm the service is not running in globalization-invariant mode, where Normalize does nothing.
Set the rule for which form the system stores identifiers in, and make every intake path follow it, so uniqueness checks hold across clients and data imports.
## Canonical equivalence Unicode often has more than one way to encode the same letter. "é" exists as a single **precomposed** code point, U+00E9 LATIN SMALL LETTER E WITH ACUTE, and as a **decomposed** sequence: `e` (U+0065) followed by U+0301 COMBINING ACUTE ACCENT. The standard declares these **canonically equivalent**: they are meant to render and behave identically, so a reader cannot tell them apart. Which form a string arrives in depends on where it came from. A keyboard on one platform may produce the precomposed form, text pasted from a document or a file name from another system may arrive decomposed, and some input methods build accented letters from combining marks. A chat app that accepts nicknames from all of these will see both. Nothing on screen gives the difference away: the two forms of "José" render the same glyphs, but one is 4 UTF-16 code units and the other 5, and a code-point dump shows `U+00E9` on one side and `U+0065 U+0301` on the other. The bug report therefore usually arrives as "two users with the same name" or "search cannot find a name I can see", not as an encoding problem. ## What `==` and the default comparers do In C#, `==` on two strings and `string.Equals(string)` perform an **ordinal** comparison: they compare UTF-16 code units one by one. "José" is 4 code units and "José" is 5, so `==` returns `false`. The default comparer used by `Dictionary<string, TValue>` and `HashSet<string>` is ordinal as well, so a set holding one form does not contain the other. Culture-aware comparisons do treat canonically equivalent strings as equal — the lesson on comparison and culture covers when those are appropriate — but identifiers and dictionary keys are usually compared ordinally, and that is where duplicates slip through. ## The four `NormalizationForm` values `string.Normalize(NormalizationForm)` rewrites a string into one canonical shape, and `string.IsNormalized(NormalizationForm)` reports whether it already is. The parameterless overloads use **FormC**. | Form | What it does | "José" → | Typical use | |---|---|---|---| | `FormC` | canonical decomposition, then recomposition | 4 chars | the default for stored and compared text | | `FormD` | canonical decomposition only | 5 chars | stripping accents, some search indexing | | `FormKC` | compatibility decomposition, then recomposition | 4 chars | derived matching keys | | `FormKD` | compatibility decomposition only | 5 chars | derived search keys | The **K** forms also fold *compatibility* characters, which only look related: the ligature fi (U+FB01) becomes `fi`, and a superscript ² becomes `2`, so `"x²"` turns into `"x2"`. That is useful for matching but changes meaning, so it should never overwrite the text a user typed. ## Where to normalize 1. **At the boundary.** Normalize nicknames, tags and other identifiers to FormC when they enter the system — registration, rename, import — so every stored copy has one shape. 2. **Before every lookup.** Normalize the probe string too; a stored FormC key is useless if the search term arrives decomposed. 3. **Before length checks.** `Normalize` can change `Length` in either direction, so validate limits on the normalized value. 4. **Keep the original when display matters.** If you need FormKC for matching, store it as a separate key next to the text as typed. ## What normalization does not do - **It does not fold case.** "JOSÉ" and "José" stay different; case-insensitive matching is a comparison choice, not a normalization form. - **It does not unify lookalikes from other scripts.** Cyrillic а (U+0430) and Latin a (U+0061) are different letters, and every form leaves them apart; catching impersonation needs a separate confusables check. - **It does not repair invalid UTF-16.** `string.Normalize()` and `IsNormalized()` throw `ArgumentException` on a string containing a lone surrogate, so a string damaged by an earlier bad cut fails here. - **It does nothing in globalization-invariant mode.** With `InvariantGlobalization` set to `true`, a common setting for small container images, `Normalize` returns its input unchanged and `IsNormalized` returns `true` for any string, so the two "José" forms stay different. - **It is not free.** `Normalize` can hand back the same instance when nothing changes (an all-ASCII string returns at once), but other text is still scanned, so normalize once at the boundary rather than on every comparison. ## The diagnosis in one line When two strings print the same but `==` says `false`, dump their code points — `string.Join(" ", s.EnumerateRunes().Select(r => $"U+{r.Value:X4}"))` — and you will usually find a precomposed letter on one side and a base letter plus combining mark on the other.
- Why not normalize every stored nickname with FormKC instead of FormC?Compatibility normalization folds characters that only look related: the ligature fi becomes `fi` and a superscript ² becomes `2`, so `x²` turns into `x2`. That is useful for a derived search or matching key, but it rewrites what the user chose to display. Store the text in FormC and, if needed, keep a FormKC key beside it.
- Does normalizing a string keep its Length the same?No. `Normalize` can lengthen or shorten a string: FormD turns the precomposed "José" (4 chars) into 5 chars, and FormC does the reverse. Apply length limits to the normalized value, or a nickname that passed validation can exceed the limit once stored.
- Will string.Normalize stop someone registering a nickname that uses a Cyrillic а in place of a Latin a?No. Cyrillic а (U+0430) and Latin a (U+0061) are distinct letters, not canonical or compatibility equivalents, so every normalization form leaves them apart. Catching lookalike impersonation needs a separate confusables check; normalization only unifies different encodings of the same character.
- The nickname check passes its tests locally but lets both forms of "José" through in a container image; what should you check?Whether the app runs with `InvariantGlobalization` set to `true`. In globalization-invariant mode .NET skips normalization: `string.Normalize()` returns its input unchanged and `IsNormalized()` reports `true`, so the precomposed and decomposed forms are never unified. Either turn the setting off for that service or move uniqueness checks to a component that has the culture data.
Like the same quantity written as 1/2 and 0.5: equal in meaning, different on paper, so a string-by-string check needs both rewritten into one agreed notation first.
- If two strings render identically, == on them must return true.
- string.Normalize() with no arguments uses FormD.
- FormKC is the safe default for every piece of stored text.
- Normalization also lowercases text, so it replaces case-insensitive matching.
- Normalization maps lookalike letters from other scripts onto Latin letters.