skip to content

In a C# chat app, two nicknames both display as "José" yet == says they differ; why, and how does string.Normalize fix it?

level: seniorimportance: should knowfreq 30%

answer

  1. one letter, two encodings
  2. a combining accent mark
  3. ordinal compares code units
  4. the parameterless overload's form

basics

~20 s

Unicode stores é either as one precomposed code point (U+00E9) or as e plus U+0301 COMBINING ACUTE ACCENT; == compares UTF-16 code units ordinally, so they differ. Normalizing both with string.Normalize(), which uses FormC, makes them equal.

solid answer

~40 s

The two strings are **canonically equivalent** but not identical: one holds U+00E9, the other `e` followed by U+0301, so their lengths are 4 and 5. C# `==` and the default comparer of `HashSet<string>` and `Dictionary<string, TValue>` are ordinal, so both forms can be registered as separate nicknames. `string.Normalize()` (FormC by default) rewrites either into the precomposed form, after which `==` is true; `IsNormalized()` checks without rewriting. Normalize identifiers once at the boundary and normalize the probe before every lookup. Avoid FormKC on stored text — it folds fi into `fi` and ² into `2` — and remember normalization neither folds case nor unifies lookalike letters from other scripts.

code

csharp · 14 lines
csharp
using System.Text;

string typed = "José";     // precomposed é, U+00E9
string pasted = "José";   // e followed by U+0301 COMBINING ACUTE ACCENT

Console.WriteLine($"{typed.Length} {pasted.Length}");
Console.WriteLine(typed == pasted);
Console.WriteLine(typed.Normalize() == pasted.Normalize());
Console.WriteLine(pasted.IsNormalized());
Console.WriteLine(typed.Normalize(NormalizationForm.FormD).Length);

var nicknames = new HashSet<string> { typed };
Console.WriteLine(nicknames.Contains(pasted));
Console.WriteLine(nicknames.Contains(pasted.Normalize()));

go deeper

for a junior

Recall that one accented letter can be stored two ways, that == compares the stored code units, and that string.Normalize exists to make the two forms match.

for a middle

Explain precomposed versus decomposed forms, name the four NormalizationForm values with FormC as the default, and show the HashSet<string> lookup missing until both sides are normalized.

for a senior

Normalize at the boundary and before every lookup, validate lengths after normalizing, keep FormKC for derived keys, and confirm the service is not running in globalization-invariant mode, where Normalize does nothing.

for a principal

Set the rule for which form the system stores identifiers in, and make every intake path follow it, so uniqueness checks hold across clients and data imports.

## Canonical equivalence Unicode often has more than one way to encode the same letter. "é" exists as a single **precomposed** code point, U+00E9 LATIN SMALL LETTER E WITH ACUTE, and as a **decomposed** sequence: `e` (U+0065) followed by U+0301 COMBINING ACUTE ACCENT. The standard declares these **canonically equivalent**: they are meant to render and behave identically, so a reader cannot tell them apart. Which form a string arrives in depends on where it came from. A keyboard on one platform may produce the precomposed form, text pasted from a document or a file name from another system may arrive decomposed, and some input methods build accented letters from combining marks. A chat app that accepts nicknames from all of these will see both. Nothing on screen gives the difference away: the two forms of "José" render the same glyphs, but one is 4 UTF-16 code units and the other 5, and a code-point dump shows `U+00E9` on one side and `U+0065 U+0301` on the other. The bug report therefore usually arrives as "two users with the same name" or "search cannot find a name I can see", not as an encoding problem. ## What `==` and the default comparers do In C#, `==` on two strings and `string.Equals(string)` perform an **ordinal** comparison: they compare UTF-16 code units one by one. "José" is 4 code units and "José" is 5, so `==` returns `false`. The default comparer used by `Dictionary<string, TValue>` and `HashSet<string>` is ordinal as well, so a set holding one form does not contain the other. Culture-aware comparisons do treat canonically equivalent strings as equal — the lesson on comparison and culture covers when those are appropriate — but identifiers and dictionary keys are usually compared ordinally, and that is where duplicates slip through. ## The four `NormalizationForm` values `string.Normalize(NormalizationForm)` rewrites a string into one canonical shape, and `string.IsNormalized(NormalizationForm)` reports whether it already is. The parameterless overloads use **FormC**. | Form | What it does | "José" → | Typical use | |---|---|---|---| | `FormC` | canonical decomposition, then recomposition | 4 chars | the default for stored and compared text | | `FormD` | canonical decomposition only | 5 chars | stripping accents, some search indexing | | `FormKC` | compatibility decomposition, then recomposition | 4 chars | derived matching keys | | `FormKD` | compatibility decomposition only | 5 chars | derived search keys | The **K** forms also fold *compatibility* characters, which only look related: the ligature fi (U+FB01) becomes `fi`, and a superscript ² becomes `2`, so `"x²"` turns into `"x2"`. That is useful for matching but changes meaning, so it should never overwrite the text a user typed. ## Where to normalize 1. **At the boundary.** Normalize nicknames, tags and other identifiers to FormC when they enter the system — registration, rename, import — so every stored copy has one shape. 2. **Before every lookup.** Normalize the probe string too; a stored FormC key is useless if the search term arrives decomposed. 3. **Before length checks.** `Normalize` can change `Length` in either direction, so validate limits on the normalized value. 4. **Keep the original when display matters.** If you need FormKC for matching, store it as a separate key next to the text as typed. ## What normalization does not do - **It does not fold case.** "JOSÉ" and "José" stay different; case-insensitive matching is a comparison choice, not a normalization form. - **It does not unify lookalikes from other scripts.** Cyrillic а (U+0430) and Latin a (U+0061) are different letters, and every form leaves them apart; catching impersonation needs a separate confusables check. - **It does not repair invalid UTF-16.** `string.Normalize()` and `IsNormalized()` throw `ArgumentException` on a string containing a lone surrogate, so a string damaged by an earlier bad cut fails here. - **It does nothing in globalization-invariant mode.** With `InvariantGlobalization` set to `true`, a common setting for small container images, `Normalize` returns its input unchanged and `IsNormalized` returns `true` for any string, so the two "José" forms stay different. - **It is not free.** `Normalize` can hand back the same instance when nothing changes (an all-ASCII string returns at once), but other text is still scanned, so normalize once at the boundary rather than on every comparison. ## The diagnosis in one line When two strings print the same but `==` says `false`, dump their code points — `string.Join(" ", s.EnumerateRunes().Select(r => $"U+{r.Value:X4}"))` — and you will usually find a precomposed letter on one side and a base letter plus combining mark on the other.

  • Why not normalize every stored nickname with FormKC instead of FormC?
    Compatibility normalization folds characters that only look related: the ligature fi becomes `fi` and a superscript ² becomes `2`, so `x²` turns into `x2`. That is useful for a derived search or matching key, but it rewrites what the user chose to display. Store the text in FormC and, if needed, keep a FormKC key beside it.
  • Does normalizing a string keep its Length the same?
    No. `Normalize` can lengthen or shorten a string: FormD turns the precomposed "José" (4 chars) into 5 chars, and FormC does the reverse. Apply length limits to the normalized value, or a nickname that passed validation can exceed the limit once stored.
  • Will string.Normalize stop someone registering a nickname that uses a Cyrillic а in place of a Latin a?
    No. Cyrillic а (U+0430) and Latin a (U+0061) are distinct letters, not canonical or compatibility equivalents, so every normalization form leaves them apart. Catching lookalike impersonation needs a separate confusables check; normalization only unifies different encodings of the same character.
  • The nickname check passes its tests locally but lets both forms of "José" through in a container image; what should you check?
    Whether the app runs with `InvariantGlobalization` set to `true`. In globalization-invariant mode .NET skips normalization: `string.Normalize()` returns its input unchanged and `IsNormalized()` reports `true`, so the precomposed and decomposed forms are never unified. Either turn the setting off for that service or move uniqueness checks to a component that has the culture data.

Like the same quantity written as 1/2 and 0.5: equal in meaning, different on paper, so a string-by-string check needs both rewritten into one agreed notation first.

  • If two strings render identically, == on them must return true.
  • string.Normalize() with no arguments uses FormD.
  • FormKC is the safe default for every piece of stored text.
  • Normalization also lowercases text, so it replaces case-insensitive matching.
  • Normalization maps lookalike letters from other scripts onto Latin letters.