skip to content

UTF-16 and Runes

A char is one UTF-16 code unit, so an emoji spans two chars and Length miscounts; Rune and text elements fix that. Interviewers ask how to truncate a string safely.

on this pageshow

explore

questions

5

In C#, why does the string "👍" report a Length of 2, and what does string.Length actually count?

level: juniorimportance: must knowfreq 50%

answer

  1. what a char really holds
  2. sixteen bits per char
  3. code points above U+FFFF
  4. high half, then low half

basics

~20 s

A C# char is one UTF-16 code unit and string.Length counts code units. The thumbs-up emoji is U+1F44D, above U+FFFF, so UTF-16 stores it as a surrogate pair of two chars and Length reports 2.

solid answer

~40 s

A .NET `string` is a sequence of UTF-16 code units and `char` is one 16-bit code unit, so `Length` counts code units, not what a reader sees. Code points up to U+FFFF fit in one `char`; anything above — most emoji, rarer CJK ideographs, historic scripts — is stored as a **surrogate pair**: a high surrogate (U+D800–U+DBFF) followed by a low surrogate (U+DC00–U+DFFF). `char.IsHighSurrogate`, `char.IsLowSurrogate` and `char.IsSurrogate` test one `char`, and `char.ConvertToUtf32(char, char)` combines a pair into its code point. So `"👍".Length` is 2 while `"Zoë".Length` is 3. `Length` is the right number for indexes and buffer sizes, and the wrong one for "how many characters did the user type".

code

csharp · 7 lines
csharp
string thumbs = "👍";
Console.WriteLine(thumbs.Length);
Console.WriteLine(char.IsHighSurrogate(thumbs[0]));
Console.WriteLine(char.IsLowSurrogate(thumbs[1]));
Console.WriteLine($"U+{char.ConvertToUtf32(thumbs, 0):X}");
Console.WriteLine($"U+{char.ConvertToUtf32(thumbs[0], thumbs[1]):X}");
Console.WriteLine("Zoë".Length);

go deeper

for a junior

Recall that char is a 16-bit UTF-16 code unit and that Length counts those units. Say the words surrogate pair and give one emoji whose Length is 2.

for a middle

Explain the high and low surrogate ranges, show char.ConvertToUtf32 combining a pair, and name what Length is right for: indexes, buffer sizes and limits that are truly counted in UTF-16 units.

for a senior

Show where code-unit counting leaks into production: character limits, previews and cursors that split a pair. Name Rune for scalar values and StringInfo for user-perceived characters as the replacements.

for a principal

Treat the word character as ambiguous in specs and reviews; ask whether a limit means UTF-16 units, scalar values, text elements or UTF-8 bytes before anyone writes the validation.

## What a `char` is In C#, `char` is an alias for `System.Char`, a 16-bit value type. The language reference describes it as a Unicode **UTF-16 code unit**, which is not the same thing as a character in the sense a reader means. A `string` is an immutable sequence of these code units, and `string.Length` returns how many code units the sequence holds. For most Western text the difference never shows. Every Unicode **code point** from U+0000 to U+FFFF (the *Basic Multilingual Plane*, or BMP) fits in one 16-bit unit, so "Zoë" has `Length` 3 and each `char` is one letter. The trouble starts with code points above U+FFFF. ## Surrogate pairs Unicode defines code points up to U+10FFFF, which 16 bits cannot hold. UTF-16 solves this with a **surrogate pair**: two code units taken from a range reserved for this purpose and never used for characters on their own. - A **high surrogate** lies in U+D800–U+DBFF and always comes first. - A **low surrogate** lies in U+DC00–U+DFFF and always comes second. - Together they encode one supplementary code point; apart, neither means anything. The thumbs-up emoji is U+1F44D, so in a .NET string it is stored as the pair U+D83D U+DC4D. That is why `"👍".Length` is 2: `Length` is counting the two halves of one code point. Most emoji, many rarer CJK ideographs, mathematical alphanumeric symbols and historic scripts all live above U+FFFF, so this is everyday data in a chat app, not an edge case. ## Inspecting a pair with `System.Char` `System.Char` has static helpers that work on code units and pairs: 1. `char.IsHighSurrogate(char)` and `char.IsLowSurrogate(char)` test one code unit for each half; `char.IsSurrogate(char)` is true for either. 2. `char.IsSurrogatePair(char, char)` checks that two code units form a valid pair in the right order. 3. `char.ConvertToUtf32(char, char)` combines a pair into the code point as an `int`; `char.ConvertToUtf32(string, int)` does the same at a string index, and returns the code unit itself for a BMP character. 4. `char.ConvertFromUtf32(int)` goes the other way and returns a one- or two-`char` string. ```csharp string thumbs = "👍"; Console.WriteLine(thumbs.Length); // 2 Console.WriteLine(char.IsHighSurrogate(thumbs[0])); // True Console.WriteLine($"U+{char.ConvertToUtf32(thumbs, 0):X}"); // U+1F44D ``` `char.ConvertToUtf32(string, int)` throws `ArgumentException` when the index lands on a low surrogate, or on a high surrogate with no low surrogate after it, so it has to be called at the start of a pair. ## What `Length` is good for — and what it is not `Length` is not wrong; it answers a specific question. The table maps common questions to the number that answers each one. | Question | Right measure in .NET | |---|---| | How big a `char[]` or `Span<char>` do I need? | `string.Length` | | Which index does `Substring` or `s[i]` take? | `string.Length` and code-unit indexes | | How many Unicode scalar values are there? | count of `string.EnumerateRunes()` | | How many characters does the user see? | `new StringInfo(s).LengthInTextElements` | | How many bytes in a UTF-8 file or payload? | `Encoding.UTF8.GetByteCount(s)` | Two letters can look the same and still differ in `Length` without any surrogate involved: "é" may be stored as the single code point U+00E9 or as `e` followed by U+0301 COMBINING ACUTE ACCENT, giving `Length` 1 or 2. That is a matter of Unicode composition, handled by `string.Normalize`, and it is a second reason `Length` is not a character count. ## Common traps - **Character limits.** A "max 40 characters" rule implemented with `Length` rejects a message of 21 emoji that the user sees as 21 characters, and counts an accent typed as a combining mark twice; decide which unit the limit means first. - **Cutting strings.** `Substring`, range slicing and `Remove` take code-unit indexes and do not check Unicode, so they can split a pair and leave a lone surrogate behind. - **Looping over `char`s.** `foreach (char c in s)` hands you halves of pairs; `char.IsLetter(c)` on a lone high surrogate returns false even when the pair is a letter. `char.IsLetter(string, int)` and `System.Text.Rune` look at the whole pair. - **Reversing.** Reversing a `char[]` swaps the halves of each pair and produces invalid UTF-16. The core rule to carry into an interview: in C#, `char` means *code unit*, `Length` counts code units, and anything that must count what a person sees has to use a different API.

  • What happens if you call char.ConvertToUtf32(s, 1) where s is "👍" and index 1 is the low surrogate?
    It throws `ArgumentException`. `ConvertToUtf32(string, int)` expects the index to land on a BMP character or on the high surrogate that starts a pair; a low surrogate at that index, or a high surrogate with nothing valid after it, is rejected. Walk from the start of the string, or use `Rune.TryGetRuneAt` or `EnumerateRunes()`, instead of indexing into the middle of a pair.
  • Does the letter é always have a Length of 1 in a C# string?
    No. The precomposed form U+00E9 is one `char`, but the same letter can be stored as `e` followed by U+0301 COMBINING ACUTE ACCENT — two `char`s that render identically. Both code points are in the BMP, so no surrogate is involved; the difference comes from Unicode composition, which `string.Normalize` can unify.

  • A C# char can hold any Unicode character, so Length equals the character count.
  • string.Length counts bytes, and the emoji happens to take two bytes.
  • Surrogate pairs only appear in rare historic scripts, never in everyday chat text.
  • A surrogate pair is two separate characters that happen to sit side by side.
  • Substring always returns valid text whatever index you pass it.
open as a page

In .NET, what does string.EnumerateRunes() give you that a foreach over the string's chars does not, and where does a Rune still fall short?

level: middleimportance: should knowfreq 35%

basics

~20 s

EnumerateRunes yields System.Text.Rune values, one per Unicode scalar value, so a surrogate pair arrives as one item and a lone surrogate as U+FFFD. A flag, a ZWJ emoji or an accented letter can still span several runes.

open as a page

In a C# chat app, two nicknames both display as "José" yet == says they differ; why, and how does string.Normalize fix it?

level: seniorimportance: should knowfreq 30%

basics

~20 s

Unicode stores é either as one precomposed code point (U+00E9) or as e plus U+0301 COMBINING ACUTE ACCENT; == compares UTF-16 code units ordinally, so they differ. Normalizing both with string.Normalize(), which uses FormC, makes them equal.

open as a page

A C# chat app builds 40-character message previews with message.Substring(0, 40), and some previews end in a garbled symbol; why, and how do you truncate safely?

level: seniorimportance: should knowfreq 40%

basics

~10 s

Substring counts UTF-16 code units, so index 40 can fall inside a surrogate pair or a multi-rune emoji; a lone half is encoded as U+FFFD. Truncate by text elements with a StringInfo.GetNextTextElementLength loop instead.

open as a page

In C# 11 and later, what does a "PING"u8 literal produce, and when would you use it instead of Encoding.UTF8.GetBytes?

level: middleimportance: nice to knowfreq 22%

basics

~20 s

A u8 suffix makes the compiler encode the literal to UTF-8 at compile time and type it as ReadOnlySpan<byte>, so the call site allocates nothing. Use it for fixed protocol text; use Encoding.UTF8.GetBytes for strings known only at run time.

open as a page