skip to content

A C# chat app builds 40-character message previews with message.Substring(0, 40), and some previews end in a garbled symbol; why, and how do you truncate safely?

level: seniorimportance: should knowfreq 40%

answer

  1. the cut lands mid-character
  2. a lone half after encoding
  3. what .NET calls a grapheme
  4. advance one element at a time

basics

~10 s

Substring counts UTF-16 code units, so index 40 can fall inside a surrogate pair or a multi-rune emoji; a lone half is encoded as U+FFFD. Truncate by text elements with a StringInfo.GetNextTextElementLength loop instead.

solid answer

~40 s

`Substring` and range slicing take code-unit indexes and never check Unicode, so a cut at 40 can leave a lone high surrogate or split a flag or ZWJ emoji. `Substring` does not throw; the damage appears later, when `Encoding.UTF8` writes the lone surrogate as U+FFFD (`EF BF BD`) — the garbled symbol. The fix is to cut only at **text-element** boundaries: loop with `StringInfo.GetNextTextElementLength(text, index)`, counting elements until you reach 40, then slice at that index. Since .NET 5, `StringInfo` follows Unicode extended grapheme clusters, so flags, ZWJ sequences and combining accents each count as one. Decide whether the limit means visible characters, `char`s or UTF-8 bytes, and measure in that unit while still cutting on element boundaries.

code

csharp · 22 lines
csharp
using System.Globalization;
using System.Text;

string message = "Family dinner tonight, bring dessert!! 👨‍👩‍👧 see you at 7";

string naive = message.Substring(0, 40);
Console.WriteLine($"naive ends with U+{(int)naive[^1]:X4}");
Console.WriteLine(Convert.ToHexString(Encoding.UTF8.GetBytes(naive))[^6..]);

string safe = Preview(message, 40);
Console.WriteLine($"safe: {new StringInfo(safe).LengthInTextElements} elements, {safe.Length} chars");

static string Preview(string text, int maxElements)
{
    int index = 0, count = 0;
    while (index < text.Length && count < maxElements)
    {
        index += StringInfo.GetNextTextElementLength(text, index);
        count++;
    }
    return text[..index];
}

go deeper

for a junior

Recall that Substring counts chars, that an emoji can be two or more of them, and that cutting in the middle damages the text.

for a middle

Explain how a lone surrogate becomes U+FFFD in UTF-8, and why runes alone do not fix flags, ZWJ families or combining accents.

for a senior

Diagnose the bug from the EF BF BD bytes, write the GetNextTextElementLength loop, and ask whether the limit means visible characters, chars or UTF-8 bytes before choosing the stopping rule.

for a principal

Push for one shared truncation helper with Unicode test fixtures, so every service and client cuts previews the same way and limits are defined in a named unit.

## Why the cut breaks `string.Substring(int, int)`, range slicing such as `text[..40]` and `string.Remove` all work in **UTF-16 code units**. They do not inspect Unicode and they do not throw when an index falls in an awkward place. In the example message, the first 39 code units are plain ASCII and code unit 39 is the high surrogate of 👨, the first emoji in a family sequence. `Substring(0, 40)` therefore returns a string that ends in a **lone high surrogate**, U+D83D — a value that is not a character on its own. The same failure has two levels: - **Splitting a surrogate pair** produces ill-formed UTF-16, as above. - **Splitting a grapheme cluster** produces well-formed but wrong text: cutting 👨‍👩‍👧 after the first rune leaves a man followed by a dangling ZERO WIDTH JOINER, and cutting 🇬🇪 after one regional indicator leaves a stray letter-symbol instead of a flag. ## What the broken string turns into Nothing fails at the `Substring` call. The damage shows up later, wherever the string is encoded or rendered: 1. `Encoding.UTF8.GetBytes` uses a replacement fallback, so the lone surrogate is written as U+FFFD — bytes `EF BF BD`. That is the "garbled symbol" users see, and once stored it is permanent. 2. A strict encoder, `new UTF8Encoding(false, true)`, throws `EncoderFallbackException` instead, which is better than silent corruption when you control the code. 3. `string.Normalize()` throws `ArgumentException` on the lone surrogate, so a later normalization step can crash on what looked like a harmless preview. ## Truncating by text element .NET calls a user-perceived character a **text element**; since .NET 5, `StringInfo` follows the Unicode extended grapheme cluster rules, so flags, ZWJ sequences, skin-tone modifiers and combining accents each count as one. The workhorse is `StringInfo.GetNextTextElementLength(string, int)`, which returns how many `char`s the text element starting at an index occupies (0 at the end of the string). A span overload, `GetNextTextElementLength(ReadOnlySpan<char>)`, does the same without a string. ```csharp static string Preview(string text, int maxElements) { int index = 0, count = 0; while (index < text.Length && count < maxElements) { index += StringInfo.GetNextTextElementLength(text, index); count++; } return text[..index]; } ``` The loop advances one whole text element at a time, so the cut can only land on a boundary. On the example message it keeps 40 text elements, which is 47 `char`s because the family emoji alone is 8. `new StringInfo(s).SubstringByTextElements(0, 40)` is a shorter alternative, but it throws `ArgumentOutOfRangeException` when the string has fewer than 40 elements, so a preview helper needs a length check first. ## Choosing the unit of the limit | The limit really means | Measure | Cut with | |---|---|---| | Characters a reader sees | text elements | `StringInfo.GetNextTextElementLength` loop | | A column or buffer counted in `char`s | UTF-16 code units | the same loop, stopping before an element that would overflow | | A field counted in UTF-8 bytes | `Encoding.UTF8.GetByteCount` | the loop, summing each element's UTF-8 byte count | Whatever the unit, the rule is the same: **cut only at text-element boundaries**, and measure in the unit the limit is defined in. ## Reversing a nickname follows the same rule Reversing `ToCharArray()` swaps the halves of every surrogate pair and yields invalid UTF-16. Reversing runes keeps each scalar whole but reorders the parts of a cluster: 🇬🇪 is the regional indicators G then E, and reversed it reads E then G — the flag of Egypt. Collect the text elements with `StringInfo.GetTextElementEnumerator(string)`, reverse that list and concatenate. ## Review checklist - Search for `Substring`, `[..n]` and `Remove` applied to user text with a fixed number. - Ask what unit each limit is in, and make the client and server agree on it. - Prefer a strict `UTF8Encoding` in tests so a split pair fails loudly. - Add test data with a flag, a ZWJ family, a skin-tone emoji and a combining accent at the cut position. - Append an ellipsis only after the safe cut, and count it against the limit if the limit is strict. - Remember that data already stored with U+FFFD cannot be repaired from the string alone; it has to be re-fetched from the original message. The same bug appears anywhere a fixed index meets user text: notification bodies, push-message titles, log lines trimmed to a width, and search snippets. One helper, tested once with hostile data, fixes all of them.

  • The preview goes into a column limited to 40 UTF-16 code units, not 40 visible characters; how does the loop change?
    Stop before an element that would overflow the budget: compute `int next = StringInfo.GetNextTextElementLength(text, index)` and break when `index + next > maxChars`, otherwise advance. The result never splits a pair or a cluster and never exceeds the column, at the cost of sometimes keeping fewer than 40 code units.
  • Why does reversing a nickname with Array.Reverse on ToCharArray() break emoji, and does reversing runes fix it?
    Reversing `char`s swaps the halves of every surrogate pair, putting a low surrogate before a high one — invalid UTF-16. Reversing runes keeps each scalar whole but still reorders cluster parts: 🇬🇪 is regional indicators G then E, and reversed it becomes E then G, the flag of Egypt. Reverse the list of text elements from `StringInfo.GetTextElementEnumerator` instead.
  • Does string.Substring throw when its cut splits a surrogate pair?
    No. `Substring` and range slicing work on code-unit indexes and do not validate Unicode, so they return a string ending in a lone surrogate without complaint. The failure surfaces later: `Encoding.UTF8` substitutes U+FFFD, a strict `new UTF8Encoding(false, true)` throws `EncoderFallbackException`, and `string.Normalize()` throws `ArgumentException`.

  • Substring throws when it would split a surrogate pair, so this bug cannot be silent.
  • Checking char.IsHighSurrogate on the last char keeps flags and emoji families intact.
  • Truncating by runes instead of chars makes every preview render correctly.
  • The garbled symbol is a client font problem, not data corruption on the server.
  • Text elements are just another name for surrogate pairs.