A Rust signup form caps display names at 12 characters with `name.len() <= 12`; why does it reject "Zoë Šimićová", and what should it count?
answer
- three views of one str
- len is bytes, chars are scalar values
- char_indices pairs offsets with chars
- graphemes need a crate
basics
~10 sstr::len counts UTF-8 bytes, and that 12-character name is 16 bytes. name.chars().count() counts Unicode scalar values; for what a reader sees as letters, count grapheme clusters, which std does not provide.
solid answer
~40 sA Rust `str` can be read three ways. `bytes()` yields the UTF-8 `u8`s, and `len()` is their count; `chars()` decodes Unicode scalar values as `char`; `char_indices()` yields each `char` with its starting byte offset. "Zoë Šimićová" has 12 `char`s but its four accented letters take two bytes each, so `len()` is 16 and the check fails. Counting `chars().count()` fixes the common case, in O(n). It still over-counts letters built from several scalar values: an `é` typed as `e` plus U+0301 is two `char`s, and emoji sequences can be many more. Counting user-visible letters means grapheme clusters, which std does not implement; a crate such as unicode-segmentation does. Keep `len()` for byte budgets such as a database column.
code
rust · 15 linesfn main() {
let limit = 12;
for name in ["Ada Lovelace", "Zoë Šimićová", "Jose\u{301} Ortega"] {
println!(
"{name:?}: len={} chars={} len_ok={} chars_ok={}",
name.len(),
name.chars().count(),
name.len() <= limit,
name.chars().count() <= limit
);
}
for (offset, c) in "Zoë!".char_indices() {
println!("{offset}: {c}");
}
}go deeper
Recall that len() counts bytes and chars().count() counts char values, and that accented letters take more than one byte in UTF-8.
Explain the three iterators, bytes, chars and char_indices, why char_indices offsets are slice positions, and why a char is a scalar value rather than a visible letter.
Pick the measure the requirement means: bytes for storage limits, chars for code-point limits, grapheme clusters from a crate for visible letters, and guard O(n) counting with a byte cap.
Push product to define what a character limit means before code is written, and keep the same definition across the frontend, backend and database so names are not accepted in one layer and rejected in another.
## Three views of the same `str` A Rust `str` stores UTF-8, and the standard library gives three iterators over it. Choosing the right one is the whole question. | Method | Yields | Unit | Count it with | |---|---|---|---| | `bytes()` | `u8` | one UTF-8 byte | `len()`, O(1) | | `chars()` | `char` | one **Unicode scalar value** | `chars().count()`, O(n) | | `char_indices()` | `(usize, char)` | a scalar value and its **byte offset** | rarely counted; used for positions | - **`len()`** is stored, so it is O(1), and it is a byte count. It is the right number for buffers, wire formats and database columns sized in bytes. - **`chars()`** decodes the bytes into `char` values. A `char` is a Unicode scalar value, always four bytes in memory, and has no notion of a "letter". - **`char_indices()`** is `chars()` plus the byte offset where each `char` starts. Those offsets are what you slice with later, because `&s[a..b]` takes byte positions. ## Why the validator rejects the name "Zoë Šimićová" is twelve `char`s: `Z o ë space Š i m i ć o v á`. Four of them (`ë`, `Š`, `ć`, `á`) take two bytes in UTF-8, so `len()` returns 16 and `name.len() <= 12` is false. The rule says characters, and the code measured bytes. Any user whose name leaves ASCII gets a smaller limit than everyone else, which is both a bug and a fairness problem. ## Scalar values are not letters either Switching to `name.chars().count() <= 12` accepts that name, but `char` is not what a reader calls a character. 1. **Combining marks.** `é` can be one scalar value (U+00E9) or two, `e` followed by U+0301 COMBINING ACUTE ACCENT. Both render identically. "Jose\u{301} Ortega" shows eleven letters but `chars().count()` returns 12. 2. **Emoji sequences.** A person-plus-microscope scientist emoji is three scalar values joined by U+200D ZERO WIDTH JOINER, and 11 bytes. 3. **Other scripts.** Many scripts build one visible syllable from several scalar values; the Rust Book's Devanagari example shows a word of four letters made from six `char`s. What a user sees as one letter is a **grapheme cluster**, defined by Unicode's segmentation rules. The standard library deliberately does not implement them, because the tables are large and change with Unicode versions. The Book points to crates.io for that, and unicode-segmentation is the usual choice. ## Picking the measure on purpose | Rule in the spec | Measure in Rust | |---|---| | fits a 64-byte column or packet field | `name.len() <= 64` | | at most N code points, matching a database that counts characters | `name.chars().count() <= n` | | at most N visible letters | grapheme clusters from a segmentation crate | Two practical notes: - `chars().count()` walks the whole string. If the input may be huge, reject on `len()` first (a string of N bytes has at most N `char`s), then count. - Two strings can render the same and still compare unequal (`é` as one or two scalar values). Comparing or counting user names often needs Unicode normalisation first, which is also outside std. ## Using `char_indices` for positions When you need to find where something is, not how many there are, use `char_indices()` or `find`. Both give **byte offsets**, which slice in O(1): - `name.char_indices().nth(3)` gives the offset and value of the fourth `char`. - `name.chars().enumerate()` gives a running *character* count instead, which is not a valid slicing position once text leaves ASCII. That difference trips people up: an `enumerate` index used in `&name[..i]` slices at the wrong byte, and can panic in the middle of a character. ## Small helpers that go with the three views - **`is_ascii()`** is a quick check: when it returns true, every `char` is one byte, so `len()` equals `chars().count()`. - **`char::len_utf8()`** tells you how many bytes one `char` takes, from 1 to 4, which is handy when accumulating a byte budget character by character. - **`chars().rev()`** works because `Chars` is double-ended, but it reverses scalar values, so combining accents and emoji sequences come out broken. - **`split_whitespace()` and `lines()`** return `&str` pieces cut at valid boundaries, so they never split a character. None of these knows about grapheme clusters; that boundary is the same for every std method.
- Why does `name.chars().enumerate()` give the wrong position to slice with?`enumerate` counts `char`s, but `&name[a..b]` takes byte offsets. After the first multi-byte character the two numbers diverge, so slicing with the `enumerate` index cuts at the wrong place and can panic inside a character. `char_indices()` yields the byte offset of each `char`, which is what slicing needs.
- Is `name.chars().count()` cheap?No, it is O(n): it decodes every byte, unlike `len()`, which is stored. On untrusted input, reject anything whose `len()` exceeds four times the character limit first, since a `char` takes at most four UTF-8 bytes, and only then count.
- What does `chars().rev()` give for a name ending in an emoji sequence?It reverses scalar values, not visible letters, so a zero-width-joiner emoji or a letter with a combining accent comes out split and reordered. Reversing text for display needs grapheme clusters from a segmentation crate.
A str is like a ruler marked in millimetres while the spec talks about words. len() reads the millimetres, chars() counts letters printed on it, and grapheme clusters count what a reader would call one sign. All three are honest measurements; a length limit has to name which one it means.
- str::len returns the number of characters in the string.
- A Rust char is always what a user sees as one letter.
- chars().enumerate() yields byte offsets that are safe to slice with.
- chars().count() is O(1) because the length is cached.
- std can split text into grapheme clusters with a built-in method.