A ticket-triage bot slices each Python str title to a fixed character budget and accents vanish - why, and how do you cut safely?

## What `len()` actually measures `len(s)` on a `str` returns the number of **code points**, and `s[:n]` cuts after `n` of them. That is a precise, useful definition, and it is not the definition a product manager has in mind when they say "trim titles to 40 characters". Three units are in play and they routinely disagree: - **bytes** — what a database column, a wire protocol or a log budget usually limits - **code points** — what `len()` and slicing count - **grapheme clusters** — what a reader calls a character The gap between the second and third is where this bug lives. ## Why an accent can disappear Text that arrives decomposed spells an accented letter as a base character followed by a **combining mark** — a zero-width code point that the renderer draws over the character before it. `"Caf\u00e9 outage"` is 11 code points; the NFD form of the same title is 12, because the accent is its own code point. Cut that decomposed title at 4 code points and you get `"Cafe"`: the base `e` survived, the mark did not, and the accent is silently gone. Cut it at 5 and you keep the mark. Cut a *different* title so that the slice starts with an orphaned mark and the renderer stacks that accent onto whatever character precedes it in the surrounding text — an ellipsis, a quote mark, a space. Nothing raises; the output is just wrong, and it is wrong only for some inputs. There is an ordering assumption underneath the naive slice: the code assumes each index step advances one visible character, and that assumption holds for every ASCII fixture in the test suite. Two related ordering facts make hand-rolled fixes fragile — a base can carry several marks, and different sources stack them in different orders, which is exactly what normalization's canonical ordering exists to settle. ## The safe cut Two steps, in order: 1. **Normalize to NFC on the way in.** For Latin text this collapses base-plus-mark back into one precomposed code point, so most cuts stop being dangerous at all, and it puts any remaining stacked marks into canonical order. 2. **Do not cut immediately before a combining mark.** Look at the character at the cut index; while `unicodedata.combining(ch)` is nonzero, move the cut one character earlier. `unicodedata.category(ch) == "Mn"` identifies the same characters as nonspacing marks. ```python import unicodedata def truncate(text: str, limit: int) -> str: text = unicodedata.normalize("NFC", text) cut = text[:limit] while cut and len(text) > len(cut) and unicodedata.combining(text[len(cut)]): cut = cut[:-1] return cut ``` That is honest about its scope: it prevents a mark from being orphaned or a base from being stripped of its mark. It does not make the count equal to what a reader sees. ## When code points are the wrong unit entirely For scripts and sequences where one visible character is many code points by design, no amount of normalization helps: - an emoji with a skin-tone modifier, or a sequence joined by ZERO WIDTH JOINER, is one cluster of several code points - a flag is two regional indicator code points - Indic and Thai text has clusters that NFC does not collapse - East Asian characters are one code point but two display columns wide, which matters for fixed-width layout The standard's answer is **grapheme cluster** segmentation, which the standard library does not implement; `unicodedata.east_asian_width` tells you about display width but not clustering. So the senior judgement is to decide which unit the requirement actually means. If it is a storage or protocol limit, measure `len(s.encode("utf-8"))` and cut on an encoded boundary. If it is a layout limit, you need width or clustering, and that is a dependency decision, not a slicing trick. If it is a loose "keep titles short", code points plus the combining-mark guard is enough. ## Operating it Normalize once when the record is created, not on every classification pass: on a hot path with a 92nd-percentile latency budget, per-request Unicode work on every stored title is avoidable cost, and the normalized value is the thing every later comparison and cut should use. Then make the failure visible — assert in tests with one decomposed and one emoji fixture, because an ASCII-only suite passes with the broken version. ## The short answer "`len` counts code points. A decomposed accent is two of them, so a slice can land between a letter and its mark. Normalize to NFC at the boundary and back the cut off while `unicodedata.combining` of the next character is nonzero. And if the requirement really means visible characters, code points are the wrong unit and I would say so."