An email-digest sender intermittently raises UnicodeEncodeError on some subject lines — how do you diagnose and fix it?

## The failure is data-dependent The first thing to establish is that a `UnicodeEncodeError` is **never random**. Encoding a given string with a given codec either works or fails, every time, on every machine. When a service fails “intermittently” the variable is the *input*, and the investigation is therefore about which records reached the failing line rather than about timing or load. In a digest sender that is easy to demonstrate: if most subject lines are served from a cache of previously rendered text — say an 83% cache-hit rate — then only the freshly rendered minority ever traverses the encode path with new user data in it, and the failure rate tracks the miss rate rather than traffic. That reframing alone rules out the tempting but wrong theories: - it is not an intermittent timeout to the mail relay, - and it is not a race between workers. ## Interrogate the exception object Step two is to make the exception talk. `UnicodeEncodeError` is a `UnicodeError` and therefore a `ValueError`, and the instance carries the same five attributes as its decoding counterpart: - `encoding` names the target codec, - `object` is the whole `str` being encoded, - `start` and `end` bound the failing slice, - and `reason` explains it. Wrap the failing boundary in a handler that logs `exc.encoding`, `exc.reason`, `exc.start` and `ascii(exc.object[exc.start:exc.end])`. Using `ascii()` matters: printing the raw character to a log stream that itself cannot encode it produces a second `UnicodeEncodeError` inside the error handler, which is how these incidents turn into empty log lines. ## Two kinds of failure Step three is to read `reason`, because it partitions the problem cleanly. ### Real text, too-narrow codec If the reason is of the `ordinal not in range(128)` family, the string is legitimate text and the codec at that boundary is too narrow — an em dash or an accented name headed into an ASCII or legacy encoder. The correct fix is at the boundary: encode with a codec that covers the text. Reaching for `errors='replace'` here downgrades a subscriber's name to question marks in an outbound message, which is a product defect, not a fix. If some part of the pipeline genuinely cannot carry the characters — a protocol field with a restricted charset — then the transformation should be explicit and appropriate to that field's rules, applied deliberately rather than as a codec error handler bolted onto a generic write. ### Escaped bytes, not text If the reason is `surrogates not allowed`, the diagnosis is completely different. The string contains lone surrogates in U+DC80–U+DCFF, which means some earlier decode used `errors='surrogateescape'` — very often implicitly, by reading a filename, an environment variable or a command-line argument. No handler at the write site is the right answer, because the string is not text: it is a carrier for raw bytes. Walk back to the boundary that produced it and decide there whether to keep the bytes, reject the record, or convert it to a printable form. ## Where a lossy handler belongs Step four is placement of any lossy handler you do add. Split the boundaries by audience. - **For diagnostics**, a helper such as `text.encode("utf-8", errors="backslashreplace").decode("ascii")` guarantees a log line can always be written and keeps the byte values visible, so the next occurrence is self-describing. - **For the outbound payload**, keep `'strict'` and let the failure be loud, because a silently mangled subject line reaches a person. - `'ignore'` deserves particular suspicion: it removes characters with no signal at all, so a run that “succeeded” may have shipped a truncated name that nobody notices for months. ## Prevention Step five is to keep the finding from recurring. 1. Add a **regression test** that feeds the exact offending string — the one you recovered from `exc.object` — through the same boundary; unlike the production incident, that test is fully deterministic. 2. Emit a **counter** whenever a lossy handler actually fires, so degradation is measurable rather than invisible. 3. And record the **encode/decode contract** for each boundary in the code that owns it, since these bugs are almost always a boundary whose codec was never chosen consciously. ## The shape of the answer The answer that lands in an interview is the shape of the reasoning: - exceptions of this class are data-dependent, - the exception object identifies the exact character and codec, - `reason` distinguishes “wrong codec for real text” from “string is carrying escaped bytes”, - and error handlers belong on human-facing output, never on the payload.