When does Python raise UnicodeDecodeError rather than UnicodeEncodeError?

## The name tells you the direction Python has exactly two codec failures worth memorising, and the verb in each name is the operation that failed. - `UnicodeDecodeError` is raised by **decode**: `bytes` in, `str` out. It means the octets you were handed are not valid input for the codec you named. - `UnicodeEncodeError` is raised by **encode**: `str` in, `bytes` out. It means the codec you named has no way to represent a code point you gave it. Both inherit from `UnicodeError`, which inherits from `ValueError`, so a broad `except ValueError` will swallow them — usually to your cost, because the message is the diagnostic. ## Decode failures UTF-8 is a self-validating format: not every octet sequence is legal. Decoding fails on - an **invalid start byte** — an octet such as `0xFF` that begins no valid sequence; - **unexpected end of data** — a multi-octet sequence cut short, which is what you get when a stream is chunked and you decode a partial buffer; - an **invalid continuation byte** — the second octet of a sequence not having the expected high bits, which is the usual symptom of text that is actually in some single-octet legacy encoding. ```python b"\xff\xfe".decode("utf-8") # UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte ``` The exception message names the codec, the offending octet, its position and the reason. In production that position is the lead: if it is 0, you probably have the wrong codec entirely or a byte-order mark you did not expect; if it is deep into a large payload, you more likely have one bad record in an otherwise clean stream. The important asymmetry is that **some codecs cannot fail on decode at all**. latin-1 maps each of the 256 possible octet values onto the code points U+0000 to U+00FF, so every byte string decodes successfully. That is why guessing latin-1 never raises and never helps: you get plausible-looking garbage — the *mojibake* where `café` shows up as two characters instead of one — instead of an exception pointing at the real problem. It is also why latin-1 is occasionally used deliberately as a lossless octet-preserving round trip when a layer insists on handing you text. ## Encode failures Going the other way, the failure is about coverage. ASCII covers 128 code points; latin-1 covers 256. Ask either to represent a euro sign, a CJK character or an emoji and it cannot: ```python "€".encode("ascii") # UnicodeEncodeError: 'ascii' codec can't encode character '\u20ac' in position 0: ordinal not in range(128) ``` UTF-8 covers the whole Unicode range, so an encode to UTF-8 almost never fails — with one real exception. **Lone surrogates**, code points in U+D800-U+DFFF, exist only as a UTF-16 encoding mechanism and are not valid characters; UTF-8 refuses them. You can end up holding one when text arrived through a layer that decoded UTF-16 sloppily, or through a filesystem API that smuggles undecodable octets into a `str`. Encoding such a value to UTF-8 raises `UnicodeEncodeError` with the reason "surrogates not allowed", and it is one of the genuinely puzzling production failures on this topic because most people believe UTF-8 encoding cannot fail. ## Diagnosing, not suppressing A codec exception is almost always a **wrong assumption about the source**, not a defect in the data. Work the question backwards: 1. Which boundary produced these octets — a socket, a file opened in binary mode, a subprocess pipe, a database driver? 2. Does that source *declare* its encoding out of band, in an HTTP header, an XML declaration or a file-format header? If yes, use the declaration rather than a guess. 3. If nothing declares it, whose encoding is it in practice? A single upstream producer with a legacy default is the common answer, and once identified the fix is one explicit codec name at the boundary. 4. Only then decide what to do about genuinely malformed input. The anti-pattern is turning the exception off and shipping. Whatever the strategy for bad input, choosing it before you know the source's real encoding converts a loud failure with an exact byte offset into a silent data-corruption bug that surfaces weeks later in someone else's report. ## What an interviewer is checking That you can say the direction of each error without hesitating, that you know the message contains the position and reason, and that you treat the exception as evidence about the source rather than an obstacle. The candidate who immediately proposes suppressing the error has told you they will corrupt data quietly.