BOM and utf-8-sig

## The code point and the bytes U+FEFF was originally ZERO WIDTH NO-BREAK SPACE. Unicode then repurposed the same code point as the **byte-order mark**: placed first in a UTF-16 or UTF-32 stream, it lets a reader work out which byte order the encoder used, because those encodings have multi-byte code units and therefore an endianness. UTF-8 has no such ambiguity — its code units are single bytes — so in a UTF-8 file the mark carries no ordering information at all. Some spreadsheet exporters and Windows editors write it anyway, as a **signature** meaning "the bytes that follow are UTF-8". Encoded as UTF-8, U+FEFF is the three bytes `\xef\xbb\xbf`, which Python spells `codecs.BOM_UTF8`. ## What each codec actually does `utf-8` is a faithful UTF-8 codec and nothing more. Decoding `b"\xef\xbb\xbfid,lat"` with it yields the string `'\ufeffid,lat'` — the mark is a valid character, so it decodes to a character. Encoding with `utf-8` never adds one. `utf-8-sig` is the same codec wrapped in signature handling: - **On decode** it removes exactly one leading `codecs.BOM_UTF8` if present, and otherwise decodes byte-for-byte identically to `utf-8`. That makes it *harmless on files that have no mark*. - **On encode** it unconditionally prepends the three bytes. The sentence worth memorising: **`utf-8-sig` is forgiving on input and opinionated on output.** Note the narrowness of the stripping. It removes one signature, only at the very start of the stream. A U+FEFF sitting in the middle of the document stays — correctly, because there it is data, not a signature. ## Why a stray mark hurts so much The character is zero-width. It does not render, most editors do not show it, and a terminal prints `\ufeffid` as something indistinguishable from `id`. Everything downstream that compares or parses that first field then fails in a way that reads like a ghost: - a header row's first name becomes `'\ufeffid'`, so a lookup keyed on `"id"` raises `KeyError` while the printed header looks perfect; - `int(first_field)` raises `ValueError` on what looks like a plain number; - `line.startswith("#")` is `False` for the first line of a file whose first line visibly starts with `#`; - a dict built from the file has a key that prints as `id` and compares unequal to `"id"`; - two files that differ only in whether the exporter wrote a signature produce different keys, so a merge silently drops rows. The diagnosis is mechanical, and knowing it is most of the value of this topic. Print `repr(field)` rather than `field`; check `ord(field[0]) == 0xFEFF`; call `unicodedata.name(field[0])` and see `ZERO WIDTH NO-BREAK SPACE`; or drop to bytes and compare `raw[:3]` against `codecs.BOM_UTF8`. Never diagnose by eye. ## Which codec where **Reading.** At any boundary where files arrive from spreadsheets, desktop editors or partner uploads, decode with `utf-8-sig`. It costs nothing on clean files and removes a whole bug class. Inside your own system, where you produced the file, plain `utf-8` is fine — but `utf-8-sig` is never *wrong* on the read side, which is why it is the better default at an ingest edge. **Writing.** Default to plain `utf-8`. Reach for `utf-8-sig` only where a consumer you do not control uses the signature to choose an encoding — chiefly desktop spreadsheet software, which otherwise opens a UTF-8 export in a legacy code page and shows mojibake. Handing a signed file to line-oriented tooling, to a parser that expects a leading `{` or `[`, or to a script whose first line is a shebang creates the mirror-image bug. One more encode-side trap. `open()` builds an *incremental* encoder that emits the signature once per stream, so appending to an existing file with `encoding="utf-8-sig"` writes a **second** signature into the middle of the file. Likewise, encoding chunk by chunk with `str.encode("utf-8-sig")` and concatenating gives one signature per chunk. Encode the document once, or append with plain `utf-8` after the first write. ## Two facts that settle most interview follow-ups First, the mark is not whitespace: `str.strip()` does not remove it, because U+FEFF is a format character (category `Cf`), not a space. Second, UTF-8 does not *need* a mark for correctness — anything claiming a UTF-8 file must start with one has confused the signature with the byte-order role it plays in UTF-16 and UTF-32.