Stripping a byte-order mark with text[1:] truncated real data in a 6,800-row geocoding batch — how should a Python ingest remove a BOM safely?

## What actually went wrong The first fix for a stray byte-order mark is nearly always `text[1:]` or `line[1:]`. It works on the file that prompted it and is wrong in general, because slicing removes *a* character, not *the mark*. Feed it a 6,800-row batch from a partner whose exporter does not write a signature and the first field of the first row quietly loses its leading character — `id` becomes `d`, an address `12 Bridge St` becomes `2 Bridge St`. Nothing raises. The batch runs green, the geocoder resolves fewer rows or resolves them to the wrong place, and the damage surfaces days later as a truncation nobody can reproduce because the file that reproduces it is the *clean* one. The general lesson: a removal must be conditional on the thing being present. Everything below is a way of making it conditional. ## Fix it at the decode boundary The cleanest answer is to never let the mark become a character. Decoding with `utf-8-sig` removes exactly one leading signature if it is there and decodes byte-for-byte identically to `utf-8` if it is not. It is idempotent in the sense that matters: safe on both kinds of file, so the ingest path needs no branch and no flag. That is why signature handling belongs at the edge where bytes become text, not sprinkled through the consumers. One function turns bytes into `str`; everything downstream receives text that is already clean; no parser, no key lookup, no comparison has to know that byte-order marks exist. ## If you already hold a str Sometimes the decode happened somewhere you do not control. Then the safe removals are: - `text.removeprefix("\ufeff")` — removes it if present, returns the string unchanged otherwise. Available since Python 3.9. - an explicit guard: `text[1:] if text.startswith("\ufeff") else text`. And the unsafe ones worth naming so you can reject them in review: - `text[1:]` — the truncation bug above. - `text.strip()` or `text.lstrip()` — U+FEFF is a format character in category `Cf`, not whitespace, so these do nothing to it while happily eating leading spaces you may have wanted. - `text.replace("\ufeff", "")` — removes marks *anywhere*, including a U+FEFF that is legitimate data in the middle of a document. - stripping per line rather than per file — a signature can only be at the very start of the stream; a per-line strip is a licence to corrupt every line. ## Mixed producers: sniff the bytes When files arrive from several sources, one partner eventually sends UTF-16. Read the first four bytes and dispatch on them: test `codecs.BOM_UTF32_LE` and `codecs.BOM_UTF32_BE` first, then `codecs.BOM_UTF16_LE` and `codecs.BOM_UTF16_BE`, then `codecs.BOM_UTF8`, falling back to `utf-8-sig`. The order matters: the UTF-32 little-endian mark begins with the UTF-16 little-endian mark, so testing two bytes first mislabels every UTF-32 file. Note also that a mark is *evidence*, not a guarantee — most UTF-8 files carry none, so the fallback has to be a real decision rather than a shrug. ## The write side The same batch usually emits files too. Default to plain `utf-8`; choose `utf-8-sig` only for an output whose named consumer needs the signature to pick an encoding. Two encode-side traps travel with that choice: `open()` builds a fresh incremental encoder per stream, so appending with `encoding="utf-8-sig"` plants a second signature mid-file, and calling `str.encode("utf-8-sig")` per chunk and concatenating gives one signature per chunk. Encode the document once. ## Making it stay fixed This class of bug returns because the test corpus contains only files that reproduce the original symptom. Pin both cases: - two fixtures, one with a signature and one without, pushed through the *same* ingest function; - assertions on exact equality of the parsed first key (`assert header[0] == "id"`), never on how the output looks — the character is zero-width, so a visual check passes on a corrupt file; - an assertion that a field which never had a mark comes back with its full length, which is the specific regression that slicing introduced; - when a partner file misbehaves in production, log `raw[:4]` in hex rather than the decoded text, since the decoded text is exactly what hides the problem. A useful operational habit is to record, per source, whether that producer emits a signature. It costs nothing, it turns "the batch truncated" into "this producer changed its exporter", and it makes the mixed-producer sniffing above auditable rather than magic.