When encoding with Python's 'utf-16' codec, what does it write that 'utf-16-le' does not?

## Why UTF-16 needs a mark and UTF-8 does not UTF-16 stores text in two-byte code units, and UTF-32 in four-byte units. A multi-byte unit can be laid down low byte first (little-endian) or high byte first (big-endian), and the bytes alone do not say which. That is the problem the **byte-order mark** solves: encode U+FEFF at the very start, and a reader that sees `\xff\xfe` knows the stream is little-endian while one that sees `\xfe\xff` knows it is big-endian. The choice is unambiguous because U+FFFE is not a valid character, so the reversed reading is never a legitimate document. UTF-8 has single-byte code units and therefore no endianness at all, which is why the same three bytes there are only a *signature*, not a byte-order mark. ## The two families of codec name Python exposes both roles, and the name tells you which you get. **`utf-16` (unsuffixed) manages the mark for you.** - On encode it writes U+FEFF first, in the host's native order — `sys.byteorder` decides — and then the code units in that same order. - On decode it looks at the first two bytes, uses a mark to select the byte order, and strips it, so the mark never reaches your string. If there is no mark, CPython falls back to the host's native order, which is exactly why a file produced on the other endianness and stripped of its mark decodes to mojibake rather than raising. **`utf-16-le` and `utf-16-be` fix the order in the name and do nothing else.** - On encode they write no mark at all. - On decode they consume no mark, so `b"\xff\xfeo\x00k\x00".decode("utf-16-le")` gives `'\ufeffok'`: a U+FEFF glued to the front of your data, the same invisible passenger the UTF-8 signature produces. `utf-32`, `utf-32-le` and `utf-32-be` split identically, with a four-byte mark. ## The constants `codecs` names all of them. `codecs.BOM_UTF16_LE` is `b"\xff\xfe"`, `codecs.BOM_UTF16_BE` is `b"\xfe\xff"`, and `codecs.BOM_UTF16` is whichever of the two matches `sys.byteorder`. `codecs.BOM_UTF32_LE` is `b"\xff\xfe\x00\x00"`, `codecs.BOM_UTF32_BE` is `b"\x00\x00\xfe\xff"`, and `codecs.BOM_UTF32` is again the native one. One consequence matters when you sniff a file's first bytes: the UTF-32 little-endian mark *begins with* the UTF-16 little-endian mark, so a sniffer must test the four-byte forms before the two-byte forms or it will call every UTF-32 file UTF-16. ## Where this bites in practice **Appending.** `open()` creates a fresh incremental encoder per stream, and the unsuffixed codec writes its mark at the start of that stream. Open a file in append mode with `encoding="utf-16"` on a second run and you get a second mark part-way through the file, which the reader hands back as a stray U+FEFF in the middle of a line rather than as ordering information. The fix is to write the file once with `utf-16`, or to append with `utf-16-le`/`utf-16-be` matching what the first write produced. **Chunked encoding.** The same asymmetry appears if you call `str.encode("utf-16")` on each chunk and concatenate: one mark per chunk. Encode the whole document once, or encode chunks with the suffixed codec after emitting the mark yourself. **Cross-platform files.** Because the unsuffixed encoder follows the *encoding* machine's native order, the same code produces different bytes on different hosts. That is fine as long as the mark travels with the file; it stops being fine the moment some tool strips the first two bytes, at which point the file only reads correctly on hosts of the original endianness. **Sizing assumptions.** A common companion misconception is that UTF-16 is two bytes per character. It is two bytes per *code unit*; characters outside the Basic Multilingual Plane occupy a surrogate pair, four bytes. So neither the mark nor the unit size makes byte counting a proxy for character counting. ## The rule of thumb Use the unsuffixed `utf-16` when you are reading or writing a whole self-describing file and want the mark handled for you. Use `utf-16-le` or `utf-16-be` when an external contract fixes the byte order — a wire format, a fixed-layout record, an appended stream — and you want the codec to add nothing you did not ask for. ## Round-tripping is not automatic A detail that catches people: `"ok".encode("utf-16").decode("utf-16")` round-trips, and so does `"ok".encode("utf-16-le").decode("utf-16-le")`, but crossing the families does not. Encode with the unsuffixed codec and decode with the suffixed one and you gain a leading U+FEFF; encode with the suffixed codec and decode with the unsuffixed one and the first two bytes of real text are read as a byte-order mark if they happen to look like one, or silently interpreted in native order if they do not. Neither mismatch raises. So the codec name is part of the file's contract, not an implementation detail one side can change: whoever writes and whoever reads must agree on which of the two families is in play, and the cheapest way to make that agreement visible is to record it beside the format rather than infer it per site.