Binary Encoding Primitives

## What byte order actually means A 32-bit integer is four bytes. Writing it to a flat stream forces a decision no single byte can express: which of those four is the most significant. The two answers have names. | Convention | Bytes on the wire for the value 0x0A0B0C0D | |---|---| | **Big-endian**, also called network byte order | `0A 0B 0C 0D` | | **Little-endian** | `0D 0C 0B 0A` | Both are complete, correct, lossless representations. Neither is discoverable from the bytes: `0D 0C 0B 0A` is a perfectly good big-endian number too, it just means something else. Byte order is therefore not a property of the data, it is a **term in the contract**, and a contract that omits it is incomplete. ## Which fields it touches This is where most confusion lives. The choice applies only to fields made of several bytes whose numeric significance is ordered: - **Fixed-width integers** - 16-, 32-, 64-bit fields: yes, this is the whole problem. - **A single byte** - a flag, a small enumerated value, a tag: no. There is nothing to order, and bit order inside a byte is not what endianness means. - **Byte strings and UTF-8 text**: no. The sequence of bytes *is* the value, in the order given. (The contrast is instructive: UTF-16 does have two byte orders for the same reason integers do, which is why it needs a byte-order mark and UTF-8 does not.) - **A base-128 variable-length integer**: no. Its groups run least-significant first by definition of the encoding, independent of what the format chose for its fixed-width fields. ## What a mismatch looks like The failure is quiet in a specific and dangerous way. 1. A 32-bit field holding **1** is written big-endian as `00 00 00 01` and read little-endian as **16,777,216**. 2. The same field holding **0** reads as **0** under both conventions. 3. Any value whose byte pattern happens to be a palindrome also survives. So a test suite whose fixtures are mostly zeros and empty records passes, and the bug surfaces later as absurd numbers in production: a count in the millions where a handful was expected, a size field that asks for gigabytes. The diagnosis is mechanical - take a hex dump of the field, interpret it both ways, and check whether the value you expected is the byte-reversed reading of the value you got. If it is, the byte order is the cause and nothing else needs investigating. ## How formats settle it There are two workable designs and one that is not. 1. **Pin one convention for the whole format.** Every reader and writer converts between its own memory layout and the wire convention at the boundary. This is the standard answer, and it is why so many wire formats simply declare big-endian: the convention needs to be stated once, not negotiated. 2. **Carry the order in the data.** The stream opens with a known constant - a magic word or a byte-order mark whose correct value is fixed - and the reader inspects it to learn the convention, then branches. This buys writers the ability to emit their native layout without conversion, and it costs every reader a branch on every multi-byte field plus a code path that is rarely exercised and therefore rarely tested. 3. **Leave it unstated.** Not a design. It works exactly as long as both sides happen to share a processor family, and breaks the first time one does not. The practical rule when defining a layout is to state byte order once, in the same breath as the integer widths, and apply it uniformly. Mixing conventions between fields of one record - which happens when a layout grows by accretion and someone copies a field definition from elsewhere - makes the record impossible to read by hand from a hex dump, which is precisely the capability the whole primitive layer exists to preserve.