Data serialization

A text encoding writes values as readable characters you can grep, diff and hand-edit; a binary encoding packs them as raw sized bytes that are typically smaller and cheaper to parse, but unreadable without a decoder.

An in-memory value is laid out for one running process: it holds machine addresses valid only there, plus alignment holes whose contents are undefined. Another process shares none of that context, so the value must be re-expressed in a flat, self-standing form.

Because a multi-byte integer can be laid out most-significant byte first or least-significant byte first, and the two readings disagree: the bytes 00 00 00 01 mean 1 one way and 16,777,216 the other. Only the format can settle which.

A base-128 varint carries seven payload bits per byte and spends the eighth as a continuation flag: set means another byte belongs to this number, clear means this is the last one. The reader stops at the first byte whose top bit is clear.

It keeps an identity-keyed map of nodes already written. A node's first visit assigns an identifier, recorded before recursing, and writes the body once; every later encounter writes only a reference, so no body is written twice.

Ubiquity and human readability win. A parser is already on every platform, so a partner integrates with no shared artifact; a person can read, paste and hand-edit a document; and the text diffs in review. Verbosity is the accepted price.

The data model survives; only the syntax is replaced. Maps, arrays, strings, numbers, booleans and null become tagged bytes instead of punctuation and decimal text, so records shrink and decode cheaper — but every key name still travels on every record.

Six workload properties carry most of the decision: payload shape and volume, how fast the contract will change, who the consumers are and what tooling they run, whether humans read the bytes, decoder safety posture, and migration cost.

Values inside one column share a type and a value domain, so runs repeat, distinct values are few and neighbouring numbers sit close together — exactly the redundancy run-length, dictionary and delta encoding exploit. Row-major order interleaves unrelated fields and destroys it.

A column-oriented layout stores each column's values contiguously, so a scan reads only the three columns the query names instead of every byte of every row; a row-major file interleaves all 200 fields, so the whole record comes along.

Backward compatibility means a reader on the new schema can decode data written under the old one. Forward compatibility is the mirror image: a reader still on the old schema can decode data written under the new one.

The registry compares the submitted schema document against the stored versions of one named lineage, in the direction that lineage's compatibility mode requires. It is a document-to-document check made at registration time, not an inspection of deployed readers or code.

Adding an optional field only asks an old reader to skip bytes it never needed, and its default gives new readers a value when old writers omit it. Making a field required invalidates every message already written without it.

Identity decides. Where a reader matches fields by number, a rename changes only the label. Where it matches by name, a rename is a delete plus an add, so readers on the old schema quietly stop finding the value.

Three placements are common: a version field inside the payload, a negotiated media type, and a version segment in the address. Each moves the cost elsewhere - routing and caching, client effort, or resource identity - and none removes it.

Absent means the sender said nothing about the field. Null means the sender said it has no value. Empty means it has a value that happens to be zero-length. Three different statements, and many encodings collapse them.

Most encodings give the encoder freedom — member order, whitespace, number spelling, escapes, omitting defaults — so one value has many valid byte sequences. A canonical encoding fixes every one of those choices so the digest depends only on the value.

They are usually dropped. Decoding keeps only the fields the reader's own schema version names, and the output is rebuilt from that smaller in-memory value, so the newer producer's data disappears with no error anywhere. Preserving it requires a deliberate side buffer.

A delimiter means 'the message ends here', so any occurrence of that byte inside the payload would end the frame early. The writer must escape it, or restrict the payload to an alphabet that excludes it.

A stream connection carries bytes, not messages, so the reader keeps a buffer across reads: append each chunk, test whether a complete message is present from its length prefix or delimiter, consume exactly that message, and retain the remainder.

Most of a short JSON record is structure, not data: field names repeated on every record, quotes and punctuation, and numbers spelled out as digit characters. A schema-driven binary encoding sends small field tags and raw integer bytes instead.

The writer already knows which fields exist and appends them into one buffer. The reader starts from opaque bytes and must discover boundaries and types, validate what it cannot trust, convert, and allocate an object per value. Discovery, validation and allocation have no write-side counterpart.

Streaming wins when the document is large relative to memory, or unbounded, and the reader consumes it in one forward pass — peak memory then tracks the largest single value, not the payload. It loses when the work needs random access or back-references.

In-place access means the received bytes are the data structure: each field is located by offset arithmetic and read on demand, so no decode pass builds a parallel tree of objects. The buffer must stay alive and unchanged while those values are used.

Because the compressor has already removed most of what the compact encoding removes. Repeated field names collapse into short back-references, so the two levers overlap instead of stacking, and the dense binary payload has far less redundancy left to squeeze.

The grammar allows a repeated member name but the data model holds one value per name, so each decoder invents its own resolution: last wins, first wins, reject, or keep both. Two decoders can pick differently.

A byte is cheap and a stack frame is not: a megabyte of opening brackets is about a million nesting levels, and a decoder that recurses per level exhausts its stack long before it exhausts that input. Depth needs its own ceiling.

By the time a parse finishes, the memory, CPU and stack the sender asked for have already been spent, so a later check only reports the damage. A ceiling has to be a counter the parser itself tests as it reads, aborting mid-stream.

Validation applies to one decoder's interpretation of the bytes, and that interpretation is thrown away when the raw bytes are forwarded. The next decoder derives its own meaning, so the payload that is used was never the payload that was checked.

A few bytes declaring four gigabytes make the server reserve four gigabytes, and the sender never has to produce them. A declared length is a claim about the stream, not authorisation to reserve memory on the sender's behalf.