Format Families and Trade-offs

The data model survives; only the syntax is replaced. Maps, arrays, strings, numbers, booleans and null become tagged bytes instead of punctuation and decimal text, so records shrink and decode cheaper — but every key name still travels on every record.

Six workload properties carry most of the decision: payload shape and volume, how fast the contract will change, who the consumers are and what tooling they run, whether humans read the bytes, decoder safety posture, and migration cost.

Values inside one column share a type and a value domain, so runs repeat, distinct values are few and neighbouring numbers sit close together — exactly the redundancy run-length, dictionary and delta encoding exploit. Row-major order interleaves unrelated fields and destroys it.

A column-oriented layout stores each column's values contiguously, so a scan reads only the three columns the query names instead of every byte of every row; a row-major file interleaves all 200 fields, so the whole record comes along.

A native snapshot carries the whole reachable object graph plus the type identity needed to rebuild it: private fields, derived caches and shared references included. A field-mapped encoding carries only the fields you chose to declare.

A small numbered tag plus the value's bytes ride instead of the name. Both sides compiled the same schema, so the name is already known. That buys much smaller messages, cheaper decoding, and a field name that is free to change.

Tag-numbered identity needs only the reader's own compiled schema, because the numbers in the bytes are the field identities. Writer-schema resolution additionally needs the exact schema the writer used, reachable by an identifier that travels with the data.

XML models documents: an element can carry attributes alongside child elements, children are ordered, and text and markup can interleave as mixed content. JSON models data — objects, arrays and scalars — so interleaved narrative must be encoded by convention.

The snapshot names types by their old identity, so the new code cannot resolve them and the rebuild fails outright. The snapshot's contract is the code's shape, so an ordinary refactor is an unannounced breaking format change.

Each value announces its own type and size, so a decoder can step over one it does not want without interpreting the contents — a bounded structural walk, where a text parser must scan character by character tracking quotes and escapes.

Both group values by column, but for opposite goals. An on-disk format such as Parquet or ORC minimises bytes at rest with encoding, compression and chunk statistics. An in-memory layout such as Arrow keeps fixed-width buffers that compute can address directly.

Rebuilding fills fields directly instead of going through the type's ordinary construction path, so constructor checks and derived-value computation do not re-run. Restoring invariants falls to the type's post-read hook, if it declares one.

YAML optimises for a human writing by hand: comments, little punctuation, multi-line text, reusable anchors. Those benefits have no consumer on a machine-to-machine hop, while indentation-carrying-meaning and implicit typing of unquoted values become live hazards.

Mostly it is a clean swap, but the binary model is richer, so numbers can arrive as an integer or a float rather than one numeric text form, raw bytes stop being Base64 text, non-string map keys and non-finite floats become expressible, and consumers switching on type see cases the text form could not produce.

Those three hops face different constraints: an internal hop optimises bytes and generated code between services that deploy together, a public hop optimises for consumers nobody controls, and an archive optimises for a reader who arrives years later with different tools.

The reader evaluates the query's filter against each chunk's recorded value range first. A chunk whose range cannot intersect the filter is skipped without being fetched at all; only chunks whose range overlaps are read, and their rows are then re-checked individually.

Every place bytes are read needs the matching schema: incident tooling, logs, dead-letter queues and archives. That means decoded projections in logs, a decoder in the on-call toolkit, and schemas retained at least as long as the data they explain.

The contract lives outside the bytes. Nothing on the decode path consults it, so a document that violates the schema still parses; validation becomes a step each boundary chooses to run, and schema and payload can drift apart unopposed.

Accept it when records must stay interpretable on their own and no schema can be rolled out to every producer and consumer in step. Pay the per-record key overhead in exchange for zero coordination; abandon it when volume makes the overhead dominate a coordinated pipeline.

Price both sides in one currency. Turn the payload fit into a measured cost per unit of traffic and the tooling gap into engineering time and support load, decide whose budget pays, then record the choice with the alternatives and what defeated each.

Classify the access paths by selectivity and columns touched, then match each to the layout whose cost model fits. A scan-optimised columnar store and a record-optimised serving store are usually both justified — and the decision you actually own is the synchronisation path between them.

It guarantees identical bytes and identical field identity: any generated reader decodes what any generated writer wrote. It does not guarantee identical meaning — units, invariants, required-ness, error behaviour and ownership of the definition are agreements no schema expresses.

The JSON grammar defines no comment production, so any comment text is a parse error. XML and YAML both have one. Teams work around it with a convention key, a sidecar file, or a superset dialect stripped before parsing.

Through an extension mechanism: a numbered tag in front of an opaque payload. The tag tells a decoder how to interpret the bytes, a decoder that knows the number produces a typed value, and one that does not can keep the pair intact rather than failing.

Evidence tied to a criterion the standard did not anticipate on this hop: a measured cost, a consumer set with no decoder, a different trust boundary, or a fidelity requirement. Preference and familiarity are not evidence.

Every advantage of the layout comes from many rows being written together. One event per file gives column blocks of a single value, no runs for encoders to collapse, per-chunk statistics that rule nothing out, and a footer that can be larger than the data.

On a remote call the schema reaches the reader at build time, compiled into both peers from one interface definition, with nothing sent per message. In a bulk file it reaches the reader inside the file, written once in the header and amortised over every record.

Refuse it as a cross-service format. Version stamps only make a mismatch detectable; they cannot translate one, and every type in the graph becomes a versioned contract nobody owns. Keep the family for short-lived, regenerable, single-build use.