Your ingestion pipeline's per-byte bill is the top cost line; how do you choose which payload-size lever to spend a quarter on?

## Start with an accounting, not a lever The failure mode in this decision is picking the lever first, usually the most technically interesting one, and measuring afterwards. The accounting comes first, and it has three columns: - **Where the bytes are.** Compressed bytes per record, attributed per field, over a real sample from several producers. The attribution is what makes the conversation concrete: "this one debug field is a fifth of the bill" ends an argument that a ratio never would. - **How many records.** Per producer, per day, with the growth trend. A saving is a rate, not a number. - **Which bill.** Bytes **moved** and bytes **stored** are charged separately, over different retentions, and a lever does not necessarily touch both. Storage may also be re-laid-out downstream in a form whose own encoding choices are a separate subject; the wire decision does not automatically follow it. ## Rank levers by saving against disruption | lever | typical saving | what it costs | |---|---|---| | prune fields nobody reads | large, on both bills | proving they are unread; a contract change | | batch and place compression well | large on short messages | latency, wider failure unit, memory | | shorten or renumber fields | modest, mostly reclaimed by compression | a contract change for every consumer | | migrate to a compact binary encoding | large raw, much smaller compressed | every producer and consumer; readability; tooling | | sample or aggregate at source | the largest of all | information you cannot get back | The ordering that falls out of this table is usually: 1. **Prune.** An unread field costs its full bytes on every record, on both bills, forever. Removing it is confined to the producer plus a contract update — the work is in *proving* nobody reads it, which means usage sampling over a window long enough to catch monthly jobs, then a deprecation window. Whether the removal itself is a compatible change for existing readers is governed by the schema-evolution rules, which are their own subject. 2. **Batch and place compression once, well.** Producer-side, reversible, no contract change, and on short messages it is often the largest remaining win. Its price is delay and a wider unit of loss, both of which are tunable. 3. **Reshape expensive fields.** A repeated string turned into an enumerated code, a wide value expressed relative to a batch-level base, a precision that nobody uses trimmed. These are contract changes but narrow ones. 4. **Migrate the encoding.** Last, and only with a compressed measurement in hand. ## Why the migration usually sits last It has the best slide and the worst ratio of saving to disruption: - On a link that already compresses, most of the raw gap has been taken by the compressor, so the number that reaches the invoice is a fraction of the headline. - It lands on **every** producer and consumer at once, which is a quarter of several teams' roadmaps rather than one team's. - It removes the ability to read a payload with your eyes during an incident, and it breaks every ad-hoc script that parsed the old shape. That tooling has to be replaced *before* the migration lands, not after. - It runs through a window in which both encodings are in flight, which is its own operational load. None of that makes it wrong. It makes it a decision that needs a number attached and a second justification — schema enforcement, decode cost, contract discipline — beyond the bytes. ## Two traps in the arithmetic - **Do not add or multiply the levers' headline savings.** Several of them are removing the same bytes: pruning removes a field the compressor was already collapsing, a compact encoding removes names the compressor had already reduced to back-references. Model the combination, or measure it, but do not stack percentages. - **Do not optimise the moved bill and forget the stored one**, or the reverse. A change that halves what crosses the network and leaves the archived copy untouched has solved the smaller half of the problem if retention is long. ## Hold the saving A byte budget regresses one convenient new field at a time. Whatever is won should be defended the same way it was found: **bytes per record, tracked per producer, alerted on when it drifts**, and visible next to the invoice. Add a review step where new fields on a high-volume record are argued for rather than merged quietly. Without that, the quarter's saving is gone in two quarters and nobody can say which change spent it.