A platform must standardise on one record representation for its shared pipelines - packed bytes or language objects - how do you decide?

## What is actually being decided Every pipeline on the platform holds its records in one of two forms while a **worker** - one process on one machine running some of the job's work, owning a fixed amount of memory nothing else can borrow - is working on them. **Host-runtime objects** are ordinary objects of the worker's language, each carrying the runtime's own per-object bookkeeping. **Engine-managed binary records** are a packed byte layout the engine allocates and interprets itself, fields read at known offsets. The decision is not which is better in the abstract. It is which one every team gets by default when they do not think about it, and that makes it three things at once: a capacity decision, a latency decision, and a constraint on how authors are allowed to write per-record logic. ## The three things the default moves - **Footprint, and therefore machines.** The object form costs a multiple of the packed form for narrow records of many small fields, and close to nothing extra for a record dominated by one large value. Multiply the typical multiple by the platform's whole workload and it is a line on an invoice, not a preference. - **Tail behaviour.** The object form manufactures enormous numbers of short-lived objects. Where the worker's runtime reclaims memory automatically - periodically finding and freeing objects nothing refers to, stopping the worker while it does - that shows up as slow units of work rather than as errors, and it hits the slowest few per cent hardest. - **What authors may write.** A packed default only pays off if most operators can read the bytes. Every step that drops into host-language code decodes each record into objects and encodes the result back, per record. A packed default with unrestricted per-record functions everywhere can be *slower* than an object default, because it pays the conversion without ever banking the saving. ## What constrains the choice before preference does 1. **Engines differ, and some have no packed form at all.** A platform that spans more than one engine of this class cannot standardise on something one of them does not have. A packed-first default is also a coupling decision: it makes moving a workload to an engine without one a rewrite. 2. **The record shape decides the size of the prize.** Narrow records of many small fields gain most. Records dominated by one large value gain almost nothing, so a platform full of those is choosing between a large disruption and a small saving. 3. **Pipeline length decides how often you pay.** A long pipeline of engine operators pays the encode once and reads the bytes many times. A short pipeline that crosses into host-language code immediately pays the conversion and banks little. | Workload profile | Default that usually wins | Why | |---|---|---| | Wide aggregations over narrow many-field records | Packed | The footprint multiple is largest and the operators can work on bytes throughout | | Heavy custom per-record logic in the host language | Objects | Every record would cross the boundary anyway, so the packed form is taxed without being used | | Records dominated by one large value | Either; decide on other grounds | The representations differ little in footprint, so the argument is about author convenience | | Mixed, spanning engines that differ in what they offer | Objects, with packed as a measured exception | A default must be one every engine on the platform can honour | ## Settling it with numbers rather than opinion The honest artefact is a before-and-after figure, not an argument. Take three or four pipelines that represent the platform's real shapes, run each under both representations, and compare the run's reported numbers per unit of work: peak memory, wall time, and the spread between the slowest unit and the median. Add the machine count each version needs to hold the same throughput. That table is the decision. Beware of deciding it on a pipeline whose records are one large value each, because it will show no difference and mislead everyone. ## Owning the blast radius A default is inherited by teams who never evaluated it, so a wrong one does not appear as a failing pipeline. It appears as a platform that needs more machines than it should, or as tail latency nobody can attribute. Two things follow: - **Make it reversible in the small.** Publish where the exception is allowed and what evidence takes it - the same before-and-after figure, on the pipeline in question. A ban invites people to route around the platform; an exception path with a measurement attached keeps the evidence flowing back to you. - **Expect reversal in the large to be a rewrite.** Per-record code is written against whatever the records are, so changing the default across an estate is a broad code change rather than a setting. That asymmetry is a reason to invest in the measurement now rather than to decide quickly. One thing this decision is not: it does not determine how data is laid out at rest, and the size of a record on disk is a separate subject with separate owners. Nor does it settle where a job's work is divided or moved - a step where every worker writes its output split by destination and every worker fetches its share, a redistribution, is dominated by other concerns. Keep the scope to what each record costs while it is being held, and the decision stays tractable.