How does an on-disk columnar file format differ in purpose from an in-memory columnar layout that engines share?

## Same grouping, different optimisation target Columnar is a grouping strategy, not a single design. Two quite different artefacts adopt it for different reasons, and confusing them is one of the more common mistakes in a data interview. An **on-disk analytical format** — Parquet and ORC are the standard examples — exists to make a dataset small at rest and cheap to fetch selectively. Every design decision follows from that: encode each column against its own domain, compress the encoded blocks, cut the data into chunks, and write a footer that lets a reader fetch only some chunks and only some columns. An **in-memory columnar layout** — Arrow is the standard example — exists to be the representation that computation runs over and that separate engines and processes can agree on without converting. Its design decisions follow from that instead: keep each column as a contiguous buffer of fixed-width slots, keep nulls in a separate validity bitmap so the slot width never varies, and leave the values uncompressed so the `i`-th element is a pointer arithmetic step rather than a decode. ## Where the designs diverge | Concern | On-disk analytical format | In-memory interchange layout | |---|---|---| | Optimised for | Bytes at rest and bytes fetched | Direct access and cross-engine sharing | | Value encoding | Dictionary, run-length, delta, then a codec | Fixed-width slots, typically uncompressed | | Reaching element `i` | Decode the block that contains it | Offset arithmetic into a buffer | | Nulls | Recorded per block, often a count plus flags | A separate validity bitmap, one bit per slot | | Metadata | Footer with schema, offsets and per-chunk ranges | Schema plus buffer layout description | | Lifetime | Durable, versioned, read for years | The lifetime of a process or a transfer | | Size | Much smaller than the values it holds | Comparable to or larger than the raw values | The row of that table that carries the most weight is the third one. Encoding is exactly what makes a file small, and it is exactly what makes positional access impossible: in a run-length encoded block there is no fixed offset at which the millionth value lives, and a dictionary-encoded block holds codes rather than values. A compute layer that wants to sum a column as a dense array therefore cannot work on the file's bytes directly, no matter how they are cached. ## The interchange half of the story The second purpose is easy to miss because it is organisational rather than mechanical. Historically, every engine held columns in its own private shape, so moving a result from one to another meant encoding it into some format and decoding it back — a cost paid for no computational reason, twice, on data that never left the machine. A **standard** in-memory layout removes that: if two components agree on the buffer shape, handing data across is handing over the buffers. That is why such a layout is specified publicly at all rather than being one engine's internal detail, and why it defines things a private representation would never bother to define, such as exact bitmap semantics and buffer alignment. Reading values in place out of a shared buffer is its own subject, and this leaf stops at the goal rather than the technique. ## Practical consequences to get right - **Neither replaces the other.** A pipeline reads the on-disk format, decodes into the in-memory layout, computes, and writes the on-disk format back out. Each is doing a job the other is bad at. - **The interchange layout is not an archive.** It is designed for a version of the data that is alive right now. Storing years of history in an uncompressed fixed-width form pays enormous storage costs for no read benefit, and durability guarantees are not what it was designed around. - **The file format is not a compute format.** Trying to run kernels straight over encoded blocks means reimplementing decode inside every operation; the exception is a filter evaluated against dictionary codes, which works precisely because it needs no positional access. - **Compression can appear on both sides, differently.** Ecosystems differ here: some transports compress interchange buffers on the way across a boundary and decompress them on arrival, which keeps the compute layout uncompressed while shrinking the transfer. That is a transport choice and does not change the layout compute runs on. - **Sizes are not comparable.** A dataset that is a few hundred gigabytes as encoded files can be several times larger once expanded into fixed-width buffers, which is why memory limits are computed against the expanded size. The short version an interviewer wants: columnar on disk is an answer to "how few bytes must I fetch and store?"; columnar in memory is an answer to "what shape should the values be in while they are being computed on, and can everyone agree on it?".