When a source can drop either CSV or Parquet into a landing bucket, what does Parquet buy you?

## What the format choice actually decides At the landing boundary the format is not an aesthetic preference; it decides how much the loader has to *guess*. Everything below follows from one difference: Parquet ships its schema with the data, and CSV does not. ## CSV: a dialect, not a format "CSV" names a family. Each drop implies decisions the file itself does not record: - delimiter (comma, tab, pipe, semicolon in locales that use comma decimals) - quote character and how a quote inside a quoted field is escaped (doubled, or backslashed) - whether fields may contain literal newlines inside quotes — which breaks any tool that splits on `\n` - character encoding: UTF-8 versus latin-1 versus UTF-16 with a BOM, a genuinely common partner problem - header row present or absent, and whether column order is stable - how null is represented: empty, `NULL`, `\N`, `NA`, `-` - timestamp format and timezone, decimal separator, thousands separator - line terminator, `\n` or `\r\n` Get any of these wrong and you do not get an error. You get shifted columns, truncated strings, or a whole file parsed as one row. On top of dialect sits **type inference**. A CSV field is a string until something decides otherwise. Readers that sniff types are the source of the classic damage: leading zeros stripped from zip codes and account numbers, `1e3` read as a float, a mostly-numeric column that turns into text the day a `N/A` appears, a phone number becoming scientific notation. And inference is *sample-based* in many tools, so the type can change between runs depending on what the first N rows happened to contain — the same file, loaded twice, yielding different types. ## Parquet: the schema travels Parquet files carry column names and types internally. The loader does not infer; it reads what the writer declared. That eliminates the entire inference class of bugs at a stroke, and it changes how you detect change: a column added at the source appears as a named field you can compare against expectation, rather than an extra positional value that silently shifts every column to its right. Being column-oriented and compressed also has direct operational effects. Files are typically much smaller than the equivalent CSV, which cuts storage and egress. A reader that needs three of eighty columns can fetch just those, so scans move far fewer bytes. And there is no dialect to agree on — nothing to misparse. ## What Parquet costs you The producer must be able to write it. Legacy systems, partner SFTP exports and "the finance team's weekly extract" often cannot, and demanding it is sometimes a six-month negotiation for a benefit the partner does not receive. It is opaque to ordinary tools. When a drop looks wrong at 2am, `head` and `grep` on a CSV answer the question in seconds; a Parquet file needs a reader. Keep one handy in the runbook. Small files are worse in Parquet than in CSV, proportionally: every file carries its own schema and footer metadata, so a stream of kilobyte-sized Parquet objects spends a large fraction of its bytes and its read latency on per-file overhead. Parquet rewards batching; if your producer emits micro-drops, fix the batching before the format. Finally, a compressed columnar file is less forgiving of partial corruption than a line-oriented text file where you can salvage the readable lines. ## Where JSON sits Newline-delimited JSON is the common middle ground: self-describing per record, tolerant of nested and irregular shapes, no dialect to negotiate, and still greppable. It costs a lot of bytes — every key repeated on every line — and it has weak typing (numbers, no distinction between integers and floats in many writers, dates as strings). It is a reasonable landing format for genuinely semi-structured sources, and a poor one for wide, regular, high-volume tables. ## The practical policy Take the best format the producer can actually emit, and land it **verbatim**. The landing zone's value is being a faithful archive of what was received; converting on the way in destroys your ability to prove what a partner sent and to reprocess with a fixed parser later. Then add a first normalisation hop that converts the raw drop to Parquet in a separate prefix, applying an explicit schema — not inference — and routing rows that do not fit to a quarantine location. Downstream loads read the normalised copy. This gives you the audit trail of the original and the type safety of the typed format, and it means an upstream dialect change is a parser fix and a replay rather than a corrupted table. When the drop must stay CSV, write the dialect down as part of the delivery agreement — delimiter, quote, escape, encoding, header, null token, timestamp format and timezone — and configure the reader explicitly against it rather than letting it sniff. Explicit beats clever every time on this boundary.