skip to content

In an immutable-file columnar table, what is a delete vector and when does it beat rewriting the file?

level: seniorimportance: nice to knowfreq 35%

answer

  1. deletes are expensive because files are sealed
  2. record what to hide instead of rewriting
  3. a bitmap over row positions in one file
  4. cheap write, taxed read, deferred settlement
  5. compaction eventually materialises the deletes

basics

~20 s

A delete vector is a small side file marking which row positions in a data file are logically deleted. Readers filter them out on the fly. It turns an expensive file rewrite into a tiny write, at the cost of extra work on every read until compaction.

solid answer

~50 s

Because data files are immutable, a delete normally means **copy-on-write**: read the file, write a replacement without the removed rows, swap the metadata. That costs the whole file for even one row. A delete vector — the **merge-on-read** approach — instead writes a compact marker, typically a bitmap of deleted row positions within a specific data file, and leaves the data file alone. Readers apply the mask as they scan, dropping masked rows before they reach the rest of the plan. It wins when deletes are frequent, small, and scattered across many files: writes stay cheap and commits stay fast, which matters for a streaming or CDC-style workload. It loses on read-heavy tables, because every scan now pays an extra file read and a per-row mask, and because accumulating markers slow reads more over time. The reconciliation is deferred, not avoided: a later compaction materialises the deletes into fresh data files and drops the vectors.

code

text · 4 lines
text
data_file: part-00042.parquet   rows=1,000,000
delete_vector: dv-00042.bin    deleted_positions={17, 4021, 998_233}

scan: read part-00042 -> apply mask -> emit 999,997 rows

go deeper

for a junior

Know the basic idea: instead of rewriting a whole file to delete a few rows, the engine can record which rows to hide and filter them out when reading.

for a middle

Explain the mechanism — a positional mask per data file, applied during the scan — and state the trade plainly: cheap writes, taxed reads, settled later by compaction.

for a senior

Choose between merge-on-read and copy-on-write from the workload's read/write ratio and deletion pattern, and describe the monitoring and compaction schedule that keeps the deferred cost bounded.

for a principal

Own the policy across the platform: which tables are mutable, what compaction cadence the deferred work demands, what it costs in compute, and how physical-erasure obligations constrain the choice.

## The problem it solves On immutable storage the default way to remove rows is to rewrite the file that holds them. The cost is proportional to files touched, not rows removed, so deleting a hundred scattered rows from a hundred large files rewrites all of them. For a batch pipeline that deletes a whole partition at a time this is fine. For a workload that continuously retracts or supersedes individual rows — change-data-capture, GDPR erasure requests, correction streams — it is ruinous. ## What a delete vector is A delete vector is a small auxiliary structure associated with one data file that identifies which **row positions** inside it are logically deleted. Physically it is usually a compressed bitmap: one bit per row position, set when the row is gone. It is written as its own small file and referenced by the table's metadata alongside the data file it masks. The read path changes accordingly. When a scan opens a data file, it also loads any delete vectors that apply, and rows whose positions are marked are dropped before any further processing. Nothing about the data file changes — its bytes, its statistics and its encodings are untouched — so readers pinned to an older version of the table simply do not load the newer vector and see the rows as still present. That is how snapshot isolation survives. ## Merge-on-read versus copy-on-write The two strategies sit at opposite ends of one trade. **Copy-on-write** pays everything at write time: read the file, rewrite it without the deleted rows, commit. Write amplification is high and commit latency scales with data size, but the resulting table is clean — reads pay nothing extra and files stay well-formed. **Merge-on-read** pays almost nothing at write time and taxes reads instead: an extra small file to fetch per masked data file, and a mask applied per row scanned. It also leaves dead bytes on disk — the deleted rows are still stored and still read off storage before being discarded — so the table's physical size does not fall until compaction. The choice is a straightforward function of the read-to-write ratio and the deletion pattern: - Frequent, small, scattered mutations, or tight write-latency requirements: merge-on-read. - Read-dominated tables with occasional bulk deletes: copy-on-write. - Anything in between: merge-on-read for freshness plus a compaction schedule aggressive enough to keep the accumulated masks small. ## The compaction obligation Delete vectors defer work; they do not remove it. Every accumulated vector is a permanent tax on every subsequent scan, and the wasted bytes are permanent storage cost. A merge-on-read table therefore **requires** a compaction process that periodically rewrites data files with the deletions applied and discards the vectors. Skipping it is the classic operational failure: reads degrade steadily, storage grows despite deletions, and nobody connects the two. Compliance deletion adds a hard edge to this. A delete vector hides a row from queries but the bytes are still on storage. If the requirement is that data be physically erased within a deadline, only the rewrite satisfies it — so the compaction schedule becomes a compliance control, not just a performance one. ## Updates on top of the same machinery An update is usually expressed as a delete of the old row plus an append of the new one: mark the old position in the vector, write the new version in a fresh file. Reads then see exactly one version. This is why the same mechanism that supports cheap deletes also supports cheap upserts, and why merge-on-read tables tend to grow more, smaller files — every mutation adds one. ## Signals to watch The number and size of delete vectors relative to data files, the fraction of scanned rows discarded by masks, and the gap between logical and physical table size all tell you whether the deferred work has piled up. When scans read materially more rows than they emit and storage exceeds logical size by a wide margin, compaction is behind. ## Boundary to keep straight Delete markers, tombstones and position-based masks appear under different names across engines and table formats, with different metadata plumbing behind them. The mechanism to explain in an interview is the general one: defer the rewrite by recording what to hide, pay for it on read, and settle up during compaction.

  • Why does a table using delete markers keep growing on disk even though rows are being deleted?
    The marker only hides rows; the bytes remain inside the untouched data file, and each marker adds its own small file. Physical size falls only when compaction rewrites the data files with the deletions applied. If compaction lags, logical size drops while physical size climbs.
  • Does a delete vector let you skip reading the deleted rows from storage?
    Generally no. The mask is positional within the file, so the reader still fetches the column blocks containing those positions and discards the rows after decoding. You save the write, not the read. That is precisely why accumulated vectors degrade scan efficiency over time.
  • When is copy-on-write clearly the better choice?
    On read-dominated tables where mutations are rare or arrive as bulk operations aligned with the physical layout — dropping a partition, correcting a day of data. You pay one rewrite and every subsequent read is clean and full-speed, with no masks to load and no dead bytes to scan.
  • How does a compliance requirement to physically erase data change the design?
    Hiding rows behind a mask does not satisfy physical erasure, since the bytes are still on storage. The compaction that materialises deletions becomes a compliance control with a deadline, and the retention window on superseded files must also be short enough that the old copies are actually garbage-collected in time.

It is the difference between reprinting a page to remove a paragraph and clipping a stencil over it: the stencil is instant, but every future reader must hold it up, and the paragraph is still on the page until someone reprints.

saying these in an interview costs you the question

  • Believes a delete vector frees storage immediately
  • Assumes marked rows are never read from storage
  • Treats merge-on-read as strictly better than rewriting
  • Forgets that compaction must eventually materialise deletes
  • Thinks hiding rows satisfies a physical-erasure requirement

context

resolving links…