In Iceberg v2, how do equality delete files differ from position delete files at read time?

## Two ways to say a row is gone Iceberg format version 2 introduced delete files so that an update or delete need not rewrite whole data files. There are two kinds, and they encode the removal at different levels of precision. **Position deletes** record a pair per removed row: the path of the data file and the ordinal position of the row inside it. Reading them is nearly free. Position delete files carry lower and upper bounds on `file_path` in the manifest, so a scan of one data file loads only delete files whose range covers that path, then drops rows by offset as it streams. The cost to the writer is that it had to know *where* each row lived — meaning it had to locate the rows first, typically with a join against the table. **Equality deletes** record values instead: one or more columns (identified by equality field IDs) and the values whose rows are deleted, for example `id = 42`. The writer never has to find the rows, which is the whole point — a change-data-capture stream can emit a delete for a primary key it has never read. The reader pays instead. It cannot know which files might contain matching rows, so it must apply the equality predicate against every data file in the partition eligible for that delete, an anti-join executed on every scan. ## Which deletes apply to which data files Iceberg resolves applicability by sequence number within a partition. Each commit gets a sequence number; a delete file applies only to data files committed earlier. Precisely, a position delete applies to data files whose sequence number is less than or equal to its own, and an equality delete to data files with a strictly smaller sequence number. This is what lets newly written rows survive an older delete of the same key — an insert after a delete is not silently removed. ## Who writes which Spark's merge-on-read `DELETE`, `UPDATE` and `MERGE` write position deletes, because Spark already resolves the matching rows as part of planning the statement. Flink's upsert/CDC sink writes equality deletes, because a streaming job applying a change log cannot afford a lookup per record. So the delete types you see are a fingerprint of the writers on your table, and an interviewer asking this is usually probing whether you have operated a streaming ingest path. ## Why equality deletes hurt On a table absorbing continuous CDC, equality delete files accumulate per partition and every one of them widens the anti-join each reader must perform. Query latency degrades non-linearly with the number of un-compacted equality deletes, and unlike position deletes there is no cheap path-based pruning to fall back on. The remedy is compaction: `rewrite_data_files` reads data files together with the delete files that apply and writes clean replacements, dropping the deletes entirely. Its `delete-file-threshold` option makes files carrying many deletes eligible even when their size is already fine. For position deletes specifically there is also `rewrite_position_delete_files`, which compacts the delete files without rewriting the data — a cheaper intermediate step with no equality-delete equivalent. ## What changed in format v3 Format version 3 replaces position delete files with **deletion vectors**: a compact bitmap of deleted row positions for one data file, stored in a Puffin file and referenced from the manifest. A reader gets at most one vector per data file instead of a scattered set of small position delete files, so applying deletes becomes a bitmap lookup rather than a merge over many files. Equality deletes remain the streaming-writer mechanism. Note that this is Iceberg's own mechanism; other table formats implement the same idea differently, so keep the vocabulary attached to the format you mean. ## Diagnosing what you have The `files` metadata table exposes a content column distinguishing data files, position deletes and equality deletes: ```sql SELECT content, count(*) AS files FROM prod.db.events.files GROUP BY content; ``` A rising equality-delete count is a signal that the streaming writer is outrunning compaction, and the fix is a compaction cadence matched to the ingest rate — not a query-side tuning knob.