How does an open lakehouse table format handle two writers committing at the same time?

## Optimistic, not pessimistic Acquiring a lock for the duration of a lakehouse write would be disastrous: writes commonly run for minutes to hours, and locking a table that long serializes the whole platform. So table formats use **optimistic concurrency control**: assume no one else is writing, do all the work unsynchronised, and detect the collision only at the moment of commit. The reason this is safe is that all the work really is unsynchronised — data files are written where nothing references them, so two writers producing files simultaneously cannot interfere at all. ## The race ```text writer A: read base = v41 ──▶ write files ──▶ CAS v41 -> v42 SUCCEEDS writer B: read base = v41 ──▶ write files ──▶ CAS v41 -> v42 FAILS (current is v42) └▶ rebase on v42, re-check, retry ``` Both writers read version 41 and plan their work against it. Both attempt a compare-and-swap conditioned on 41 still being current. The swap is a single atomic conditional operation, so exactly one succeeds. This is the entire mutual-exclusion story — there is no lock, only a losing conditional write. ## What the loser does next Failing the swap is not an error the user should ever see; it is the start of a retry loop with a genuine correctness check in the middle. The loser re-reads the current snapshot and compares the winner's changes with its own operation's assumptions: - **Compatible (rebase and retry).** Two appends touch nothing in common: the loser's data files are still perfectly valid, so it simply rebuilds its commit on top of version 42 and swaps again. Nothing is recomputed and no data is re-read. Most real concurrency is this case. - **Conflicting (recompute or abort).** The winner removed or rewrote files the loser also intends to remove or rewrite — two concurrent compactions over the same files, two updates hitting the same rows, an overwrite of the same partition. Or the loser's operation had a **read set**: a `MERGE` or a conditional `UPDATE ... WHERE` decided which rows to change by reading data the winner has since changed. Re-committing the loser's precomputed result would silently apply it to a state it never saw, producing a lost update. The loser must redo the operation against version 42, or fail loudly. The important nuance for an interview is that conflict detection is about **overlap of what each commit read and wrote**, not about wall-clock timing. Two writers to different partitions of the same table usually both succeed, one after a harmless rebase. ## What readers see meanwhile Nothing surprising: a reader resolves the current pointer once and then reads an immutable file set. A commit that lands mid-query does not change that query's results, and a query started after the commit sees all of it. That is snapshot isolation, and it falls out of the same design. ## Where it goes wrong in production - **Starvation.** A writer whose work takes twenty minutes competing with a job that commits every thirty seconds may lose every race until it exhausts its retries. Retry counts do not fix this; reducing commit frequency, batching the fast writer, or splitting the slow one does. - **Retry storms on conflicting work.** If two pipelines genuinely update the same rows, retries burn compute and still conflict. Route both through one writer, or partition ownership so each partition has exactly one writer. - **Compaction fighting ingestion.** Maintenance that rewrites files an ingest job is also touching is a classic recurring conflict; schedule it against quiet partitions or accept and re-run it. - **False confidence in "idempotent retry".** A retry that blindly re-swaps a precomputed result without re-checking the read set is how a lost update happens. If you implement anything like this yourself, the check is the point, not the loop. ## The design rules that follow One writer per table (or per partition) is the simplest correct topology and the one most platforms converge on. Commit in reasonably sized batches rather than continuously, so the pointer is not being swapped constantly. Keep long-running rewrites narrow so their conflict surface is small. And treat repeated commit failures as a contention signal to be designed away, not an error to be retried harder.