How does LanceDB versioning work, and how do you read an older table version?

## Why versions exist at all The Lance format never edits data files in place. An append writes new fragments; an update rewrites the affected fragments; a delete writes a deletion file marking rows as removed. Each of those operations finishes by committing a new manifest — a small metadata file naming the schema, the fragments and the deletion files that make up the dataset at that instant. The manifest is the version. Because manifests are cheap and data files are shared between them, keeping history costs metadata, not a full copy of the table. This is the same idea as a copy-on-write table format in the analytics world, and it is why LanceDB gets versioning "for free": it is not a bolt-on feature, it is how writes commit. ## The API - `table.version` — the version the table object is currently reading. - `table.list_versions()` — the version history, each entry carrying a version number and a timestamp, which is how you translate "what did this look like on Tuesday" into a number. - `table.checkout(n)` — pin this table object to version n. Reads, filters and vector searches now run against that snapshot. - `table.checkout_latest()` — jump back to the newest committed version. - `table.restore()` — take the currently checked-out version and commit it as a new latest version. A table pinned to an old version is read-only; you cannot append to the past. `restore()` is the escape hatch: it makes the old state current *by moving forward*, so the history that recorded the bad write is still there. Rolling back never destroys evidence. ## What actually creates a version Everything that commits: `add()`, `delete()`, `update()`, `merge_insert(...).execute()`, `create_table(..., mode="overwrite")`, schema changes such as adding a column, and index creation. A read never does. This surprises people who assume delete is a maintenance operation outside history — deleting a million rows produces a new version whose predecessor still references every one of those rows' data files. ## What it buys you **Reproducibility.** A retrieval evaluation run against version 42 can be re-run months later against version 42 and give the same results, even though ingestion has continued. Pinning the version in the experiment's config is the whole mechanism. **Cheap rollback.** A bad embedding backfill is undone by `checkout` to the last good version, sanity-checking the data, then `restore()` — seconds of metadata work rather than restoring a backup. **Debugging.** "When did this row change?" is answerable by checking out successive versions and comparing, instead of guessing from application logs. **Consistent readers.** Because a table object reads one version, a long analytical pass is not disturbed by concurrent appends; it sees a stable snapshot until it explicitly refreshes. ## What it costs Storage, and unbounded storage if ignored. Old versions hold references to fragments and deletion files, so nothing is reclaimed while a version referencing it survives. A table under frequent small writes accumulates both versions and small fragments. `table.optimize()` is the maintenance path: it compacts data files and can prune versions older than an age you pass. Pruning is destructive to history — after it, `checkout` on a pruned version fails — so retention is a real decision: long enough to cover your rollback and reproducibility needs, short enough that the bucket does not grow forever. ## Multi-process behaviour Since each table object reads a specific version, a reader opened before a writer's commit keeps returning the older data until it refreshes. That is not staleness caused by a bug; it is the snapshot semantics working. `checkout_latest()` refreshes on demand, and `read_consistency_interval` on `connect` makes the refresh automatic on a cadence. ## How to answer Lead with the mechanism — immutable manifests committed per write — then the three calls (`list_versions`, `checkout`, `restore`), then the tradeoff: history is nearly free to keep and expensive to keep forever, so `optimize` and a retention policy are part of running a LanceDB table, not an afterthought.