Backup, Restore & Migration

## The defect: a copy that spans instants Backing up a Docker named volume means reading its files. The standard idiom mounts the volume into a throwaway container next to a bind mount of a host directory and runs `tar` between them. That is a perfectly good way to move bytes, and it is exactly the wrong thing to point at a database that is running. `tar` walks the tree. On a 412 MB volume it might take four minutes. The first file is read at one instant and the last at another, and between those two instants the database engine has been writing: appending to its log, checkpointing pages, renaming files, deleting others. The archive is a mosaic of different moments — often called a *torn* or *fuzzy* copy. Nothing in it is guaranteed to satisfy the engine's own on-disk invariants, because those invariants span files. A concrete flavour of the problem: an engine that keeps a write-ahead log alongside its main data file — SQLite in WAL mode keeps a `-wal` sidecar, and most server engines have an equivalent — can easily have the data file captured before a checkpoint and the log captured after it, producing a pair that never coexisted. Two clarifications matter because candidates often conflate them: * **Crash consistency** is what you get from an *atomic* copy taken at a single instant — a filesystem or block-device snapshot. It looks to the engine exactly like a power cut: recoverable, because every serious engine is built to survive that. * **A sequential file-by-file copy of a live tree gives you neither** a clean shutdown nor crash consistency. It is a state the machine was never actually in, so no recovery procedure is promised to fix it. `docker cp` of a live data directory has precisely the same defect. It is a different command, not a different guarantee; it too reads the tree file by file while the writer runs. The reason "just `docker cp` the data dir" fails a backup review is consistency, not convenience. ## Quiescing: the three honest options **1. Stop the writer.** `docker stop <container>` sends SIGTERM, the engine performs its shutdown sequence — flush buffers, checkpoint, close cleanly — and after the container exits the volume is static. Now tar it from a helper container; nothing is changing, so the archive is a faithful copy of a clean shutdown state, the easiest thing in the world to restore. The cost is downtime: for the email-digest builder that is the stop plus the copy plus a six-second cold start on the way back. Give the engine a stop timeout long enough to finish (`docker stop -t 60`), or SIGKILL after ten seconds turns your clean shutdown back into a crash. **2. Use the engine's own online backup.** Databases ship tools that produce a consistent copy while serving traffic, because they can coordinate with the internals — take a checkpoint, note a log position, stream a copy, note the end position. Run that tool and back up *its output* rather than the raw data directory. This is the standard answer for anything that cannot take downtime, and it moves the correctness burden onto the engine's authors, where it belongs. **3. Snapshot atomically at the storage layer.** A filesystem or cloud-disk snapshot captures the whole volume at one instant, giving crash consistency; the engine then recovers on first start, and you copy the snapshot at leisure without holding the service still. Note what this requires: the volume's data must live on a storage layer that can do that. Docker's own tooling cannot — there is no `docker volume snapshot` — so this is a property of the host filesystem, the block device, or a network-backed volume, not of Docker. ## What `docker pause` does and does not buy `docker pause` freezes the container's processes with the kernel's cgroup freezer. Files do stop changing, so the tar is no longer torn — but the on-disk state is still whatever the engine happened to have written when it was frozen, with any in-memory state unflushed. That is crash consistency at best, never a clean shutdown, and the pause lasts the entire copy, so it is an outage that also produces a weaker artifact than simply stopping the container. It is not a shortcut. ## Restore is part of the design Two properties separate a backup from a directory of tarballs. * **Restore into a fresh, empty volume.** Extracting over a volume that already holds data leaves a mix of old and new files — an engine can fail in inventive ways when its data directory contains files from two eras. Create a new volume, restore into it, point a container at it. * **Prove it.** Restore into that fresh volume on a scratch host, start the service against it, and check that it comes up without recovery errors and that the data is the size and shape you expect. Do it on a schedule, not during the incident. An unverified backup is an untested code path, and the failures described above are all silent — the tar exits 0 every time. ## Choosing under pressure If downtime is cheap, stop the container: it is the least machinery and the strongest guarantee. If it is not, use the engine's online backup tool. Reach for storage snapshots when the volume is large enough that copying it is the bottleneck, and accept that you are then relying on the storage layer's atomicity claim. Reserve the plain live tar for volumes holding data that is not a database at all — static assets, generated reports, an append-only spool — where a torn copy has no invariants to break.