You own a 4 TB relational OLTP database with a two-hour nightly low-traffic window. Design its backup strategy in terms of backup types, and explain the tradeoffs behind your choices.
answer
- 4 TB → physical base, logical is secondary
- Weekly full + daily incrementals, taken from a replica
- Cap chain length / synthetic fulls
- Off-site + immutable copy
- Retention chain-aware; time a real restore
basics
~20 sPhysical backups are the base: a weekly full plus daily block-level incrementals, because a 4 TB logical dump will not fit a two-hour window. Add a periodic logical dump for portability and object-level recovery, keep copies off-site, and size retention around how far back you must be able to go.
solid answer
~60 sAt 4 TB a logical dump is out as the routine path: dumping is slow, restoring is far slower, and the long snapshot would bloat the primary nightly. So: - **Base layer**: physical backups. Weekly full, daily block-level incrementals, taken from a replica so the primary sees no IO cost. Compressed and parallel, streamed to object storage. - **Chain management**: cap chain length with a weekly full or synthetic full, since restore cost and fragility both grow with the chain. - **Portability layer**: a weekly or monthly logical dump — smaller, version-independent, and the only practical way to recover one dropped table without restoring 4 TB. - **Copies**: at least one in a different region or account, immutable so credential compromise cannot delete it. - **Retention**: dailies for a few weeks, weeklies for months, monthlies for whatever compliance demands, expired chain-aware. The binding tradeoff is that incrementals fit the window but lengthen restore; I would size the full cadence by how long a full restore actually takes on this hardware, measured, not assumed.
go deeper
Say that a database this size needs physical backups with a weekly full and daily incrementals, and that a dump is impractical.
Add the reasoning: dump and restore times, incremental chain length, off-site copies, and a concrete retention rotation.
Cover taking backups from a replica, immutability, synthetic fulls, chain-aware expiry, and the need to time an actual restore.
Frame the whole thing as choosing where cost sits — nightly window, storage bill, or incident duration — make the assumptions explicit and measurable, and keep the scheme simple enough to execute under pressure.
## Start from the constraints Four numbers drive the design: dataset size (4 TB), daily change volume, the two-hour window, and how long a full restore is allowed to take. The first three are known or measurable; the fourth is a business decision. Everything below follows from them. ## Why logical dumps cannot be the base layer A 4 TB logical dump would take many hours to produce, hold an MVCC snapshot open the whole time (bloating the primary), and — worse — restore in a multiple of that, because restore rebuilds every index and revalidates every constraint. Whatever the tolerated downtime is, a logical restore of this database will not meet it. Logical stays in the design, but as a secondary artifact. ## The base layer: physical, full + incremental Physical backups copy data files at IO speed and restore without rebuilding indexes, so they are the only type that scales here. - **Weekly full**: even at good throughput, 4 TB is a long read. Run it at the weekend, from a replica so the primary's IO and cache are untouched. - **Daily incrementals**: only changed blocks, so a nightly run of a fraction of the dataset fits the two-hour window comfortably. - **Chain length**: seven links by Saturday. Restore means full + up to six incrementals applied in sequence, and any damaged link truncates recovery. If the measured restore time is unacceptable, either shorten the cycle (twice-weekly fulls), or use synthetic fulls where the backup system merges the chain server-side so restore is always a two-step operation. Differentials are the middle option: bigger nightly artifacts, but restore is always full + latest. On a database whose daily change rate is low relative to size, differentials stay small for most of the week and buy real restore simplicity. ## The portability layer: logical Keep a weekly or monthly logical dump — ideally schema-only plus per-table dumps of the important tables, or a full dump taken from a restored copy rather than from production. It buys three things physical backups cannot: recovery of a single dropped or corrupted table without restoring 4 TB, a path across major versions and platforms, and an artifact a human can inspect. It is cheap precisely because it is infrequent. ## Where the backups are taken and stored - **Source**: a dedicated replica. Backup load then never competes with user traffic, and a saturated backup does not degrade the application. - **Destination**: object storage, with at least one copy in a separate region and, ideally, a separate account with object-lock/immutability. A backup an attacker or a runaway script can delete is not a backup; ransomware scenarios are the reason immutability is now table stakes. - **Compression and parallelism**: both shorten the window and shrink storage; both cost CPU on the backup host, which is fine when that host is a replica. ## Retention A grandfather-father-son shape: daily incrementals kept a few weeks, weekly fulls a few months, monthly fulls for the compliance horizon. Expiry must be chain-aware — deleting a full silently invalidates every incremental that depends on it. Longer retention is not only about disaster: silent logical corruption is often discovered days later, and retention is what decides whether a clean copy still exists. ## The tradeoffs to state explicitly - **Window vs restore time**: incrementals optimise the nightly window; fulls and differentials optimise the incident. You cannot maximise both, so decide which failure the business fears more. - **Storage cost vs recovery breadth**: longer retention and more copies cost money and buy protection from late-discovered corruption. - **Backup load vs freshness**: taking backups from a replica removes production load but couples backup freshness to replication health, which must then be monitored. - **Simplicity vs optimisation**: every clever mechanism — synthetic fulls, split dumps, tiered storage classes — is one more thing to be broken at 3 a.m. Prefer the simplest scheme that fits. ## What makes the plan credible Numbers, not adjectives: measured full-backup throughput, measured incremental size, and a measured full restore on representative hardware. Until a restore of this exact artifact set has been timed end to end, every duration in the plan is an estimate — and cold storage tiers, throttled downloads and single-threaded restore steps are where estimates usually break.
- Your CFO asks why you keep monthly backups for a year when the database is fine. What is the argument?Recent backups protect against loss of the machine; old backups protect against loss of correctness. A bad migration, a subtle application bug, or a malicious change can corrupt data that nobody notices for weeks, by which time every short-retention copy contains the corruption. Long retention is also often a compliance obligation, and cold storage tiers make it cheap relative to the risk.
- Would you ever run backups against the primary rather than a replica?Yes, when the replica cannot be trusted as a source or the topology has none — the backup must reflect the authoritative data, and a lagging or diverged replica silently backs up the wrong state. In that case throttle the backup's IO, run it in the low-traffic window, and prefer physical over logical so no long MVCC snapshot is held. The general rule is that backing up a replica is an optimisation that depends on replication health being monitored as carefully as the backup itself.
- How does the choice of incremental versus differential change if the daily change rate is 30% of the dataset?Differentials stop being attractive: by midweek each one approaches the size of a full and will not fit the two-hour window. Incrementals stay bounded by the daily delta, so they remain viable, and the chain should be kept short with more frequent or synthetic fulls to control restore time. At that change rate it is also worth questioning whether the full cadence should be daily on faster storage.
saying these in an interview costs you the question
- Proposing nightly logical dumps for a multi-terabyte database
- Quoting restore times that have never been measured
- Ignoring that incremental chains lengthen and can break
- Keeping every copy in the same account or region as the database
- Expiring a full backup without noticing its dependent incrementals become useless