Explain the 3-2-1 backup rule and how you would apply it to a production relational database. What failure modes do off-site and immutable copies protect against that a second local copy does not?

## The rule 3-2-1 is a heuristic for **independence of failure**: at least **3** copies of the data, on at least **2** different media or storage platforms, with at least **1** copy off-site. It comes from photography and general IT, but maps cleanly onto databases. An extended form, 3-2-1-1-0, adds one **immutable or offline** copy and **zero** errors in verification. The rule is not about paranoia arithmetic. It exists because copies that share a dependency fail together. Two backups in the same bucket, under the same credentials, in the same region, written by the same tool, are close to one copy for risk purposes. ## Mapping it onto a relational database - **Copy 1 - the live cluster.** The primary and its data. Note that streaming replicas and multi-AZ standbys are **not** backups: they faithfully reproduce logical damage. A dropped table, a bad migration, or a `DELETE` without a `WHERE` clause reaches the standby in milliseconds. - **Copy 2 - a local or same-region backup repository.** Physical base backups plus the archived write-ahead log, on a different storage system from the database volumes. This is the copy you actually restore from most of the time, because it is fast to fetch. - **Copy 3 - an off-site copy.** Another region, and ideally another cloud account or provider. This is the copy that exists for events that take the whole site or account with them. The "two media" leg used to mean disk plus tape. Today it usually means block storage plus object storage, or two providers - the intent is that a defect or outage in one storage technology does not take both copies. ## What the off-site copy buys you A second local copy protects against exactly one class of event: media or host failure. It does nothing about: - **Site or region loss** - power, network, flood, fire, or a provider regional outage. - **Blast-radius automation** - a Terraform apply or a cleanup script that deletes a whole account's resources, including both local copies. - **Account compromise** - credentials that reach production also reach the local repository. ## What immutability buys you Ransomware and malicious insiders explicitly hunt backups first, because destroying recovery leverage is what makes the extortion work. If the credentials that write backups can also delete them, an attacker with those credentials has both. Countermeasures: - **Object lock / write-once-read-many retention** on the remote copy, so objects cannot be deleted or overwritten before their retention date, not even by the account root. - **A separate credential and trust domain** - a different account, subscription, or project, with a one-way push or, better, a pull performed by the backup account so production holds no delete rights on it. - **Versioning plus MFA-protected deletion** so accidental or scripted deletes are recoverable. - **Air-gapped or offline copies** for the highest tiers, where nothing online can reach the media at all. The practical test question is: *which single credential, if stolen, destroys every copy?* If there is one, 3-2-1 is decorative. ## Cost and pragmatism Off-site copies cost storage plus cross-region transfer, and restoring from them is slower. The usual compromise is tiering: keep a short window of dense, frequent backups locally for fast recovery, and push a thinner set (say weekly fulls plus log, or daily fulls with a shorter retention) to the remote immutable copy for disaster and ransomware scenarios. Compress and encrypt before transfer. Finally, the rule is worthless without the "0 errors" clause. A remote immutable copy that has never been restored from is an untested backup that also happens to be expensive. Every leg of 3-2-1 needs its own periodic proof-of-restore, and the off-site leg is the one most often skipped and most often needed.