A team is deciding between synchronous and asynchronous replication for a database that must tolerate a primary node failure. What availability and durability guarantees does each give, and what specific failure mode does asynchronous replication risk that synchronous replication avoids?

## What the acknowledgment actually promises **Replication** is the core redundancy mechanism distributed systems use to survive node failure: by keeping multiple copies of data on independent machines, the system can continue serving, and if designed carefully keep, data even after losing one or more of those machines. But how replication is performed - specifically, whether the primary waits for replicas before acknowledging a write - determines exactly what guarantee the write succeeded actually carries, and this is one of the sharpest trade-offs in fault-tolerant system design. ## Synchronous replication In **synchronous replication**, the primary node does not tell the client a write succeeded until at least one, or more robustly a quorum of, replicas has durably received and persisted that write. This means that at the instant the client is told success, the data is provably present on more than one machine, so if the primary immediately crashes, a replica can be promoted with zero loss of that write - this durability guarantee is what's usually meant by **RPO=0**, a recovery point objective of zero data loss. What it costs: - The cost is paid on the write path: every write's latency is bounded below by the round trip to the slowest replica required for acknowledgment, not just the primary's local disk. - Worse, if a required replica becomes unreachable, the primary faces an **availability dilemma**: it must either block or refuse new writes until the replica is reachable again, sacrificing availability to preserve the durability guarantee, or silently downgrade to asynchronous behavior, which breaks the guarantee it claims to offer. Systems that need true zero-loss synchronous replication typically use a quorum rather than all replicas specifically so that one slow or dead replica out of several doesn't halt the whole system. ## Asynchronous replication **Asynchronous replication** takes the opposite stance: the primary persists the write locally, immediately acknowledges the client, and ships the write to replicas in the background, independent of the client-facing response. This decouples client-perceived latency entirely from replica health and network conditions - writes are fast and the primary stays fully available even if every replica is temporarily unreachable, since replication is not on the acknowledgment critical path. This introduces two failure modes: - **A data-loss window.** The risk this introduces is precisely a data-loss window: if the primary crashes after acknowledging a write but before that write has propagated to any replica, the write is gone the moment a replica is promoted to primary, even though the client was already told it succeeded. This isn't a rare edge case in high-throughput systems - it's a routine, expected possibility whose size is governed by the replication lag at the moment of failure. - **A silent revert.** A related and subtler failure mode is a silent revert: a client is told write X succeeded, the primary crashes before shipping X, a lagging replica is promoted, and that replica now serves reads that don't reflect X at all - to the client, data that was confirmed durable has simply vanished, which can be worse than an outright failure because there's no error to react to. ## Matching the mode to the workload The choice between the two is a direct, concrete instance of the durability/latency/availability trade-off that underlies most of fault-tolerant systems design, and it's usually resolved by matching the replication mode to the actual **RPO** and **RTO** requirements of the workload rather than picking one universally. | Workload | Mode it usually picks | The reasoning | |---|---|---| | Financial ledgers, payment systems, and anything where a confirmed success must be an unbreakable promise | typically require synchronous, or synchronous-to-quorum, replication | despite the latency cost, because silent data loss is unacceptable | | High-throughput logging, analytics ingestion, or systems where a small, bounded loss window is tolerable | often use asynchronous replication | in exchange for much higher write throughput and lower tail latency | ## Semi-synchronous, the middle ground Many production systems land on a middle ground called **semi-synchronous replication**: the primary waits for acknowledgment from at least one replica, not all and not necessarily a full quorum, before confirming to the client, then ships to the remaining replicas asynchronously. This bounds the worst-case data loss to at most what wasn't yet on that one synchronous replica, while avoiding the full latency and availability cost of waiting on every replica or a full quorum. It's a pragmatic compromise, though it doesn't provide the same formal guarantee as full quorum-synchronous replication - if that one synchronous replica and the primary fail simultaneously, a **correlated failure** such as both being in the same rack or availability zone, the guarantee is void, which is why the physical placement of the synchronous replica matters just as much as the replication mode itself.