Transactional Outbox & CDC

## What the dual write actually is The dual-write problem arises whenever a single logical operation requires committing changes to two independent systems that share no transaction coordinator - most commonly an **OLTP database** and a **message broker**. Concretely: a service handling 'place order' needs to - (1) persist the order row so the business fact is durable, and - (2) tell the rest of the system about it by publishing an event such as `OrderPlaced` to Kafka, RabbitMQ, or SNS. These are two separate network calls to two separate pieces of infrastructure, each with its own failure modes, its own latency profile, and nothing built in that makes them succeed or fail together. ## Why it matters Why does this matter? Because a huge number of real systems are built around the assumption that 'if it's in the database, other services eventually find out.' - A **shipping service** waits for `OrderPlaced` to schedule a warehouse pick. - A **notifications service** waits for it to email the customer. - An **analytics pipeline** waits for it to update a dashboard. If the event silently never arrives, none of that happens, and there is often no error anywhere - the order just sits there, correctly saved, invisibly orphaned from the rest of the system. This is worse than a loud failure because nothing pages anyone; it surfaces days later as 'why didn't this customer get their confirmation email,' and by then the causal thread is cold. ## Neither ordering is safe The mechanics of the failure are simple to state: | Ordering | What the failure looks like | |---|---| | **save-then-publish** | means a crash, deploy restart, OOM kill, or network partition between the two calls leaves the DB committed and the event never sent | | **publish-then-save** | inverts the risk: the broker successfully has the message, but the subsequent DB write then fails (constraint violation, connection pool exhaustion, deadlock), and now consumers process an event describing an order that doesn't exist | Neither ordering is safe; you've just chosen which side of the story goes missing. ## The naive fixes, and why they collapse Naive 'fixes' don't hold up either. - **Wrapping both calls in a try/catch** and retrying the publish on failure doesn't help if the process itself dies before the retry logic ever runs - the retry loop lives in the same doomed process as everything else. - **Blindly retrying without deduplication** trades missing events for duplicate ones, pushing the problem downstream onto every consumer instead of solving it. ## Why the textbook fix is not used The theoretically 'correct' fix - a distributed transaction (two-phase commit, XA) spanning the database and the broker - is rarely used in practice. Most modern brokers (Kafka, SQS, SNS, most managed RabbitMQ setups) don't support XA participation at all, and even where **2PC** is technically available, it - introduces a transaction coordinator as a new single point of failure, - holds locks/resources open across both systems for the duration of the coordination protocol, and - interacts badly with autoscaled, ephemeral service instances that come and go. The performance and operational cost of 2PC across a database and a broker is high enough that essentially no mainstream microservices architecture uses it for this purpose. ## The practical resolution The practical resolution, which this leaf's sibling questions build on, is to convert the two-system problem into a one-system problem: 1. Write the event's intent to publish into the same database, in the same local **ACID** transaction as the business change (a dedicated **outbox table**). 2. Let a separate, decoupled process (a polling relay or a change-data-capture connector like `Debezium`) pick up that durably-recorded intent and deliver it to the broker asynchronously, with retries, independent of the original request's lifecycle. Because the outbox row and the order row commit or roll back together as one atomic unit, there is no window in which the database has the order but no record exists anywhere that an event needs to go out. The remaining relay step is decoupled and can safely retry indefinitely without risking the original business transaction, because it operates entirely after that transaction is already durably committed. ## Where it shows up A concrete real-world instance of this: Chris Richardson's microservices.io catalog documents the transactional outbox pattern explicitly as the standard answer to 'how to reliably send events when using a database and a message broker' in a microservices architecture, precisely because ad hoc dual writes were a recurring, hard-to-debug source of silent data loss across early microservice migrations.