When a service updates its own database and then needs to publish an event about that change so other services can update their read models, what can go wrong if it does these as two separate steps, and what's a common pattern to fix it?

## Two writes that cannot be made one Consider a service handling a request that both changes its own data and needs to tell the rest of the system about that change: it opens a local database transaction, updates its own tables, and commits — then, as a separate step, calls a message broker to publish an event describing what happened. The problem is that these are two independent operations against two different systems that don't share a transaction coordinator, so there's no way to guarantee both happen or neither does. - **Commit, then publish.** If the process crashes, the network hiccups, or the broker is briefly unreachable in the gap between the commit and the publish call, you get a database that changed with no event ever sent — other services relying on that event to update their own read models never find out, and their copies silently fall behind with no error raised anywhere. - **Publish before commit.** The reverse ordering is just as broken: if the publish succeeds but the subsequent database write then fails or rolls back due to a constraint violation, you've announced something to the rest of the system that never actually happened. ## Why it happens This is called the **dual-write problem**, and it exists because a relational database transaction and a message broker publish are fundamentally different kinds of operations with no shared all-or-nothing guarantee between them. In principle you could use a distributed transaction protocol like two-phase commit across both systems, but in practice this is rarely used: - most modern message brokers don't support distributed-transaction protocols well or at all; - even where it's available it adds real latency and couples the availability of the write path to the availability of the broker, which defeats much of the point of decoupling services with asynchronous messaging in the first place. ## The standard fix The standard fix is the **transactional outbox pattern**. Instead of publishing to the broker directly, the service writes the event as a row into an 'outbox' table, inside the exact same local database transaction as the business data change. Because both writes go to the same database in the same transaction, they are atomic by construction: either both the business change and the outbox row commit together, or neither does, using nothing more exotic than the local ACID guarantee every relational database already provides. Delivery to the broker is then handled by a separate process: 1. either a lightweight **relay** that polls the outbox table for unpublished rows and forwards them, 2. or, increasingly common, a **change-data-capture** tool that tails the database's transaction log for new outbox rows and streams them to the broker without the application needing to poll anything itself. Once a row is confirmed delivered, the relay marks it published, or deletes and archives it. ## What the fix still costs This fix has its own costs and its own remaining failure mode to design around. It's inherently at-least-once delivery, not exactly-once: if the relay crashes after successfully publishing but before marking the row as sent, it will republish that same event on restart. Consumers of these events therefore have to be idempotent, typically by tracking the highest event ID or version they've already applied per source and ignoring anything not newer, because delivered exactly once isn't a guarantee this pattern, or most real message brokers, actually provides. There's also new operational surface: - the outbox table needs monitoring for a growing backlog; - a stuck relay is now a silent failure mode of its own; - it needs periodic cleanup or archival so it doesn't grow unbounded, since every published business event otherwise sits there forever. ## Putting it together A concrete walk-through: in an order-placement flow, the Order service inserts a new order row and an `OrderPlaced` row into its outbox table in one local transaction, so both succeed or fail together. A change-data-capture tool reads the database's transaction log, notices the new outbox row, and forwards it to the message broker. The Inventory service and the Shipping service each independently consume that topic to update their own local read models, reserving stock and scheduling a shipment respectively, and each one is written to be idempotent, keyed on the event's unique ID, so that if the relay ever redelivers the same `OrderPlaced` event after a crash, Inventory doesn't double-reserve the stock and Shipping doesn't schedule two shipments for one order.