A team implements the outbox pattern using a database polling relay that queries SELECT * FROM outbox WHERE published = false every second. A colleague suggests replacing it with Debezium reading the database's write-ahead log via change data capture instead. What changes, and what are the trade-offs?

## The same job, read two different ways Both approaches solve the same problem - getting rows out of the outbox table and onto the broker - but they read that table in fundamentally different ways, and the difference has real consequences for latency, load, and operational surface area. | | Polling relay | Change data capture | |---|---|---| | **Latency** | bounded by the poll interval | as soon as the transaction commits | | **Query load** | a query against the live table on every poll cycle | a stream the database is already producing | | **Operational surface area** | an application process with a loop | a replication slot or binlog access, and a connector | ## How a polling relay reads the table A polling relay is the simplest possible implementation: a loop that periodically issues a SQL query like `SELECT * FROM outbox WHERE published = false ORDER BY id LIMIT 100`, publishes each returned row to the broker, and marks it published (or deletes it) on success. - **Load.** Every poll cycle is a query against the live table, so at high poll frequency (every 100-500ms) this adds sustained read load, and even with a good index on the published/id columns, that's still periodic contention on a table that's also being written to by every incoming request. - **Latency.** Latency is bounded by the poll interval - on average, an event sits in the table for about half the interval before the next poll picks it up, so a 1-second interval means roughly 500ms of added delivery latency on top of everything else, worse in the tail when a poll cycle happens to just miss a fresh insert. ## How change data capture reads it instead Change data capture flips the direction of information flow: instead of the relay asking the database 'anything new?', the database tells the relay 'here's what changed,' by exposing its internal transaction log. - **Postgres** does this via logical replication slots (using an output plugin like `pgoutput` or `wal2json`). - **MySQL** exposes its binlog. `Debezium` is a Kafka Connect source connector that attaches to that log, decodes committed row-level changes, and emits them as Kafka messages essentially as soon as the transaction commits - typically single-digit-to-low-double-digit milliseconds of added latency, and no repeated `SELECT` load on the table at all, since it's reading a stream the database is already producing for replication purposes. ## What that improvement costs The cost of that improvement is operational, not conceptual. Standing up CDC means: 1. enabling logical replication or binlog access on a production database (a configuration change with its own resource and retention implications - unconsumed WAL segments can accumulate if the connector falls behind or goes down, risking disk pressure on the primary), 2. deploying and monitoring a Kafka Connect cluster running the Debezium connector, 3. handling connector restarts and offset management so it resumes from the right point in the log after a crash, and 4. understanding connector rebalancing behavior in a distributed Connect deployment. None of that exists with a polling relay, which is just an application process with a database connection and a loop - trivial to write, deploy, and reason about, at the cost of higher latency and steady query overhead. ## What CDC does not change A point worth being precise about: CDC does not remove the need for the outbox table itself, and it does not upgrade delivery to exactly-once. - **The outbox table.** Teams still capture from a dedicated outbox table rather than business tables directly, because business tables have internal-shape columns and high-frequency unrelated updates that would leak implementation detail and generate noisy, irrelevant change events; the outbox table exists specifically to be a clean, intentional, append-only public event schema, regardless of which relay mechanism reads it. - **Delivery.** CDC-based relays are still at-least-once in practice - connector restarts, offset resets after a rebalance, and downstream retry logic can all produce redelivery, so consumers still need to dedupe. ## Choosing between them The practical decision is a cost-benefit call tied to scale and existing infrastructure. - **A well-indexed poller.** A low-to-moderate volume service where sub-second latency doesn't matter, or a team without an existing Kafka Connect footprint, is often better served by a well-indexed poller at a reasonably short interval - simpler to debug, fewer moving parts, one less thing that can silently stall. - **Debezium.** A team already running Kafka Connect for other services, or one whose downstream consumers genuinely need near-real-time freshness (e.g., search indexing, fraud detection), gets a much better marginal return from adopting Debezium. This mirrors how it's used at real scale: companies like Shopify and Airbnb have documented using Debezium-based CDC on outbox tables specifically to get low-latency, reliable event propagation out of sharded MySQL/Postgres fleets without hand-rolling pollers per service.