Why does a logical replication slot that no CDC consumer reads fill the source primary's disk?

## What a slot actually promises A replication slot is a small piece of durable server state that says: *a consumer named `x` has confirmed up to position P, so do not throw away anything from P onward.* That promise is exactly what makes log-based capture reliable — a connector can crash, be redeployed, or be offline for a maintenance window and resume without losing a single change, because the server held the log for it. The same promise is what makes it dangerous. The server has no way to know the difference between "this consumer will be back in five minutes" and "this consumer was deleted three weeks ago". A slot is persistent: it survives connector disconnects and server restarts, and it is removed only when someone explicitly drops it. ## The failure sequence 1. A slot exists with a restart position P. 2. The consumer stops confirming — the pipeline is paused, the connector container is gone, credentials expired, the task is failing in a crash loop, or the pipeline was retired and nobody dropped the slot. 3. The application keeps committing, so the log keeps growing beyond P. 4. Checkpoints run, but the server may not recycle segments at or after P, because the slot still needs them. 5. The WAL directory grows monotonically until the filesystem is full. 6. A primary that cannot write WAL cannot commit. Writes fail; the database is effectively down. That last step is why interviewers ask this: it is the case where the *analytics* pipeline takes out the *transactional* database. Nothing else in a typical warehouse stack has that reach. ## The second, quieter cost A logical slot also pins a catalog `xmin` — the decoder needs the catalog as it stood when those changes were made, so vacuum is prevented from removing catalog tuples newer than that horizon. An old slot therefore also produces catalog bloat and degraded planning over time, which is easy to misdiagnose as an unrelated problem. ## The low-traffic-database trap In Postgres, WAL is written for the whole cluster, but a logical slot decodes changes for exactly one database. If the captured database is quiet while other databases in the same cluster are busy, the decoder finds nothing to emit, so the consumer has nothing to confirm and the slot's confirmed position stays where it is — while WAL from the *other* databases piles up behind it. The pipeline looks perfectly healthy (no lag in events, no errors) and the disk fills anyway. The standard remedy is a heartbeat: the capture periodically writes a row to a small dedicated table in the captured database, which produces a change the decoder can emit and the consumer can confirm, dragging the slot's position forward. Most capture tools expose this as a heartbeat interval setting; the mechanism is the same whoever implements it. ## Operating slots properly - **Monitor retained bytes per slot, not connector lag.** Connector-reported lag can be near zero while the slot retains gigabytes. ```sql SELECT slot_name, active, wal_status, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained FROM pg_replication_slots ORDER BY pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) DESC; ``` - **Alert on `active = false`.** An inactive slot on a busy database is a countdown. - **Cap the retention.** Postgres 13 added `max_slot_wal_keep_size`, which is unlimited unless you set it. Setting it means an over-retaining slot is *invalidated* — the pipeline breaks and must be re-seeded — rather than the primary filling its disk. That is usually the right trade: a broken pipeline is recoverable, a wedged primary is an outage. - **Make slot lifecycle part of decommissioning.** Retiring a pipeline must include dropping its slot. Orphan slots from deleted environments are the single commonest cause of this incident. - **Keep headroom and know the drain path.** During an incident the options are: restart the consumer so it confirms and the slot advances (best), drop the slot and accept re-seeding the pipeline (fast, costly downstream), or add disk to buy time. Deleting WAL files by hand corrupts the database — never an option. ## The framing that matters This risk is the price log-based capture charges for its main advantage. Polling cannot take the source down; it just scans. A slot buys you resumable, delete-accurate, low-latency capture, and in exchange the source database now has a dependency on a downstream consumer staying alive. Owning log-based CDC means owning that dependency explicitly — with monitoring, a retention cap, and a decommissioning checklist.