A PostgreSQL standby without a replication slot stops streaming with 'requested WAL segment has already been removed'; what happened, and how do wal_keep_size and restore_command help?
answer
- checkpoints recycle old segments
- a floor in megabytes, default 0
- the archive as a second source
- who pays for the outage
basics
~20 sThe primary recycled WAL past its last checkpoint before the slotless standby fetched it. wal_keep_size (default 0, in megabytes) keeps a minimum of past WAL in pg_wal; a restore_command reading a WAL archive lets the standby catch up from there.
solid answer
~50 sAfter each checkpoint the primary removes or recycles WAL segments it no longer needs for crash recovery, unless something holds them, such as `wal_keep_size`, a replication slot, or a segment not yet archived. A standby without a slot that is disconnected or slow for longer than that window asks for a segment that is gone, and the WAL sender fails with `requested WAL segment ... has already been removed`. `wal_keep_size` (default `0`, megabytes, reload) keeps at least that much past WAL in `pg_wal` — a floor, not a cap — so size it as WAL rate times the outage you want to ride out. If the standby also has a `restore_command` reading a WAL archive, it fetches the missing segments from there and switches back to streaming on its own. Without an archive, the standby must be rebuilt from a new base backup.
code
ini · 5 lines# postgresql.conf on the newsroom primary
wal_keep_size = 8GB # floor of past WAL kept for slotless standbys
# postgresql.conf on the reporting standby
restore_command = 'cp /mnt/wal_archive/%f %p'go deeper
Recall that the primary deletes old WAL after checkpoints, so a standby that falls too far behind without protection can no longer stream.
Explain wal_keep_size as a megabyte floor with default 0 and reload context, and the standby's loop over restore_command, pg_wal and streaming.
Show the judgement: size the keep window from measured WAL rate, pair it with an archive, choose slot versus keep window by who should pay for an outage, and alert before byte lag crosses the window.
Set a retention policy per standby class: what disk the primary may spend, how long an outage each class must survive, and when rebuilding a standby is cheaper than keeping its WAL.
## Why the segment was gone The primary writes WAL into **segment files** in `pg_wal` (16 MB each unless `initdb` chose another size). After each **checkpoint**, segments older than the checkpoint's redo point are no longer needed for crash recovery, and they are removed or recycled. The usual reasons a primary keeps older segments anyway are: 1. **`wal_keep_size`** — a minimum amount of past WAL kept for standbys. 2. **Replication slots** — each slot holds WAL back from its `restart_lsn`; that is a separate subject with its own safety limits. 3. **Archiving still pending** — with archiving on, a segment is not removed until it has been archived. A standby **without a slot** gets none of the second kind of protection. If it is disconnected — a network partition, a long maintenance window, a standby that was simply too slow — for longer than the primary takes to move past the retained window, the segment it needs next is gone. Its WAL sender then fails with `requested WAL segment 00000001000000A30000004F has already been removed`, and the standby's log shows the same text when its receiver gets the error. ## What the standby does next A server in standby mode keeps cycling through its WAL sources until it is stopped or promoted: 1. Run `restore_command` to fetch the next segment from a **WAL archive**. 2. Look in its own `pg_wal` directory. 3. Connect with `primary_conninfo` and stream. Between failed rounds it waits `wal_retrieve_retry_interval` (5 s by default). So the error is **final only when there is no archive**. With an archive that still holds the segment, the standby replays from it until it reaches WAL the primary still has, then switches back to streaming without operator action. Without one, restarting does not help: the standby must be rebuilt from a new base backup. ## The wal_keep_size setting | Property | Value | |---|---| | Default | `0` — keep nothing extra for standbys | | Unit | megabytes when written without a unit (`wal_keep_size = 8192` is 8 GB) | | Context | `sighup` — a reload is enough | | Applies on | any sending server: the primary, and a cascading standby at each restartpoint | | Meaning | a **minimum**, not a maximum: checkpoints and archiving may keep more | `wal_keep_size` replaced the older `wal_keep_segments`, which counted segments instead of megabytes; PostgreSQL 18 no longer has the old name. ## Three ways to cover a disconnected standby | Mechanism | Disk cost on the primary | What a long outage does | |---|---|---| | `wal_keep_size` | fixed: about that much extra in `pg_wal`, always | standby falls off the end and needs the archive or a rebuild | | WAL archive + `restore_command` | none in `pg_wal`; the archive is stored elsewhere | standby catches up from the archive, if the archive kept enough | | replication slot | unbounded unless capped | primary keeps everything; the risk moves to the primary's disk | The trade is **who pays for an outage**. With `wal_keep_size` the standby pays: it may need rebuilding. With a slot the primary's disk pays. For a disposable reporting standby, a bounded `wal_keep_size` plus an archive is often the safer pair, because an abandoned reporting server can never fill the primary's disk. ## Sizing it 1. Measure the WAL rate: note `pg_current_wal_lsn()`, wait through a busy hour, then compute `pg_wal_lsn_diff()` between the new and the noted position. 2. Multiply by the longest disconnection you want to survive without the archive — a standby reboot, a network maintenance window. 3. Check that the `pg_wal` filesystem holds that **on top of** normal checkpoint WAL. `max_wal_size` (1 GB by default) is a checkpoint trigger, not a retention promise, and `min_wal_size` (80 MB) only governs recycling. ## When it has already happened - If an archive holds the segment, add or fix `restore_command` on the standby and reload: it catches up from there. - If not, rebuild the standby from a fresh base backup; no setting on the primary can bring a removed segment back. - Before rebuilding, check whether the same thing will recur: compare the byte lag the standby reached before it broke with the window `wal_keep_size` provides. - Afterwards, decide which of the three mechanisms this standby should have had, and alert on byte lag before it crosses the retained window.
- Why might you give a PostgreSQL reporting standby wal_keep_size and an archive rather than a replication slot?Disk safety on the primary. `wal_keep_size` costs a fixed amount of `pg_wal` space, so a reporting standby that is abandoned or down for days can never fill the primary's disk. The price is that after a long outage it must catch up from the archive or be rebuilt. A slot guarantees the WAL instead and moves that risk to the primary.
- Does a cascading PostgreSQL standby need its own wal_keep_size?Yes, for its downstream standbys. A cascading standby is a sending server too, and it removes old segments at its restartpoints. Its own `wal_keep_size`, or a slot on it, is what protects the standbys streaming from it; the primary's setting protects only standbys connected to the primary.
- wal_keep_size caps how much WAL pg_wal may hold.
- wal_keep_size is counted in 16 MB segments.
- Restarting the standby fixes a removed-segment error.
- max_wal_size makes the primary keep WAL for slow standbys.
- Without a slot, the primary waits for standbys before recycling WAL.