skip to content

Replication from a primary to a replica happens in several stages. Which stages can lag independently, and what do PostgreSQL's pg_stat_replication view and MySQL's Seconds_Behind_Source field each tell you about them?

level: middleimportance: must knowfreq 56%

answer

  1. Stages: generate → send → write → flush → replay
  2. sent_lsn ≥ write_lsn ≥ flush_lsn ≥ replay_lsn
  3. Big gap after flush → applier is the bottleneck
  4. Seconds_Behind_Source = apply lag only; NULL when threads stopped
  5. Heartbeat row measures true end-to-end write-to-visible latency

basics

~20 s

Stages: the primary generates log records, sends them, the replica writes them, flushes them to disk, then replays them. pg_stat_replication exposes sent/write/flush/replay positions and the matching lag intervals per stage. MySQL's Seconds_Behind_Source covers only the apply stage — receive lag is separate.

solid answer

~60 s

Replication is a pipeline, and each hop can be the bottleneck: 1. **Generate** — the primary writes the change to its log. 2. **Send** — the record goes over the network. 3. **Write/receive** — the replica writes it into its own log. 4. **Flush** — the replica makes it durable on disk. 5. **Apply/replay** — the replica changes its actual data, making it visible to readers. **PostgreSQL:** `pg_stat_replication` on the primary shows, per connected standby, `sent_lsn`, `write_lsn`, `flush_lsn`, `replay_lsn`. Subtracting each from `pg_current_wal_lsn()` gives byte lag per stage; the `write_lag`, `flush_lag`, `replay_lag` columns give the same as time intervals. `replay_lsn` is the one that governs what a read on the standby sees. **MySQL:** the receive side (I/O thread) and apply side (SQL/applier threads) are separate. Replica status shows `Read_Source_Log_Pos` (received) versus `Exec_Source_Log_Pos` (applied); `Seconds_Behind_Source` is derived from the timestamp of the event the applier is currently executing — it is an **apply-lag** metric and reads NULL when replication is not running. Diagnosing means asking *which* stage grew: send/write points at network or replica I/O, replay points at the applier.

code

sql · 6 lines
sql
SELECT client_addr,
       pg_wal_lsn_diff(pg_current_wal_lsn(), sent_lsn)   AS send_bytes,
       pg_wal_lsn_diff(sent_lsn,            flush_lsn)   AS flush_bytes,
       pg_wal_lsn_diff(flush_lsn,           replay_lsn)  AS replay_bytes,
       write_lag, flush_lag, replay_lag
FROM pg_stat_replication;

go deeper

for a junior

Know that receiving a change and applying it are different steps, and that the applied position is what a query on the replica can see.

for a middle

Name the four PostgreSQL positions and the MySQL I/O-versus-applier split, and say which metric maps to which stage.

for a senior

Lead with the diagnostic use: locate the widening gap, then act on that stage. Mention NULL-handling and heartbeat measurement as monitoring hygiene.

for a principal

Specify the standard lag telemetry every replica must emit — per-stage bytes, heartbeat seconds, thread state — so incident response is uniform across a fleet.

## Why staging matters Saying "the replica is behind" is not actionable. The fix for a saturated network link is nothing like the fix for a single-threaded applier that cannot keep up. Every mature lag investigation starts by locating **which hop in the pipeline** the backlog is sitting in. The pipeline, engine-agnostically: 1. **Generate** — the primary appends the change to its write-ahead/binary log as part of commit. 2. **Send** — a sender process streams those bytes to the replica over the network. 3. **Receive/write** — the replica writes the arriving bytes into its own local log (PostgreSQL: the WAL receiver writes WAL; MySQL: the I/O thread writes the relay log). 4. **Flush** — the replica fsyncs that log so the received data would survive a crash. This is the position that matters for durability and for how much data a failover can recover. 5. **Apply/replay** — a separate process reads the local log and mutates the actual data pages/rows. Only after this can a query on the replica see the change. Because 3–5 are decoupled, a replica can be perfectly current on receive and badly behind on apply. This is extremely common and is the single most useful distinction in the whole topic. ## Reading it in PostgreSQL On the **primary**, `pg_stat_replication` has one row per connected standby with four position columns that map one-to-one onto the pipeline: - `sent_lsn` — how far the primary has sent. - `write_lsn` — how far the standby has written into its WAL. - `flush_lsn` — how far the standby has fsynced. - `replay_lsn` — how far the standby has replayed into its data. Subtract each from the primary's current WAL position to get byte lag for that stage. PostgreSQL additionally publishes `write_lag`, `flush_lag`, `replay_lag` as time intervals — the measured round-trip time for a change to reach each stage on that standby. These are only populated while changes are actually flowing. The invariant is `sent_lsn >= write_lsn >= flush_lsn >= replay_lsn`. Where the big gap opens tells you the culprit: - gap between the primary's current position and `sent_lsn` → sender or network backpressure; - gap between `sent_lsn` and `flush_lsn` → replica disk write/fsync throughput; - gap between `flush_lsn` and `replay_lsn` → **recovery/apply is the bottleneck** (the usual case). On the **standby** itself, `pg_last_xact_replay_timestamp()` gives the commit time of the most recently replayed transaction; `now() - pg_last_xact_replay_timestamp()` is the classic time-lag expression — with the standard caveat that it grows on an idle primary simply because no new transaction has arrived. ## Reading it in MySQL MySQL splits the replica into threads: the **I/O (receiver) thread** pulls binlog events into the relay log, and the **SQL/applier thread(s)** execute them. Replica status therefore exposes two positions on the source's coordinates: - `Read_Source_Log_File` / `Read_Source_Log_Pos` — received. - `Relay_Source_Log_File` / `Exec_Source_Log_Pos` — applied. The difference between those two is **relay backlog**, i.e. apply lag in bytes. The difference between the source's current binlog position and `Read_Source_Log_Pos` is **receive lag**. `Seconds_Behind_Source` is computed from the timestamp stamped on the event the applier is currently executing versus the replica's clock. Three consequences follow, and interviewers probe all of them: - It is an **apply-side** metric only. If the I/O thread is stalled or disconnected, the applier drains its relay log and the field can fall to 0 while the replica drifts arbitrarily far behind reality. - It reads **NULL** whenever replication threads are not both running — that is a state, not a number, and dashboards that coerce NULL to 0 hide outages. - In a **chained** topology (A → B → C), events carry the original timestamp from A, so C's value reflects lag behind A, not behind B. Useful, but only if you know that is what you are reading. The performance-schema replication tables (`replication_connection_status`, `replication_applier_status_by_worker`) expose the same split with per-worker detail, which is what you use once multi-threaded apply is enabled. ## The heartbeat technique Because both engines' time metrics degrade when writes are sparse, production setups add a **heartbeat**: a one-row table on the primary updated with the current timestamp on a fixed interval (typically 1s) by an external agent. On the replica, `now() - heartbeat_ts` is a truthful, always-flowing end-to-end lag measurement that also proves the whole pipeline is alive. It measures exactly what a user experiences — write-to-visible latency — rather than an internal position. ## Putting it together A useful lag dashboard carries, per replica: byte lag at receive, byte lag at apply, heartbeat-derived seconds, and replication-thread/connection state. That set makes each failure mode distinguishable at a glance instead of leaving you with one ambiguous number.

  • On a PostgreSQL standby, which position determines what a read-only query can see?
    The replay position. Received and flushed WAL exists on the standby's disk but has not yet been applied to the data pages, so a query cannot observe it. That is why replay_lsn — and the gap between flush_lsn and replay_lsn — is the number to watch for read staleness, while flush_lsn is the number that matters for how much data survives a failover.
  • Why would you add a heartbeat table when the engine already reports lag?
    Engine metrics are computed from internal positions and degrade exactly when you need them: they read zero or null on an idle primary, and MySQL's Seconds_Behind_Source can read zero while the receiver is dead. A heartbeat guarantees a steady write stream, so the replica-side subtraction is always meaningful, and it measures true end-to-end write-to-visible latency across every stage including the network.

saying these in an interview costs you the question

  • Treating replication lag as a single number with no stage breakdown
  • Believing Seconds_Behind_Source covers network/receive lag
  • Coercing a NULL Seconds_Behind_Source to 0 in dashboards, hiding a stopped replica
  • Comparing flush position when the question is about read staleness (that is replay)
  • Assuming a chained replica's time-behind is measured against its immediate upstream

context