How would you choose replication-lag alerting thresholds for a fleet of read replicas, and what should happen automatically when a replica exceeds them?

## Start from the purpose, not from a number "Alert at 10 seconds" is a number with no derivation, and it is wrong for most replicas in a fleet. A replica exists for a reason, and the reason dictates the budget. **Read-serving replicas** carry a *staleness budget*: the maximum age of data the feature can tolerate. A product-catalogue browse might tolerate 30 seconds; a post-checkout order page tolerates essentially none and should not be served from a replica at all. Different replica pools serving different features legitimately get different thresholds. **Failover candidates** carry a *recovery-point budget*: how much committed work may be lost when this node is promoted. This is measured against the **flushed** position (data durably on the replica's disk), because that — not the replayed position — bounds what a promotion can recover. Express it both in seconds (business language) and bytes (operational language). **Every replica**, whatever its role, is bounded by **log retention**. The primary keeps a finite amount of change log; a replica that falls further behind than that can no longer stream forward and must be rebuilt from a full copy — hours of work and, on a fleet, a capacity event. The mirror risk applies where a replication slot pins log on the primary for a replica's benefit: the backlog then consumes the *primary's* disk, and a full disk on the primary is an outage. Both directions justify a hard, high-severity threshold set well below the actual limit — with enough headroom to intervene. ## Alert on shape, not on instants Lag is spiky by nature: checkpoints, nightly batches, index builds, deploys. Threshold-only alerting on a spiky signal produces noise, and noise produces ignored pages. The informative distinction is between: - **A drainable spike** — lag rises, then falls back. Apply throughput exceeds the average write rate; the replica was momentarily overwhelmed. Only actionable if the peak breaches a budget. - **A sustained deficit** — lag rises monotonically over many minutes. Apply throughput is *below* the incoming write rate, and it will not recover on its own until the write rate falls or the replica's capacity changes. This is the condition that eventually hits log retention. So alert conditions should combine level and duration: "above X for N minutes", or "rising for N minutes with no drain". Recording the *derivative* alongside the level makes catch-up behaviour legible — during recovery lag should fall along a predictable slope, and a stalled recovery (flat, high lag) means the applier is blocked rather than working. A useful mental model: if the primary produces changes at rate *W* and the replica applies at rate *A*, then lag drains at *A − W*. Catch-up time from a backlog *B* is roughly *B / (A − W)*, and when *A ≤ W* it is infinite. That formula is what tells an operator whether waiting is a strategy. ## Severity tiers A workable ladder for a read-serving fleet: 1. **Info / ticket** — lag above a fraction of the staleness budget, sustained. Something changed; investigate during hours. 2. **Automated action** — lag above the staleness budget. The replica is drained from the read pool; no human needed yet. 3. **Page** — a large share of the pool is draining at once, or lag on a failover candidate breaches the recovery-point budget, or lag approaches the retention threshold. 4. **Page, highest severity** — replication stopped or errored, a replica has vanished from the primary's connection list, or a slot's retained log is nearing the primary's disk capacity. These are state alerts, independent of any lag number, and they must also fire when the metric stops arriving at all. ## What automation should do **Drain, don't panic.** Above the staleness budget, remove the replica from the read load balancer. This protects correctness (users stop seeing stale data) without touching replication, and it usually *helps* the replica catch up by freeing I/O and cache that were serving reads. **Use hysteresis.** Re-add only after lag has been below a lower threshold for a sustained period. Without it, a replica flaps in and out of the pool as lag oscillates around the line. **Protect against fleet-wide evacuation.** A cause common to all replicas — a giant batch job on the primary — will breach the threshold everywhere at once. Automation must cap how much of the pool can be drained (for example, never below a quorum), and the fallback must be a deliberate choice: shed load, serve degraded/cached results, or route to the primary if it can absorb it. Silently evacuating every replica onto the primary converts a staleness problem into an availability outage. **Never auto-promote a lagging replica.** Failover selection must consider the flushed position; promoting the most-behind node is data loss by automation. Where the engine supports it, prefer a synchronous or quorum-acknowledged candidate for promotion and let lag disqualify the rest. **Escalate rebuilds explicitly.** Once a replica passes the retention threshold, catch-up is off the table; the runbook is a re-clone. Automating that is reasonable only with rate limits, because re-cloning several replicas simultaneously loads the primary or the backup store precisely when things are already bad. ## Closing the loop Thresholds are only meaningful if the causes are addressed: chunk large write transactions to bound the spikes they create, keep index parity between primary and replicas so apply stays index-driven, and treat a sustained deficit as a capacity or sharding decision rather than an alert to be raised. An alert that fires nightly and is always resolved by waiting is a configuration bug, not monitoring.