What causes replication lag between a leader and its followers, and how would you design a system to guarantee 'read-your-writes' consistency for a user despite that lag?

## What replication lag is **Replication lag** is the time gap between the moment the leader durably commits a write and the moment a specific follower has fetched, applied, and made that write visible to its own readers. It is not a fixed constant - it fluctuates based on several concrete factors. - **Under normal load**, lag might be single-digit milliseconds, driven purely by network round-trip time between leader and follower. - **Under a write burst** (e.g., a batch job or flash-sale traffic), the leader can produce log entries faster than a follower's apply thread can consume and index them, so the follower's queue backs up and lag grows into seconds or minutes. - **When a follower is resource-starved** - doing a backup, running a heavy analytical query, rebuilding an index, or simply running on slower hardware than the leader - lag also grows, because applying incoming writes competes with that other work for CPU, disk I/O, and lock contention. - **Network partitions or packet loss** between leader and follower can pause replication entirely, in which case lag grows unbounded until the connection recovers. ## Why the lag is unavoidable This lag exists as an unavoidable consequence of physics and engineering trade-offs: making every follower synchronously caught up on every write (eliminating lag entirely) would mean synchronous replication to all followers, which collapses write availability and inflates write latency to the speed of the slowest replica. So virtually every system that reads from followers for scale accepts some amount of lag as the price of that scale-out. ## The read-your-writes problem it forces The trade-off it forces onto application design is exactly the **read-your-writes** problem: a user submits a write to the leader, gets a success response, and then immediately issues a read (e.g., reloading the page after posting a comment) that a load balancer routes to a follower which hasn't yet applied that write. From the user's point of view their own action appears to have silently failed or reverted, which is confusing and erodes trust even though no data was actually lost - it's purely a visibility problem caused by reading a stale replica. ## Techniques that guarantee read-your-writes There are several concrete techniques to guarantee read-your-writes despite lag, each with its own cost. 1. **The simplest** is to route any read that could plausibly be 'my own recent write' back to the leader - for example, always read a user's own profile/posts from the leader, while reading other users' content from replicas; this sacrifices some read scaling on that hot path but is simple to reason about. 2. **A more general version is time-based:** after a user writes, pin their reads to the leader (or force a read from a replica no more than N seconds behind) for some window, say one minute, then fall back to any replica once the risk of hitting stale content is low. 3. **A more precise version tracks log position:** the leader returns a log sequence number (`LSN`) or similar token with the write acknowledgment, the client (or a session store) remembers 'you're entitled to read at least up to position X,' and a router only sends the subsequent read to a follower whose applied position is at or past X, blocking briefly or falling back to the leader if no follower qualifies yet. This is more precise (it doesn't over-penalize users whose replica actually caught up quickly) but requires the follower's replication position to be queryable and the routing layer to track it per-request. ## Failure modes Failure modes to watch for in production: - **Naively pinning 'read from leader after every write'** for all users, all the time, defeats the entire purpose of having followers and just re-funnels all read traffic back to the leader, causing the exact write-throughput/read-scaling bottleneck replication was meant to solve. - **Under-provisioning the 'stickiness window'** (too short a TTL, or too coarse a time-based rule) lets stale reads leak through during traffic spikes when lag is worst - precisely when the naive fix is least effective, because lag is largest exactly when load is highest. - **Systems that use device/session affinity to a single follower** rather than log-position tracking can also produce the related but distinct monotonic-reads anomaly, where a second read from a different, more-lagged follower appears to go backwards in time relative to the first. ## Where it shows up A concrete real-world example: many web applications backed by MySQL or PostgreSQL read replicas implement 'read from primary after write' by writing to a session cookie or a fast key-value store the timestamp or LSN of the user's last write, and having the request router consult that value before choosing a replica; distributed SQL systems like Amazon Aurora expose a read-after-write consistency feature that does exactly this token-based routing under the hood, so application code just calls a consistent-read API without manually plumbing LSNs through every request.