A team documents that their Redis instance can lose at most one second of writes because it runs with 'appendfsync everysec'. Under what circumstances is that claim wrong, and how would you detect it on a running instance?

## Where the "one second" number comes from With `appendfsync everysec`, Redis writes buffered AOF data to the OS with `write()` on each event-loop cycle, and a **background thread** issues `fsync()` roughly once per second. The nominal exposure to a machine crash is therefore the data written since the last completed fsync — about one second. That is the number everybody quotes, and under healthy conditions it is right. ## Failure mode 1: the disk cannot keep up An fsync is not instantaneous. On network-attached or heavily contended storage it can take hundreds of milliseconds, occasionally seconds. Redis has to decide what to do when it is time to flush the buffer with `write()` but the previous `fsync()` has not finished. Blocking on `write()` would be catastrophic: the call can stall behind the in-flight sync, and because commands are processed serially, every client would stall with it. So Redis takes the other option: it **postpones the write**, keeps the data in its own buffer, and carries on serving. It records the moment it started postponing. If the fsync still has not finished after roughly **two seconds**, Redis gives up waiting and performs the write anyway — at which point it may genuinely block. Two consequences follow, and both contradict the naive claim: - **The durability window widens.** Data that has not even reached the page cache certainly has not been fsynced, so the crash exposure grows to roughly two seconds rather than one. - **Latency spikes appear from nowhere.** A command completes in microseconds but the reply is delayed because the event-loop cycle stalled on a write that finally went through. Note that this stall happens *outside* command execution, so the slow-command log does not attribute it to any command — a classic "Redis is slow but nothing shows up as slow" investigation. ## Failure mode 2: rewrites suppress fsync entirely A background AOF rewrite is a second process writing a fresh copy of the dataset to the same disk. On spinning disks and on contended cloud volumes, that competition can make the parent's fsyncs pathologically slow. The escape valve is `no-appendfsync-on-rewrite`: - `no` (default): keep fsyncing per policy during the rewrite. Durability holds; latency may suffer. - `yes`: **do not fsync at all** while a rewrite (or a snapshot child) is running. Latency stays smooth; for that entire window the instance is effectively running `appendfsync no`, with a loss exposure of however long the kernel takes to flush — tens of seconds. Many "tuned for latency" configurations set this to `yes` and then describe the instance as "one second durable", which is only true when no rewrite is in progress. If rewrites happen often — high write volume, aggressive growth thresholds — the unprotected fraction of the day can be substantial. ## Failure mode 3: the device lies fsync's guarantee is only as strong as the storage stack honouring it. Virtual disks with volatile write-back caches, some container storage layers, and misconfigured RAID controllers can acknowledge an fsync before data is on stable media. Redis has no way to detect this; it is a platform property you verify out of band. Say so explicitly when someone asks for a hard durability number. ## Detecting the problem on a live instance `INFO persistence` is the primary source: - **`aof_delayed_fsync`** — a cumulative counter of how many times Redis postponed a write because an fsync was still running. Zero is healthy. A number that grows during traffic peaks is direct proof the disk is behind the policy. - **`aof_pending_fsync`** — fsyncs currently outstanding. - **`aof_last_write_status`** — `err` means writes to the AOF are failing outright, which is an incident in itself (Redis will start rejecting writes). - **`aof_rewrite_in_progress`** — combine with `no-appendfsync-on-rewrite yes` to know when durability is suspended. The latency monitor (`CONFIG SET latency-monitor-threshold <ms>`, then `LATENCY LATEST` / `LATENCY HISTORY`) reports dedicated events such as `aof-write`, `aof-write-pending-fsync`, `aof-write-active-child` and `aof-fsync-always`, which distinguish "slow because a child is running" from "slow because a sync is pending". At the OS level, elevated `%util` and await times on the AOF's device, or a shared volume's IOPS credits being exhausted, corroborate the diagnosis. ## Remediation 1. **Give the AOF faster or dedicated storage** — local NVMe rather than a shared network volume, or at least separate the AOF device from the snapshot device. 2. **Reduce rewrite pressure** so the competing child runs less often. 3. **Decide the trade honestly**: keep `no-appendfsync-on-rewrite no` and accept latency during rewrites, or set it to `yes` and document that durability is suspended for that window. 4. **Move the guarantee elsewhere** if the number really matters — an upstream durable system of record, or acknowledgement from replicas — rather than asking a single node's fsync to carry it. ## The answer in one breath "Everysec means one second only when the disk can finish an fsync in under a second. Redis postpones writes while a sync is in flight, up to about two seconds, so the real window is up to ~2s — and with `no-appendfsync-on-rewrite yes` there is no fsync at all during a rewrite. `aof_delayed_fsync` in INFO persistence is the tell."