After switching a Redis instance to 'appendfsync always', p99 latency for all clients jumps and occasionally the whole instance appears to freeze for tens of milliseconds. How do you confirm the AOF sync path is the cause, and what are your options?

## The mechanism you are diagnosing Under `appendfsync always`, each event-loop cycle that produced writes performs the sequence: `write()` the AOF buffer to the OS, then `fsync()` the file, and only then send the replies for those commands. Because Redis executes commands one at a time, the fsync is not "the writer's problem" — it occupies the same execution path that every other client's command must pass through. A 3 ms fsync repeated hundreds of times a second is hundreds of milliseconds per second of the instance doing nothing but waiting on the device. This is why the symptom is *global*: read-only clients that never wrote anything see their p99 rise, which is often what confuses the initial investigation. ## Step 1 — establish that the stall is not command execution The first surprising observation is that the slow-command log is usually clean. That log records time spent *executing* a command; the AOF write and fsync happen in the surrounding event-loop cycle. So "latency is terrible but nothing is slow" is itself a strong clue pointing at the I/O path (or at another out-of-band stall). ## Step 2 — attribute the stall with the latency monitor Redis has a built-in latency monitor with named events. Enable it and look: ``` CONFIG SET latency-monitor-threshold 20 # ms LATENCY LATEST LATENCY HISTORY aof-fsync-always LATENCY DOCTOR ``` The events that matter here are `aof-fsync-always` (time spent in the synchronous fsync), `aof-write` (the write syscall itself), `aof-write-pending-fsync` and `aof-write-active-child` (write slowed because a sync or a background child was in progress), and `aof-rewrite-diff-write`. `LATENCY LATEST` gives the last and maximum durations per event, which is usually enough to close the question in one command. `LATENCY DOCTOR` will even name the likely cause in prose. ## Step 3 — corroborate with INFO and the OS `INFO persistence` supplies `aof_last_write_status` (an `err` here means writes are failing, at which point Redis starts refusing writes — a different and worse incident), `aof_pending_fsync`, and `aof_delayed_fsync`. `INFO stats` gives `instantaneous_ops_per_sec` to correlate stalls with load; `latency-tracking` plus `LATENCY HISTOGRAM` provides per-command percentile histograms so you can show that *all* commands, not one family, degraded at once. At the OS level, look at the AOF device's average service time and utilisation, and at whether the volume is a shared/network device subject to throttling or burst-credit exhaustion. Instances that behaved fine for days and then degraded are a classic credit-exhaustion signature. ## Step 4 — the options, with their trade-offs 1. **Faster storage.** Local NVMe rather than a network volume is the single most effective change. If the AOF and the snapshot output share a device, separate them so a background child is not competing with the sync path. 2. **Move to `everysec`.** This takes the fsync off the serving path entirely (a background thread does it) at the cost of about one second of exposure on a machine crash. For the overwhelming majority of workloads this is the right answer, and it is the default for exactly this reason. 3. **Keep `always` but shrink what it must protect.** If only a small subset of the data genuinely needs per-write durability, put that subset on its own low-throughput instance running `always`, and let the high-throughput instance run `everysec`. Durability policy becomes a per-dataset decision rather than a per-process one. 4. **Relocate the guarantee.** Node-level fsync protects one machine; asynchronous replication means a failover can still lose acknowledged writes. If the real requirement is "this write must not vanish", replica acknowledgement (for instance, requiring propagation to N replicas before treating the write as committed) or an upstream durable system of record addresses it more honestly than any fsync setting — and often lets you go back to `everysec`. 5. **Amortise with pipelining.** One fsync covers everything written in that cycle, so a client that pipelines batches of writes pays far fewer syncs per write than one issuing them singly. Where the client can batch, throughput under `always` improves substantially without changing durability. ## What not to do Do not set `no-appendfsync-on-rewrite yes` as a fix for this. It will smooth the graph during rewrites, but it does so by suspending durability for the whole rewrite window — which is exactly the property you switched to `always` to obtain. Choosing latency is legitimate; choosing it silently while still claiming per-write durability is not. ## How to present it in an interview Walk the chain: the mechanism (fsync in the serving path, serial command processing, so global impact), the evidence (latency monitor `aof-fsync-always`, INFO persistence, device stats, and the notable absence of slow commands), and the decision (faster disk, `everysec`, split the dataset, or move the guarantee to replicas) with the durability consequence of each stated out loud.