A Linux application server with swap enabled becomes extremely slow while its CPUs sit mostly idle and its swap usage climbs. What is the kernel doing, and why does adding more swap not fix a memory shortage?

## What reclaim actually does When free pages fall below the kernel's low watermark, reclaim starts. The kernel walks LRU lists and frees pages in two categories: - **File-backed pages** — page cache. A clean one is dropped for free; a dirty one is written back first. - **Anonymous pages** — heap, stack, and anything with no file behind it. These have nowhere to go except a swap device, so with no swap configured they are simply not reclaimable at all. So swap is not "extra RAM". It is a backing store that makes anonymous pages eligible for reclaim, exactly as a file makes file pages eligible. ## Thrashing, and why the CPUs look idle Reclaim is a bet that the evicted page will not be needed soon. When the working set — the set of pages actually being used in a given interval — fits in RAM, the bet is good. When it does not, the kernel evicts a page and the application touches it milliseconds later. That access is a **major fault**: the faulting task blocks in uninterruptible sleep while the kernel reads the page back, then something else must be evicted to make room. The system converts memory pressure into disk I/O at the worst possible place, on the critical path of every memory access. The signature is distinctive and worth naming in an interview: - throughput collapses, latency goes to seconds, but CPU utilisation is *low*, because the work is blocked, not computing; - swap-in traffic is as high as swap-out — steady-state swap-out alone is healthy, but pages coming back in mean the eviction was wrong (`pswpin`/`pswpout` in `/proc/vmstat` count them); - on kernels with `CONFIG_PSI` (4.20+), `/proc/pressure/memory` shows a large share of time stalled on memory — the cleanest single signal that this is memory pressure and not a slow database or a network problem. The same mechanism explains the classic "why is my swap full but the machine is fine" case: cold pages parked in swap and never read back cost nothing. **Swap usage is not the symptom; swap traffic is.** ## vm.swappiness, correctly stated `vm.swappiness` is routinely described as "how eager the kernel is to swap", which invites the belief that lowering it eliminates swapping. It is better described as the kernel's assumed *relative I/O cost of reclaiming anonymous memory versus file-backed memory*. It biases the split between the two LRU lists during reclaim. The default is 60. Raising it favours evicting anonymous pages and keeping cache; lowering it favours evicting cache and keeping anonymous pages resident. Setting it to 0 does not disable swap — it tells the kernel to avoid anonymous reclaim until it is nearly out of alternatives, which in practice means the machine will chew through its page cache first and then still swap rather than die. On fast NVMe storage, and especially with compressed swap in RAM (`zram`) or a compressed cache in front of the swap device (`zswap`), the cost assumption behind a low swappiness is often wrong, and modest swapping is genuinely cheap. ## Should a server have swap at all? Both extremes are defensible and the interviewer wants the reasoning: - **Swap off** makes behaviour crisp: the machine either has memory or the out-of-memory killer runs, with no long grey zone where everything is alive but unusably slow. It also removes anonymous pages from the reclaim toolkit entirely, so pressure escalates to a kill much sooner, and genuinely idle pages are stuck occupying RAM. - **Swap on** gives the kernel somewhere to put cold anonymous memory, which is a real win for machines running long-lived processes with large initialisation-time allocations, and it converts a cliff into a slope — often the difference between a degraded service you can shed load from and a dead one. A good middle answer is a modest swap device plus an explicit pressure alert, so degradation is detected long before it becomes a stall. What you should not do is size swap as if it were memory: if the working set exceeds RAM, more swap only lengthens the period of thrashing before the inevitable. ## Fixing it Shrink the working set (bound caches and per-request buffers, cap the runtime's heap), give the machine reclaimable headroom, move a tenant off the box, or add RAM. Tuning `vm.swappiness` redistributes the pain; it does not create memory.