On a Linux box running a pre-forking web server with 40 worker processes, adding up the RSS column from `ps aux` gives about 60 GB on a machine with 16 GB of RAM, and the machine is not swapping. Why can that sum exceed physical memory, and what would you measure instead?

## What RSS counts Resident set size is the number of pages of a process's virtual address space that are currently present in physical memory, converted to kibibytes for the `RSS` column of `ps` and the `RES` column of `top`. The definition is entirely process-centric: the kernel walks that process's page tables and counts resident mappings. It does not ask whether a physical page is also mapped by somebody else. That is fine when you look at one process. It falls apart the moment you add the column up, because on Linux a great deal of memory is legitimately mapped by many processes at once: - **Copy-on-write pages after fork.** `fork()` does not copy the parent's memory; it marks the pages read-only and shares them until one side writes. In a pre-forking server the parent may have loaded a large configuration, template cache, or interpreter heap before forking, and every worker reports all of it as resident. - **Shared libraries.** libc, OpenSSL, the language runtime — one physical copy in the page cache, mapped into every process, counted in every RSS. - **File-backed text pages.** The executable's own code, likewise shared. - **Explicitly shared memory.** `tmpfs` files, `shm` segments, `MAP_SHARED` mappings — one region, many mappers, counted once per mapper. So the 60 GB is not a measurement error, it is the wrong arithmetic on a correct measurement. ## PSS: the number that adds up Proportional set size solves exactly this. For each resident page, PSS adds `page_size / number_of_processes_mapping_it` to the process's total. A page private to one process contributes fully; a page shared by 40 workers contributes a fortieth to each. The consequence is the useful property: **the sum of PSS across all processes approximates real physical memory in use**, without double counting. The kernel exposes it per process in two places: ``` grep -H ^Pss: /proc/*/smaps_rollup 2>/dev/null | sort -t: -k3 -n | tail ``` `/proc/<pid>/smaps_rollup` gives one pre-summed set of counters for the whole address space, which is cheap to read. `/proc/<pid>/smaps` gives the same fields broken down per mapping, which is what you read when you want to know *which* region is large — but it is expensive on a process with thousands of mappings, so prefer the rollup for scanning. Alongside `Pss` you will find `Rss`, `Shared_Clean`, `Shared_Dirty`, `Private_Clean`, `Private_Dirty` and `Swap`. The private counters are what people mean by USS, the unique set size: the memory that would actually be returned to the system if you killed this one process. For a fleet of forked workers, private-dirty is the interesting number — it is the memory each worker has genuinely made its own by writing to shared pages. If private-dirty grows steadily per worker, you have a real per-worker leak; if it stays flat while RSS looks huge, the workers are just sharing the parent's image as designed. Reading these files for another user's processes requires privilege, so this is usually a `sudo` operation, and tools such as `smem` exist to do the walk and present PSS and USS in one table. ## Two more ways RSS misleads - **RSS omits swapped-out pages.** A process being swapped out shows a *falling* RSS, which reads like memory being released when the opposite is happening. The `Swap` line in the rollup is what tells you where those pages went. - **RSS says nothing about reclaimability.** Much of a large RSS can be clean, file-backed pages the kernel can drop instantly under pressure. A 4 GB RSS that is mostly clean page cache mappings is a very different situation from 4 GB of private dirty anonymous memory, and only the breakdown distinguishes them. ## How to answer this at the terminal The practical sequence is short. Confirm the shape with `ps`: ``` ps -eo pid,ppid,rss,args --sort=-rss | head -n 20 ``` If the large entries are siblings with a common parent and near-identical RSS, that pattern alone is the tell — forked workers sharing a parent image. Then read PSS from the rollup for a handful of them, and compare private-dirty across workers. If PSS per worker is a small fraction of its RSS, the sum was an illusion and the machine's memory accounting is fine. If private-dirty is large and climbing, you have found the actual consumer, and the question becomes what each worker is allocating for itself. The interviewer is testing one specific instinct here: whether you know that per-process memory columns are not additive, and whether you know the field that is.