A long-running Linux daemon starts logging "Too many open files" after a few days of uptime. Using `ps`, `lsof` and `/proc`, how do you confirm it is leaking file descriptors and find what it is holding open?

## The error names the resource "Too many open files" is the userspace text for `EMFILE`: the process tried to obtain a file descriptor and had hit its own ceiling. Descriptors are not just files — every open socket, pipe, epoll instance, timer, eventfd and inotify instance consumes one. The investigation is therefore about counting and classifying descriptors, and the whole thing is visible through `/proc`. ## Step 1: count what is actually open `/proc/<pid>/fd` is a directory of symlinks, one per open descriptor, named by descriptor number and pointing at what it refers to. Counting the entries is the authoritative count: ``` pid=$(pgrep -x mydaemon) ls /proc/$pid/fd | wc -l ``` A very common mistake is to use `lsof -p $pid | wc -l` for this. That number is larger, and not by a constant: `lsof` prints a header line, and its `FD` column includes entries that are not descriptors at all — `cwd` (current directory), `rtd` (root directory), `txt` (the executable's own text mapping) and one `mem` line for every memory-mapped library. On a process linked against a dozen shared libraries the inflation is significant. Use `/proc/<pid>/fd` when you want the count; use `lsof` when you want to know what the descriptors *are*. ## Step 2: find the ceiling that applies ``` grep 'Max open files' /proc/$pid/limits ``` This reads the soft and hard limits in force for that specific process. It matters that you read it from `/proc` rather than typing `ulimit -n` in your shell: your shell's limits are its own, and a daemon started by a service manager typically inherits limits from the manager's configuration, not from any login session. `prlimit --pid $pid` prints the same information, and can raise a running process's limit in place when you need to buy time before a restart. Compare the count with the soft limit. Sitting at 1021 out of 1024 is the smoking gun; sitting at 300 out of 65536 means the error came from somewhere else — including a **system-wide** ceiling rather than a per-process one, which is a different diagnosis and shows up in `/proc/sys/fs/file-nr` and `/proc/sys/fs/file-max`. ## Step 3: distinguish a leak from a load level One reading cannot tell you whether descriptors are leaking. Sample it: ``` while sleep 60; do printf '%s %s\n' "$(date +%T)" "$(ls /proc/$pid/fd | wc -l)"; done ``` - A **leak** rises monotonically and does not fall when traffic drops. Uptime correlates better with the count than request rate does. - A **sizing problem** rises and falls with load and plateaus below the limit — the process is behaving correctly and the limit is simply too low for the concurrency you are asking of it. That distinction decides whether you file a bug or change a configuration, so it is the part interviewers most want to hear. ## Step 4: classify what is being held Now `lsof` earns its keep: ``` sudo lsof -nP -p $pid ``` `-n` suppresses reverse DNS on socket addresses and `-P` suppresses port-name lookup; both are network round trips you do not want on a struggling host, and without them lsof can hang for a long time. Read the `TYPE` and `NAME` columns and look for a dominant group: - Hundreds of `IPv4`/`TCP` entries to the same peer with a state like `CLOSE_WAIT` — the application is not closing connections it has finished with, or a connection pool is unbounded. - The same regular file open many times — a handle reopened per request or per rotation. - Entries whose name ends in `(deleted)` — the file was unlinked but the descriptor keeps it alive, which also means its disk space is not returned. - `a_inode` entries backed by inotify or eventfd — a watcher or event loop that creates and never destroys. Because `/proc/<pid>/fd` symlinks are readable directly, `ls -l /proc/$pid/fd | awk '{print $NF}' | sort | uniq -c | sort -rn | head` gives the same grouping without lsof at all, which is useful on a minimal image where lsof is not installed. One access note: reading another user's `/proc/<pid>/fd`, or listing their files with `lsof`, needs privilege. An unprivileged `lsof` silently shows only what you own, which has fooled plenty of people into thinking a process holds almost nothing. ## What to do about it Raising the limit — in the service manager's unit configuration for a permanent change, or with `prlimit` on the running process for an emergency — stops the bleeding but does not fix a leak; a monotonic curve will simply take longer to reach the new ceiling. Restarting resets the count and destroys the evidence, so capture the `lsof` output *before* you restart. The durable fix is in the application: close what you open, bound the pool, and reuse handles rather than reopening them.