A Linux daemon is restarted and fails to bind its listening TCP port with EADDRINUSE, even though the old daemon process is gone. A helper program that the old daemon had launched is still running. How can that be, and what should the daemon have done differently?
answer
- the socket outlives the process that made it
- descriptors are references, held by anyone
- fork copies the table, exec keeps it
- inheritance is the default, opt out
- not TIME_WAIT — genuinely still open
basics
~20 sThe helper inherited the listening socket. File descriptors survive fork and survive exec unless marked close-on-exec, and a socket stays bound while any process still holds it. The fix is to create such descriptors with O_CLOEXEC or SOCK_CLOEXEC, or set FD_CLOEXEC before exec.
solid answer
~50 sA socket is not owned by a process; it is a kernel object that processes hold references to through their descriptor tables. `fork()` duplicates that table wholesale, and `execve()` keeps every entry that is not flagged close-on-exec — which is precisely how a shell hands stdin and stdout to the program it runs. So when the daemon forked and exec'd its helper, the helper silently acquired a reference to the listening socket, and the port stays bound for as long as the helper lives, long after the daemon that created it died. `SO_REUSEADDR` does not rescue you: nothing here is in `TIME_WAIT`, the socket is simply still open. The correct discipline is to make descriptors close-on-exec at creation — `socket(..., SOCK_STREAM | SOCK_CLOEXEC, 0)`, `open(..., O_CLOEXEC)`, `accept4(..., SOCK_CLOEXEC)` — or set `FD_CLOEXEC` with `fcntl()` before spawning anything.
code
c · 15 lines#include <fcntl.h>
#include <sys/socket.h>
#include <unistd.h>
int main(void) {
/* Created close-on-exec: no exec'ed child can inherit it. */
int lfd = socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0);
/* Same effect for a descriptor you did not create yourself. */
int flags = fcntl(lfd, F_GETFD);
fcntl(lfd, F_SETFD, flags | FD_CLOEXEC);
close(lfd);
return 0;
}go deeper
Know that a child process inherits its parent's open file descriptors, that this survives running a new program unless the descriptor is marked close-on-exec, and that a port stays bound while any process holds the socket.
Explain the two steps — fork duplicates the descriptor table, exec preserves the entries without FD_CLOEXEC — and name the mechanisms that opt out: O_CLOEXEC, SOCK_CLOEXEC, accept4(), and fcntl with F_SETFD.
Reason from EADDRINUSE with no obvious holder to an inherited descriptor in an unrelated helper, distinguish that from a TIME_WAIT binding, and state the discipline: flag at creation to avoid the fork race, and know exactly what each child inherits.
Treat descriptor inheritance as a confinement boundary: an inherited handle keeps its access after a child drops privileges, so make close-on-exec-by-default and an explicit, auditable list of passed handles a platform rule for anything that spawns processes.
## Descriptors are references, not ownership A file descriptor is a small integer indexing a per-process table. Each entry points at an open file description in the kernel, which in turn refers to the underlying object — a file, a pipe, a socket. The object lives as long as *any* descriptor anywhere refers to it. "Closing the socket" in one process only drops that process's reference; the bound port is released when the last reference in the whole system goes away. This is the entire explanation for the symptom. The dead daemon's references are gone, but the helper it started still has one. ## How the helper got it Two inheritance steps, both by design: 1. **`fork()` copies the descriptor table.** The child gets the same numbered descriptors pointing at the same open file descriptions — shared, not copied: they share the same file offset and status flags. 2. **`execve()` replaces the program but keeps the table.** The address space, heap, threads and signal handlers of the old program are thrown away; the descriptor table survives, minus any entry flagged `FD_CLOEXEC`. That second rule is not an accident to be worked around — it is the mechanism that makes Unix work. When a shell runs `cmd > out.txt`, it forks, opens the file onto descriptor 1 in the child, and execs. The new program inherits a redirected stdout without knowing anything happened. Service managers pass listening sockets to services the same way. The cost of that generality is that inheritance is the *default*, and every descriptor you leave unflagged leaks into every program you spawn. ## What else survives, and what does not Across `execve()` on Linux, roughly: **Preserved:** the PID and PPID; the process group and session; the controlling terminal; open descriptors without `FD_CLOEXEC`; the current and root directories; `umask`; resource limits; real user and group IDs (the effective IDs can change if the new binary is set-user-ID); the pending signal set; the signal mask. **Discarded or reset:** the entire address space, so all memory mappings, heap data, locks and threads other than the calling one are gone; signal handlers, which necessarily reset to the default since their code no longer exists — though signals set to be *ignored* stay ignored; the alternate signal stack; any timers created for the old program's address space. That "handlers reset, ignores persist" asymmetry is a favourite follow-up, and it explains a real bug: exec a child while `SIGPIPE` is ignored and the new program inherits that disposition, quietly changing how it behaves on a broken pipe. ## The correct discipline Mark descriptors close-on-exec **at creation**, not later, because a later `fcntl()` leaves a window in which another thread can fork: ```c int fd = socket(AF_INET, SOCK_STREAM | SOCK_CLOEXEC, 0); int f = open(path, O_RDONLY | O_CLOEXEC); int c = accept4(lfd, NULL, NULL, SOCK_CLOEXEC); ``` For a descriptor you did not create, set the flag explicitly: ```c int flags = fcntl(fd, F_GETFD); fcntl(fd, F_SETFD, flags | FD_CLOEXEC); ``` And where you deliberately *want* a child to inherit something — stdin, stdout, a pipe you just made — simply leave that one unflagged, or clear the flag on the duplicate you pass down (`dup2()` does not copy `FD_CLOEXEC`, which is convenient here). As a backstop, `close_range(3, ~0U, 0)` (Linux 5.9 and later) closes everything above the standard three in one call between fork and exec, replacing the old loop over every possible descriptor number. ## Why SO_REUSEADDR is the wrong answer `EADDRINUSE` has two distinct causes, and candidates habitually merge them. One is a socket left in `TIME_WAIT` from a previous connection, which `SO_REUSEADDR` is designed for. The other — this case — is that the address is genuinely still bound by a live open socket. No socket option overrides a live binding; the holder has to let go. Diagnosing it means asking which process holds a reference to that socket in its descriptor table, and the answer is a helper nobody thought of as a network program at all. ## Why it matters beyond a stuck port A leaked descriptor is also a security problem. Inheriting an open log file lets a helper append to it; inheriting a connected socket lets it read or inject traffic; inheriting a directory descriptor can defeat a chroot. A leaked descriptor to a privileged resource survives even when the child drops privileges, because the access check happened at open time. "Which descriptors does this child receive?" is a question every process that spawns another should be able to answer exactly.
- Would setting SO_REUSEADDR have prevented this?No. That option addresses binding over an address left in TIME_WAIT by a previous connection. Here the address is held by a live, open socket that a running helper still references, and no socket option overrides a live binding. The port frees only when the last descriptor referring to that socket is closed — which means the helper must exit or close it.
- Why is O_CLOEXEC at creation better than calling fcntl() to set FD_CLOEXEC afterwards?Because of the window between the two calls. In a multithreaded process another thread can fork and exec during that gap, and the child inherits the descriptor exactly as if you had never set the flag. Creating the descriptor close-on-exec in the same syscall closes the race, which is why O_CLOEXEC, SOCK_CLOEXEC and accept4() exist.
- If descriptors survive exec, how does a shell manage to redirect a command's output to a file?It relies on that survival. The shell forks, and in the child it opens the file and uses dup2() to place it on descriptor 1 — dup2() deliberately does not carry FD_CLOEXEC over — then execs. The new program starts with stdout already pointing at the file and never learns that anything was rearranged.
- Beyond a stuck port, what is the risk of leaking descriptors into a child?It is a privilege leak. Access is checked when a file is opened, not when it is used, so an inherited descriptor keeps working after the child drops privileges. A helper can then append to a log it should not write, read or inject on a connected socket, or use a directory descriptor to escape a confinement its own credentials would forbid.
saying these in an interview costs you the question
- Blames TIME_WAIT and reaches for SO_REUSEADDR
- Assumes exec closes all inherited file descriptors
- Thinks a socket dies with the process that created it
- Believes dropping privileges revokes an already-open descriptor
- Sets FD_CLOEXEC after creation in a threaded program