skip to content

A Go proxy wraps os.File.Fd's number in os.NewFile, both files get closed, and bytes then land on the wrong connection — what happened and how do you confirm it?

level: seniorimportance: nice to knowfreq 22%

answer

  1. two owners, one kernel object
  2. adoption is not duplication
  3. the kernel reissues the lowest free number
  4. the race detector sees nothing here
  5. log the number at accept and at close

basics

~20 s

os.NewFile adopts the descriptor rather than duplicating it, so two owners close the same number. After the first close the kernel reissues that number to a new socket, and the second close hits the wrong connection.

solid answer

~40 s

`os.NewFile` takes ownership of the number you hand it; it does not duplicate. Wrapping `f.Fd()` therefore creates a second owner for one descriptor. The first `Close` frees the number, the kernel hands the lowest free number to the next `accept`, and the second `Close` tears down an unrelated connection, so one client's bytes go to another or a socket dies for no visible reason. The tell is intermittency under load and cross-talk between sessions. Confirm it by logging the descriptor number next to a connection id at accept and comparing with `lsof -p` or `/proc/<pid>/fd`: the same number appears under two identities. The race detector will not find it — nothing in Go memory is raced. Fix it by ownership: use `SyscallConn` instead of `Fd`, or duplicate explicitly, as `net.FileListener` and `(*net.TCPConn).File` already do.

code

go · 6 lines
go
// BUG: f still owns this descriptor number.
sock := os.NewFile(f.Fd(), "adopted")

f.Close()    // closes descriptor N
// ... the accept loop is handed N for a brand new connection ...
sock.Close() // closes N again - now somebody else's live socket

go deeper

for a junior

You are unlikely to face this scenario, but carry the rule it comes from: exactly one owner closes a descriptor, and learning its number does not make you that owner.

for a middle

Explain that os.NewFile adopts rather than duplicates, that the kernel reissues the lowest free number immediately, and how that turns a harmless-looking second close into a stranger's dead socket.

for a senior

Walk the diagnosis end to end: instrument the descriptor number at accept and at close, correlate with lsof or /proc/<pid>/fd, say plainly why the race detector is the wrong tool, and land on an ownership fix rather than a lock.

for a principal

Turn the incident into an invariant. Decide whether raw descriptors may appear in your APIs at all, make adoption and duplication explicit at package boundaries, and give reviewers a rule they can apply without re-deriving this failure.

## The bug ```go // BUG: two owners, one descriptor. sock := os.NewFile(f.Fd(), "adopted") // ... f is still live and still owns the same number ``` `os.NewFile` wraps an existing descriptor in an `*os.File` and **adopts** it: closing the returned file closes that descriptor. It does not dup. So after this line, two `*os.File` values each believe they will close descriptor N. Both eventually do — explicitly, or through the runtime's cleanup of whichever one goes unreferenced first. ## Why the second close is worse than a no-op On Unix, `close(N)` frees the number, and the kernel allocates the **lowest free descriptor** for the next `open`, `accept` or `dial`. In a busy proxy that is microseconds later. The sequence is: 1. `f.Close()` closes N. 2. The accept loop accepts a new connection; the kernel assigns it N. 3. `sock.Close()` runs and closes N — which is now somebody else's live connection. The new client is disconnected for no reason it can see. If any code still holds a wrapper around N and writes, one session's bytes are delivered into another session's socket, which is a correctness *and* a confidentiality problem. Symptoms are exactly the ones that make this hard: fine under test, intermittent under load, worse the more connections churn, and the stack trace at the point of failure points at innocent code. ## Confirming it Descriptor numbers are the only handle you have, so instrument them: - Log the descriptor alongside a connection id in the accept loop and again at every close. Borrow the number for logging with `SyscallConn`, or record it once at accept. - Snapshot `ls -l /proc/<pid>/fd` (or `lsof -p <pid>`) while the proxy runs and line it up with those log lines. The signature of this bug is one number appearing under two identities in your own log, or a number logged as closed reappearing under a different peer address in the very next snapshot. - Count opens and closes. A correct program closes each descriptor exactly once; a counter keyed by number that ever reaches two is the proof. - Do **not** reach for the race detector. It instruments Go memory accesses, and there is no data race here: the conflict is over a kernel object named by an integer. A clean `-race` run is not evidence. ## Fixing it The rule is one owner per descriptor, and every transfer is explicit. **Prefer not to have a number at all.** If the reason you called `Fd` was to set a socket option or call `fstat`, use `SyscallConn` and do it inside `RawConn.Control`. No number escapes, so no second owner can exist. **If you truly need a second owner, duplicate.** The standard library already does this for you at the boundaries where it matters. `(*net.TCPConn).File` and `(*net.TCPListener).File` return an `*os.File` holding a **duplicate** descriptor: closing one does not affect the other, and you are expected to close both, exactly once each. Symmetrically, `net.FileListener` and `net.FileConn` duplicate the descriptor of the file you pass in, so the file stays yours to close. Note the trade: taking `File` from a connection also puts that connection into blocking mode, so its deadlines stop working. **Adopt only what nobody else owns.** The legitimate use of `os.NewFile` is a descriptor that entered the process from outside — a socket a supervisor passed down, typically starting at descriptor 3, or an inherited pipe. Nothing else in the process owns it, so adoption is honest: ```go f := os.NewFile(3, "inherited-listener") ln, err := net.FileListener(f) // dups if err != nil { return err } f.Close() // safe: ln holds its own duplicate defer ln.Close() ``` ## The design change that prevents a recurrence Make ownership visible in types rather than in comments. Keep the raw number inside one package; expose an `io.Closer`, an `*os.File`, or a `syscall.Conn` and let the type say who closes. Ban `syscall.Close` on a number that came from a Go type that still exists. In review, treat any `Fd()` call whose result outlives the statement as a defect, the same way you would treat a returned pointer to a freed buffer in C — because that is precisely what it is.

  • Why won't the race detector catch this?
    Because there is no data race in Go memory. The detector instruments loads and stores to Go variables and reports unsynchronised access to the same address. Here every Go access is properly ordered; the conflict is over a kernel file-descriptor slot identified by an integer, which the detector knows nothing about. You need descriptor accounting, an lsof or /proc listing, or a type that owns the close.
  • Does (*net.TCPConn).File return the same descriptor as the connection?
    No, it returns an *os.File holding a duplicate. Closing the connection does not affect the file and closing the file does not affect the connection, so both must be closed exactly once. The trade-off is that taking File puts the connection into blocking mode, after which its deadline methods no longer work.
  • A supervisor passes your process a listening socket as descriptor 3. How do you adopt it safely?
    Wrap it once with os.NewFile(3, "listener"), then pass that to net.FileListener, which duplicates the descriptor and gives you a net.Listener. Close the *os.File once the listener exists, and close the listener separately at shutdown. The critical invariant is that only one place in the process ever wraps descriptor 3.
  • How would you stop this class of bug recurring in the codebase?
    Keep raw descriptor numbers inside a single package and export types that carry ownership, such as an *os.File or a syscall.Conn, so the compiler and the reader both know who closes. Treat any Fd result that outlives its statement as a review defect, and require an explicit dup whenever a descriptor genuinely needs a second owner.

Two people are each told they are the one who returns the rental car. The first returns it, the lot rents it to someone new, and the second walks up and drives away a car that now belongs to a stranger.

saying these in an interview costs you the question

  • Blames a data race and reaches for the race detector
  • Says closing a descriptor twice is harmless
  • Assumes os.NewFile duplicates the descriptor
  • Claims descriptor numbers are never reused
  • Adds a mutex around Close instead of fixing ownership