skip to content

Why can open() without a with block leak a file descriptor in Python?

level: juniorimportance: must knowfreq 70%

answer

  1. Two lifetimes, one object
  2. The kernel does not count references
  3. Tracebacks and cycles keep locals alive
  4. Refcount closing is CPython-only
  5. Tie the close to a block

basics

~20 s

A file object holds a kernel descriptor until something closes it. CPython usually closes it when the last reference disappears, but that is a refcounting implementation detail. A with block closes deterministically, even when the body raises.

solid answer

~50 s

`open()` returns a Python object wrapping an OS file descriptor, and the descriptor is released only when `close()` runs. In CPython the object's finalizer closes it as soon as the last reference goes away, which is why careless scripts usually get away with it — but that is a CPython implementation detail, not a language guarantee. The reference can outlive the block: a live traceback keeps the frame and its locals alive, a reference cycle defers the object to the cyclic collector, and a cache or another implementation may hold it far longer. Meanwhile the process has a hard ceiling on open descriptors, and crossing it makes the *next* `open()`, socket or subprocess launch fail with `OSError` — *Too many open files*. `with open(path) as f:` ties the close to the block, and `contextlib.ExitStack` does the same when the number of files is dynamic.

code

python · 17 lines
python
import pathlib
import tempfile

p = pathlib.Path(tempfile.mkdtemp()) / "data.txt"
p.write_text("first\nsecond\n")


def leaky():
    return open(p).readline()  # closed only when CPython drops the last reference


def safe():
    with open(p) as f:  # closed on the way out, exception or not
        return f.readline()


print(leaky().strip(), safe().strip())

go deeper

for a junior

Be ready to say what open() returns, why the with statement exists, and that the descriptor is a kernel resource separate from the Python object. Writing with open(...) as f by reflex is the expected habit here.

for a middle

Explain the mechanics: the finalizer closes the descriptor when the refcount hits zero in CPython, and name the cases where that is delayed - live tracebacks, reference cycles, caches, other implementations. Know contextlib.ExitStack for a dynamic set of files.

for a senior

Connect it to production: a per-request leak exhausts the descriptor ceiling in hours, the failure lands on unrelated code as OSError, and memory graphs stay flat. Show how you would detect it before the outage rather than after.

for a principal

Own the policy angle: make resource ownership explicit in APIs your teams build, so an object that opens a descriptor either closes it or is a context manager, and put the detection switches into the standard test and CI configuration rather than into folklore.

### A descriptor is a kernel resource, not a Python object When you call `open(path)`, two different things come into existence. The kernel creates an entry in the process's descriptor table and hands back a small integer — the *file descriptor*. CPython wraps that integer in a Python file object (a `_io.TextIOWrapper` over a buffered binary layer). The Python object lives on the interpreter heap and is managed by reference counting and the cyclic garbage collector. The descriptor lives in the kernel and is released only when something calls `close(2)` on it. The two lifetimes are connected by exactly one thing: the file object's `close()` method, which CPython also calls from the object's finalizer. That is why "the object will be collected eventually" is not a safe answer. The kernel does not care about your refcounts. It cares about descriptors, and the process has a fixed ceiling on how many it may hold open at once. Cross that ceiling and every subsequent `open()`, `socket()` or `subprocess` launch raises `OSError` with `errno.EMFILE` — the message reads *Too many open files*. The failure lands wherever the next open happens to be, which is almost never the code that leaked. ### Why refcounting is not a guarantee In CPython, dropping the last reference to a file object deallocates it immediately, and the finalizer closes the descriptor. That behaviour is real, it is the reason sloppy scripts usually work, and it is an **implementation detail of CPython**, not a rule of the language. It stops holding in several ordinary situations: * **An exception keeps the frame alive.** A raised exception carries a traceback, the traceback references the frame, and the frame references every local — including your file object. While the exception is being handled, logged or stored, the descriptor stays open. * **Reference cycles.** If the file object is reachable only from a cycle, refcounting cannot free it; it waits for the cyclic collector, which runs at an unpredictable moment. * **Something kept a reference on purpose.** A cache, a registry, a closure or a list of "open handles" holds the object and therefore the descriptor. * **Another implementation.** Runtimes that use a tracing collector rather than reference counting close the file whenever they get around to it, which can be much later. * **Long-lived scopes.** A file opened at module level, or bound to a local in a long-running loop body that never rebinds, simply stays open. ### The fix is deterministic closing `with open(path) as f:` calls the file object's `__exit__` on the way out of the block — on the normal path, on `return`, on `break`, and when an exception propagates. That is the whole point of the context-manager protocol: the release is tied to a lexical scope rather than to when the collector notices. The same applies to `socket.socket`, to the pipe objects on a `subprocess.Popen`, to `zipfile.ZipFile`, `tempfile.NamedTemporaryFile` and every other descriptor-owning stdlib object. Two supporting tools cover the shapes `with` alone does not: * `contextlib.ExitStack` when the number of resources is dynamic — open them in a loop, register each with `enter_context`, and the stack unwinds them all when the block ends. * `contextlib.closing` for an object that has `close()` but no context-manager protocol. Where a resource genuinely must outlive a block — a connection pool, a long-lived listening socket — the object that owns it becomes responsible for closing it, ideally by being a context manager itself, so ownership stays explicit rather than accidental. ### How the interpreter tells you that you got it wrong When a file object is deallocated while still open, CPython emits a `ResourceWarning` naming the object. It is silent by default because `ResourceWarning` is in the default ignore filters, so a leak is invisible until the descriptor ceiling is hit. Turning warnings on for test and development runs is what converts a mystery outage into a stack trace, and it is the reason "we never saw a warning" is not evidence of correctness. ### Scale of the problem A single leaked descriptor per request in a service handling a few requests a second exhausts a default ceiling in minutes to hours — which is exactly the profile of the classic bug report: *it runs fine for an hour, then everything fails at once*. Memory looks flat, CPU looks flat, and the only rising number is one nobody was graphing. Writing `with` at every open is a one-character-per- line habit that removes the entire class.

  • If CPython closes the file when the last reference goes, when does that actually fail to happen?
    Whenever something still references the object. A live exception traceback holds the frame and every local in it; a reference cycle defers the object to the cyclic collector; a cache, registry or closure holds it deliberately. On a runtime that uses tracing collection instead of reference counting, the delay is unbounded. Each of these keeps a kernel descriptor open for an arbitrary period, which under load is enough to exhaust the process ceiling.
  • You need a file open for the lifetime of a worker object rather than one block. How do you keep that safe?
    Make ownership explicit: the object that opens the descriptor also closes it, and exposes that through the context-manager protocol so callers can write `with Worker(...) as w:`. Failing that, provide a `close()` method and call it from a `finally` in the caller. What you avoid is an object that opens a descriptor and leaves closing to whoever happens to drop the last reference.
  • What error do you actually see when a process runs out of descriptors?
    `OSError` with `errno.EMFILE`, printed as *Too many open files*. It surfaces at the next attempt to open anything — a file, a socket, a subprocess pipe — so the traceback usually points at innocent code far from the leak. That mismatch between the reported line and the guilty line is what makes the bug feel mysterious.

The Python object is the library card; the descriptor is the book. Losing track of the card does not put the book back on the shelf, and the library has a limit on how many you may hold at once.

saying these in an interview costs you the question

  • Says the garbage collector always closes files promptly
  • Treats CPython refcounting as a language guarantee
  • Claims files close automatically at end of function scope
  • Thinks an unclosed file only wastes memory, not descriptors
  • Believes an exception path cannot hold a file object alive
  • Suggests calling gc.collect() instead of closing

context

resolving links…