Pipe Buffer Deadlocks

## The mechanism When you pass `stdout=subprocess.PIPE`, the `subprocess` module asks the kernel for a pipe: a fixed-size ring buffer with a write end handed to the child and a read end kept by the parent as `Popen.stdout`. "Fixed-size" is the whole story. On Linux the default capacity is 65,536 bytes; on other systems it is often smaller. A pipe never grows to fit its writer and never drops data to make room. When it is full, the writing process blocks inside the `write()` system call until a reader takes bytes out. That back-pressure is a feature — it is what stops a fast producer from exhausting memory — but it means the reader is a required participant. Now consider the parent. `Popen.wait` blocks until the child terminates. It reads nothing. So the sequence is: 1. The child writes; the first ~64 KiB lands in the pipe buffer. 2. The buffer fills; the child's next `write()` blocks. 3. The parent is inside `Popen.wait`, waiting for a child that is blocked in a write the parent will never service. Neither side has a timeout, so the program hangs until something kills it. This is the single most common `subprocess` bug in production Python, and the Python documentation calls it out explicitly in the `Popen.wait` entry. ## Why it survives testing The bug is data-dependent. A command that prints a version banner produces far less than the buffer capacity, so it exits, the pipe is closed with its contents intact, and the parent's later read succeeds. Every unit test passes. The hang appears the first time the child gets talkative — a verbose flag, a warning per input row, a stack trace. A job that quietly succeeded on a five-row sample stalls forever on a 6,800-row batch, and because the parent looks idle rather than crashed, monitoring often reports nothing at all: the worker simply stops taking new work while its supervisor still sees a live process. ## What Popen.communicate actually does `Popen.communicate` is not a convenience wrapper around `Popen.wait`; it is a different algorithm. It sends any `input` you pass to `Popen.stdin`, closes it, then reads `Popen.stdout` and `Popen.stderr` **concurrently** until both reach end-of-file, and finally calls `Popen.wait` to reap the child. On POSIX it drives all three descriptors through a selector so no stream can starve another; on Windows, where selectors do not work on pipes, it starts a reader thread per stream. Because it never sits in a blocking wait while a buffer is filling, the deadlock cannot occur. `subprocess.run` is a thin wrapper over `Popen` plus `communicate`, which is why the high-level call is the safe default. ## The variants that still bite - **Reading one stream while the other is a pipe.** `out = proc.stdout.read()` followed by `proc.wait()` looks careful, but if `stderr` is also `subprocess.PIPE` and the child writes heavily to `stderr`, the child blocks on `stderr` while you block reading `stdout`. Either merge them with `stderr=subprocess.STDOUT`, or use `communicate`, or run a reader per stream. - **Writing a large input before reading.** `proc.stdin.write(big)` can block once the child's own input pipe fills, if the child is not consuming because it is itself blocked writing to a full `stdout` pipe. `communicate(input=...)` interleaves both directions and avoids it. - **Iterating `Popen.stdout` and calling `wait()` inside the loop.** The loop stops feeding the pipe the moment it enters the wait. - **`Popen.poll` is not a fix.** It returns immediately instead of blocking, so a poll loop does not deadlock — but if the loop body does not read the pipes, it spins forever against a child that never exits. ## The line-streaming case Sometimes you genuinely want output as it arrives — progress lines, a log tail — and `communicate`, which returns only at end-of-file, is the wrong shape. Then iterate the stream and wait afterwards: ```python with subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True) as proc: for line in proc.stdout: handle(line) proc.wait() ``` The merge into one stream is what makes this safe: there is no second pipe to starve. If you need the two streams kept apart, give each one its own reader thread. ## What to say in an interview Name the bounded buffer, name both blocked parties, and name the one call that reads and waits together. Then add the detail that separates a middle answer from a senior one: the bug is invisible until the child's output crosses the buffer size, so "it worked in staging" is exactly the symptom you should expect.