Why does writing 200 MB to cmd.StdinPipe() before reading cmd.StdoutPipe() deadlock when piping a dataset through an external compressor?

seniorshould knowfreq 35%

answer

pipes are not unbounded queues
roughly 64 KB, then the writer blocks
a blocked writer has stopped reading
both processes are parked in a write
make the write and the drain concurrent

basics

~10 s

The child's output pipe fills after roughly 64 KB, so the compressor blocks writing output and stops reading input; the parent then blocks writing input. Feed stdin on one goroutine while another drains stdout.

solid answer

~50 s

A pipe is a small fixed-size kernel buffer — typically about 64 KB on Linux — not an unbounded queue. The compressor starts emitting bytes almost immediately, nobody is reading them, and once that buffer fills the child blocks inside its own write to standard output. A blocked child stops draining its standard input, so that pipe fills too, and the parent blocks inside io.Copy to cmd.StdinPipe(). Neither side can move and cmd.Wait is never even reached. The fix is to make writing and reading concurrent: one goroutine writes the payload and closes stdin, while another reads cmd.StdoutPipe() to end of file, and only then do you call cmd.Wait. The simpler variant is to assign a reader to cmd.Stdin and a writer to cmd.Stdout and let os/exec run both copies itself. Payload size is what makes it reproducible — below the buffer size the same code appears to work.

A pipe is a small fixed-size kernel buffer — typically about 64 KB on Linux — not an unbounded queue. The compressor starts emitting bytes almost immediately, nobody is reading them, and once that buffer fills the child blocks inside its own write to standard output. A blocked child stops draining its standard input, so that pipe fills too, and the parent blocks inside `io.Copy` to `cmd.StdinPipe()`. Neither side can move and `cmd.Wait` is never even reached. The fix is to make writing and reading concurrent: one goroutine writes the payload and closes stdin, while another reads `cmd.StdoutPipe()` to end of file, and only then do you call `cmd.Wait`. The simpler variant is to assign a reader to `cmd.Stdin` and a writer to `cmd.Stdout` and let os/exec run both copies itself. Payload size is what makes it reproducible — below the buffer size the same code appears to work.

code

go · 10 lines

stdin, _ := cmd.StdinPipe()
stdout, _ := cmd.StdoutPipe()
cmd.Start()

// blocks once the child's output pipe fills and it stops reading input
io.Copy(stdin, payload)
stdin.Close()

compressed, _ := io.ReadAll(stdout) // never reached
cmd.Wait()

go deeper

for a junior

Understand that a pipe holds only a small fixed amount of data and that a writer blocks once it is full, so a program that fills a pipe nobody reads simply stops there.

for a middle

Explain the cycle end to end: the child blocks writing its output, therefore stops reading its input, therefore the parent blocks writing input, and cmd.Wait is never reached.

for a senior

Be ready to diagnose it from a goroutine dump, explain why it reproduces only above a size threshold, and restructure the code so writing and draining run concurrently with errors from both sides reported.

for a principal

Take a position on how the organisation prevents this class of hang: a shared exec helper that owns the concurrency, a mandatory timeout on every child, or a rule that large payloads move through files rather than pipes.

The mental model that causes the bug

The code reads like a sequence of steps, so it gets written like one:

write the whole payload to the child's standard input;
close standard input;
read all of the child's standard output;
call cmd.Wait.

That is correct only if step 1 can complete on its own. It cannot, because the child is not a function you call — it is a peer running at the same time, and the two pipes between you are tiny.

The mechanism

An operating-system pipe is a fixed-size kernel ring buffer. On Linux the default is about 64 KB. A write that would exceed the free space blocks until a reader takes bytes out; a read on an empty pipe blocks until a writer puts some in. This is back-pressure, and it is the whole point of pipes — but it is also what closes the cycle here.

A compressor is a streaming filter: it reads input and emits compressed output as it goes, without waiting for the end of its input. So while your parent is still in step 1:

the child writes compressed bytes to its standard output;
nobody is reading that pipe, because your program is still in step 1;
after about 64 KB, the child's write blocks;
a child blocked in a write is not reading anything, so it stops consuming standard input;
your standard-input pipe fills, and your io.Copy blocks on its next write.

Both processes are now parked in a write, each waiting for the other to read. Nothing times out. The job simply stops, using no CPU, with the child still alive in the process table.

Why it only happens with real data

If the child's total output fits inside the pipe buffer, the child never blocks, so it keeps consuming input, so the parent never blocks either, and the sequential code completes. That is why this defect has the signature of an environment problem: it passes on a laptop against a small fixture and hangs in CI against the real dataset. The threshold is the size of the child's output, not its input, which makes the correlation with input size look approximate and confusing.

Confirming it rather than guessing

Let the hang be observed rather than reasoned about. In a test, go test has a -timeout; when it fires, it panics and dumps every goroutine's stack. You are looking for two things: a goroutine parked in a write to the child's standard input, down through the file and poller internals, and the absence of any goroutine reading the child's standard output. A blocked write with no corresponding reader is the fingerprint of a full-pipe deadlock; contrast that with a slow child, where the parent would be parked in a read. Outside a test, sending SIGQUIT to the stuck binary produces the same dump.

The fix

Make the write and the read concurrent.

stdin, err := cmd.StdinPipe()
// ...
stdout, err := cmd.StdoutPipe()
// ...
if err := cmd.Start(); err != nil {
	return err
}
go func() {
	defer stdin.Close() // the child needs end of file to finish
	io.Copy(stdin, payload)
}()
compressed, err := io.ReadAll(stdout) // drains while the writer runs
if err != nil {
	return err
}
return cmd.Wait()

Now the child is never blocked for long: whatever it emits is taken out of the pipe immediately, so it keeps consuming input, so the writing goroutine keeps making progress. Note the two rules from elsewhere in this area still apply — stdin must be closed for the child to finish, and the read must complete before cmd.Wait.

The error from the copying goroutine deserves a channel rather than the floor; a write that fails because the child died should be reported, not swallowed.

The simpler fix

Often you do not need the pipes at all:

cmd.Stdin = bytes.NewReader(payload)
var out bytes.Buffer
cmd.Stdout = &out

os/exec creates both pipes, runs both copies on its own goroutines, closes the input pipe at end of file, and cmd.Wait joins the copies. The concurrency is still there — you have simply delegated it, and there is no ordering rule left for you to break. The cost is memory: the whole output is held in the buffer, so for genuinely large results stream to a file or a bounded writer instead.

The general rule

Any time both directions of a child's I/O are live at once, they must be serviced concurrently. A pipeline through an external process is not a function call, and treating it as one works right up to the size where a kernel buffer fills.

## The mental model that causes the bug The code reads like a sequence of steps, so it gets written like one: 1. write the whole payload to the child's standard input; 2. close standard input; 3. read all of the child's standard output; 4. call `cmd.Wait`. That is correct only if step 1 can complete on its own. It cannot, because the child is not a function you call — it is a peer running at the same time, and the two pipes between you are tiny. ## The mechanism An operating-system pipe is a fixed-size kernel ring buffer. On Linux the default is about 64 KB. A write that would exceed the free space blocks until a reader takes bytes out; a read on an empty pipe blocks until a writer puts some in. This is back-pressure, and it is the whole point of pipes — but it is also what closes the cycle here. A compressor is a streaming filter: it reads input and emits compressed output as it goes, without waiting for the end of its input. So while your parent is still in step 1: - the child writes compressed bytes to its standard output; - nobody is reading that pipe, because your program is still in step 1; - after about 64 KB, the child's write blocks; - a child blocked in a write is not reading anything, so it stops consuming standard input; - your standard-input pipe fills, and your `io.Copy` blocks on its next write. Both processes are now parked in a write, each waiting for the other to read. Nothing times out. The job simply stops, using no CPU, with the child still alive in the process table. ## Why it only happens with real data If the child's total output fits inside the pipe buffer, the child never blocks, so it keeps consuming input, so the parent never blocks either, and the sequential code completes. That is why this defect has the signature of an environment problem: it passes on a laptop against a small fixture and hangs in CI against the real dataset. The threshold is the size of the child's *output*, not its input, which makes the correlation with input size look approximate and confusing. ## Confirming it rather than guessing Let the hang be observed rather than reasoned about. In a test, `go test` has a `-timeout`; when it fires, it panics and dumps every goroutine's stack. You are looking for two things: a goroutine parked in a write to the child's standard input, down through the file and poller internals, and the absence of any goroutine reading the child's standard output. A blocked write with no corresponding reader is the fingerprint of a full-pipe deadlock; contrast that with a slow child, where the parent would be parked in a *read*. Outside a test, sending SIGQUIT to the stuck binary produces the same dump. ## The fix Make the write and the read concurrent. ```go stdin, err := cmd.StdinPipe() // ... stdout, err := cmd.StdoutPipe() // ... if err := cmd.Start(); err != nil { return err } go func() { defer stdin.Close() // the child needs end of file to finish io.Copy(stdin, payload) }() compressed, err := io.ReadAll(stdout) // drains while the writer runs if err != nil { return err } return cmd.Wait() ``` Now the child is never blocked for long: whatever it emits is taken out of the pipe immediately, so it keeps consuming input, so the writing goroutine keeps making progress. Note the two rules from elsewhere in this area still apply — stdin must be closed for the child to finish, and the read must complete before `cmd.Wait`. The error from the copying goroutine deserves a channel rather than the floor; a write that fails because the child died should be reported, not swallowed. ## The simpler fix Often you do not need the pipes at all: ```go cmd.Stdin = bytes.NewReader(payload) var out bytes.Buffer cmd.Stdout = &out ``` os/exec creates both pipes, runs both copies on its own goroutines, closes the input pipe at end of file, and `cmd.Wait` joins the copies. The concurrency is still there — you have simply delegated it, and there is no ordering rule left for you to break. The cost is memory: the whole output is held in the buffer, so for genuinely large results stream to a file or a bounded writer instead. ## The general rule Any time both directions of a child's I/O are live at once, they must be serviced concurrently. A pipeline through an external process is not a function call, and treating it as one works right up to the size where a kernel buffer fills.

The job hangs in CI but never on a developer laptop. Why the difference?
The data, not the machine. The local fixture is small enough that the child's entire output fits in the pipe buffer, so nothing ever blocks; the real dataset pushes past it. Any deadlock whose trigger is crossing a fixed buffer threshold looks environment-specific until you reproduce it with a payload larger than the pipe.
How do you confirm the diagnosis from a hung test rather than guessing?
Let go test's -timeout fire: it panics and dumps every goroutine's stack. You want to see a goroutine parked in a write to the child's standard input and no goroutine reading its standard output. A blocked write with no reader is a full-pipe deadlock; a parent parked in a read would instead mean a slow or stuck child.
Would a larger buffer, say a bufio.Writer around the stdin pipe, avoid the deadlock?
No. Buffering in your process only delays the moment the bytes reach the kernel pipe; the child still fills its output pipe and stops reading, so the eventual flush blocks exactly as before. It raises the size threshold at which the hang appears, which makes the bug rarer and harder to reproduce rather than fixed.

Two people pass buckets through a hatch that holds only a few at a time. If neither will take a bucket off the hatch until the other stops putting them on, both stand there holding a bucket forever.

saying these in an interview costs you the question

Says the pipe buffers as much as the child writes
Blames the compressor for being slow
Adds a longer timeout instead of draining the output
Thinks closing stdin sooner would fix it
Concludes it works because the small fixture passed
Wraps the stdin pipe in a bigger buffer and calls it fixed

Why does writing 200 MB to cmd.StdinPipe() before reading cmd.StdoutPipe() deadlock when piping a dataset through an external compressor?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

The mental model that causes the bug

The mechanism

Why it only happens with real data

Confirming it rather than guessing

The fix

The simpler fix

The general rule

▾common follow-ups3 questions

▾analogy~20 s

▾red flags6 flags

The mental model that causes the bug

The mechanism

Why it only happens with real data

Confirming it rather than guessing

The fix

The simpler fix

The general rule

answer

go deeper