What does io.Pipe() give you in Go, and how do the PipeReader and PipeWriter ends synchronise?
answer
- no capacity between the two ends
- one side pushes, the other pulls
- a Write waits for a Read
- two goroutines, or nothing moves
basics
~20 sio.Pipe returns a connected PipeReader and PipeWriter with no buffer between them: each Write blocks until reads have consumed it. The two ends must run in different goroutines, and it adapts writer-based producers to reader-based consumers.
solid answer
~40 s`io.Pipe()` returns a `*io.PipeReader` and a `*io.PipeWriter` joined in memory, with no internal buffering at all. A `Write` on the `PipeWriter` blocks until one or more `Read` calls on the `PipeReader` have consumed every byte of it, so the two ends have to live in different goroutines — driving both from one goroutine deadlocks immediately. Its purpose is impedance matching between APIs: something like `json.NewEncoder`, `gzip.NewWriter` or `tar.NewWriter` wants to *push* into an `io.Writer` you give it, while something like `http.NewRequest` wants to *pull* from an `io.Reader` you give it. The pipe joins the two without staging the payload in a `bytes.Buffer`, so a relay streams at constant memory. It is safe to use the two ends concurrently, and closing the write end is what tells the reader the stream is over.
code
go · 14 linespr, pw := io.Pipe()
go func() {
err := json.NewEncoder(pw).Encode(payload)
// a nil error closes the pipe as a normal end of stream
pw.CloseWithError(err)
}()
req, err := http.NewRequest("POST", url, pr)
if err != nil {
pr.Close()
return err
}
resp, err := http.DefaultClient.Do(req)go deeper
Know that io.Pipe hands you a reader and a writer joined together with no storage in between, and that the two ends need two goroutines. Be able to name one case where you needed an io.Reader but only had something that writes.
Explain the synchronous handoff: a Write returns only once reads have consumed its bytes, which is why a single goroutine driving both ends deadlocks. Contrast it with staging the payload in a bytes.Buffer and say when each is the right call.
Show the judgment: use a pipe when the payload is unbounded or must start flowing early, and a buffer when it is small and bounded. Mention the backpressure it gives for free, the synchronisation cost per write, and the close obligations on both ends.
Own where streaming is mandated in a codebase: which boundaries may hold a whole payload, what the per-request memory ceiling is when a remote peer picks the size, and whether relays should go through one reviewed helper rather than a hand-rolled pipe per service.
## The problem it solves Go's stream APIs come in two flavours. Some *push*: you hand them a destination and they write into it — `json.NewEncoder(w)`, `gzip.NewWriter(w)`, `tar.NewWriter(w)`, `template.Execute(w, data)`, `csv.NewWriter(w)`. Others *pull*: they take a source and read from it when they are ready — `http.NewRequest(method, url, body io.Reader)`, `io.Copy(dst, src)`, `csv.NewReader(r)`. When a producer of the first kind must feed a consumer of the second kind, the naive fix is a `bytes.Buffer`: encode into the buffer, then hand the buffer over as a reader. That works and is often correct for small payloads, but it holds the entire payload in memory, and for a stream of unknown length that means a remote peer chooses your heap size. `io.Pipe` is the streaming answer. ## The mechanism `func Pipe() (*PipeReader, *PipeWriter)` returns two connected halves. There is no byte buffer between them — none, not one byte. The implementation hands the writer's slice directly to a waiting reader and blocks the writer until the whole slice has been consumed. From the documentation's own framing: reads and writes match one to one, except that a single large `Write` may be satisfied by several smaller `Read` calls, in which case the `Write` returns only after the last of them. Three consequences follow directly. **1. Two goroutines are mandatory.** If one goroutine writes and then tries to read, the `Write` parks waiting for a reader that will never run, and the program deadlocks. In practice the producer goes into a `go func()` and the consumer stays on the calling goroutine (or vice versa). **2. Backpressure is automatic and total.** The producer cannot run ahead. If the consumer is a slow network socket, the encoder upstream simply blocks. You get flow control without writing any, which is exactly what a relay wants — and it is also why an `io.Pipe` in a latency-sensitive path adds a synchronisation round trip per write. Wrapping the write end in a `bufio.Writer` is the usual way to make many small writes cross the pipe as few large ones. **3. Nothing survives a dropped end.** Because there is no buffer, an abandoned reader leaves the writer parked forever — the failure mode that makes pipe-based relays leak goroutines. ## Closing `PipeWriter.Close()` signals the normal end of the stream: subsequent reads on the `PipeReader` return no bytes and the end-of-stream sentinel. `PipeWriter.CloseWithError(err)` does the same but delivers `err` to the reader instead, which is how a producer that failed halfway reports the failure rather than presenting a truncated stream as complete. Symmetrically, `PipeReader.Close()` makes subsequent (and currently blocked) writes fail with `io.ErrClosedPipe`, and `PipeReader.CloseWithError(err)` delivers a chosen error to the writer. Closing is not optional bookkeeping here: it is the only signal either side gets. ## Concurrency safety The pipe is safe for parallel use. `Read` and `Write` may be called concurrently with each other and with the `Close` methods, and parallel calls to `Read` (or to `Write`) are serialised internally. That said, several goroutines writing into one pipe produce interleaved output at unpredictable boundaries, so it is rarely what you want. ## `io.Pipe` versus a channel An unbuffered channel of `[]byte` looks similar, and for internal producer/consumer code it is often the better tool. `io.Pipe` earns its place when the thing on one side is not your code but a standard-library API that only speaks `io.Reader` or `io.Writer`. That is the whole point: it produces a real `io.Reader` you can hand to anything. ## When not to reach for it If the payload is small and bounded, a `bytes.Buffer` is simpler and faster — no goroutine, no synchronisation, no close discipline to get wrong. Reach for the pipe when the payload is large, unbounded, or must start flowing before it is fully produced.
- Why must the two ends of an io.Pipe be driven from different goroutines?There is no buffer, so a `Write` parks until reads consume every byte it handed over. A single goroutine that writes first never reaches its own `Read`, so the write waits forever and the program deadlocks — with only that goroutine involved, the runtime may not even be able to report it as a deadlock, since other goroutines are still runnable.
- Can a Write on an io.PipeWriter return before all its bytes are consumed?Only when something went wrong. A successful `Write` returns after readers have consumed the whole slice, possibly across several `Read` calls. A short count comes with a non-nil error — typically `io.ErrClosedPipe` because the read end was closed, or whatever error the reader passed to `CloseWithError`.
- Is it safe to call an io.Pipe's ends from several goroutines at once?Yes. `Read` and `Write` may be called in parallel with each other and with the `Close` methods; parallel calls on the same side are serialised internally. It is still rarely useful to have several writers, because their writes interleave at arbitrary boundaries and the consumer sees a scrambled stream.
It is a hand-to-hand relay rather than a mailbox: nobody can put anything down, so the giver stands still until someone takes it.
saying these in an interview costs you the question
- Thinks io.Pipe has an internal buffer like a buffered channel
- Writes and reads the same pipe from one goroutine
- Confuses io.Pipe with an OS pipe backed by file descriptors
- Uses io.Pipe for a small payload where a bytes.Buffer is simpler
- Assumes the writer may run ahead of a slow consumer