What does io.ReadAll do, and why is it risky on a stream you do not control?
answer
- who decides how big the result is
- reads until the stream ends
- no parameter for a maximum
- a nil error at a normal end
- bound the reader you hand it
basics
~20 sio.ReadAll reads an io.Reader to end of stream and returns every byte in one slice. It takes no size limit, so a sender that keeps sending can grow that slice until the process runs out of memory.
solid answer
~50 s`io.ReadAll(r)` loops on the reader's Read method into a slice it keeps growing, stops at end of stream, and returns `([]byte, error)`. A clean end is success: it absorbs `io.EOF` and hands back a nil error, so a nil error tells you nothing about how much data arrived. The size of the result is chosen entirely by whoever is writing into that reader, and there is no parameter to cap it, so on a socket or any other untrusted stream io.ReadAll is an unbounded allocation and a cheap denial of service: a few bytes of request can cost you gigabytes of heap. Use it freely for input you produced yourself. For anything you do not control, bound the reader first, for example `io.ReadAll(io.LimitReader(r, maxBytes))`, and treat reaching that cap as a rejection rather than as the end of the data.
code
go · 6 lines// unbounded: the peer on the other end decides how much memory this costs
data, err := io.ReadAll(conn)
// bounded: never much more than maxBytes, but note that hitting the cap
// looks exactly like a clean end of stream
data, err = io.ReadAll(io.LimitReader(conn, maxBytes))go deeper
Know the signature, and that io.ReadAll hands back everything the reader produced in one slice with a nil error at a normal end. Be ready to say why that is fine for a small file and risky for a network connection.
Explain that end of stream is absorbed into a nil error, and that the slice grows by allocate-and-copy, so peak heap during the read is higher than the final length.
Show where you draw the trust boundary in a real service: which readers get wrapped, what the limit is, and how an over-limit input is reported rather than silently truncated.
Own the policy rather than the call site: a default limit for the whole service, where it is configured, and how you learn which callers a new limit would break before you enforce it.
## What io.ReadAll does `io.ReadAll(r io.Reader) ([]byte, error)` is the "give me the whole thing" helper of the `io` package. It calls `r.Read` repeatedly into a byte slice that it grows as needed, and stops when the reader reports end of stream. It returns the accumulated bytes and an error. Two details of that contract catch people out. **End of stream is success, not an error.** `io.EOF` is how an `io.Reader` says "there is nothing more", and io.ReadAll swallows it. A successful, complete read returns `err == nil`. Writing `if err == io.EOF` after io.ReadAll is dead code. Conversely, any non-nil error you do get is a genuine failure — a broken connection, a closed file, a deadline — and the slice returned alongside it holds the bytes that arrived before the failure, which may be worth logging but is not a complete message. **There is no size argument, and there is no way to add one.** The function signature accepts one reader and nothing else. The amount of memory the call consumes is decided by the writer on the other side of that reader. ## Why that is a problem on an untrusted stream Think about the asymmetry. A peer that wants to hurt you does not have to do anything clever: it opens a connection, writes bytes steadily, and never closes. Your side sits inside io.ReadAll appending. The peer spends bandwidth; you spend resident memory. Multiply by the number of connections you accept concurrently and the process is out of memory long before anything logs an error. Peak memory is also higher than the final length suggests. The slice grows by allocating a bigger backing array and copying, so at the moment of a growth step both the old and the new array are live. A result of N bytes can transiently need noticeably more than N bytes of heap, and the discarded arrays are garbage the collector then has to deal with. A few defences people reach for do not actually help: - **A read deadline** bounds how long you wait, not how much you allocate. A peer sending fast fills your heap well inside any sane deadline. - **The operating system** does not cap this; from the kernel's point of view your process is simply asking for memory, and the answer is yes until it is a kill. - **The garbage collector** cannot help, because the slice is live — you are still holding it. - **A soft memory limit** such as `GOMEMLIMIT` makes the collector work harder as you approach the limit; it does not make the allocation smaller, and under this kind of pressure you often get a slow, thrashing service rather than a clean failure. ## The fix: bound the reader, not the read Because io.ReadAll takes no limit, the limit has to live in the reader you hand it. `io.LimitReader(r, n)` returns a reader that yields at most `n` bytes from `r` and then reports end of stream. Composed together: ```go data, err := io.ReadAll(io.LimitReader(conn, maxBytes)) ``` the call can never allocate more than roughly `maxBytes`. One subtlety follows immediately and is worth stating out loud in an interview: io.LimitReader reports the cap by returning `io.EOF`, exactly the way a genuine end of stream looks, so this composition silently truncates oversized input instead of rejecting it. If truncated data is dangerous — and for anything you are going to parse, it is — read with a limit of `maxBytes+1` and treat a result longer than `maxBytes` as "too large", so you get an explicit rejection instead of a half message. The other structural answer is not to buffer at all. If the work you do with the bytes can be done incrementally, stream them: hand the reader to whatever consumes it and let it pull. Buffering the whole payload is a choice, and on a hostile input it is the expensive one. ## When io.ReadAll is the right call The rule is not "never use io.ReadAll". It is "never call it on bytes whose size someone else chooses". A configuration file you ship, a test fixture, a small local file, a payload you have already capped upstream — all fine, and reaching for a manual read loop there is noise. The judgment being tested is whether you can tell those two situations apart in your own codebase.
- Does io.ReadAll return io.EOF when the stream ends normally?No. It treats end of stream as success and returns a nil error together with whatever it read, so a check for `err == io.EOF` after io.ReadAll never fires. Any non-nil error it returns is a real failure, and the slice returned alongside that error holds only the bytes that arrived before it.
- Is the capacity of the slice io.ReadAll returns the same as its length?Not necessarily. io.ReadAll starts with a small buffer and grows it as data arrives, so the final slice can carry spare capacity, and peak heap during the read exceeds the final length because the old array is still live while the bigger one is being filled. If you will hold the result for a long time, copy it into a right-sized slice.
- When is io.ReadAll the right thing to call?When you know the input is bounded because you produced it or already capped it: a config file you ship, a test fixture, a small local file, a payload a limit upstream has already constrained. The rule is not to avoid io.ReadAll, it is to avoid calling it on bytes whose size someone else chooses.
It is a bucket with no rim: you hold it under the tap until the water stops, and the person at the tap decides when that is.
saying these in an interview costs you the question
- Says io.ReadAll returns io.EOF at a normal end of stream
- Assumes a nil error means the input was small
- Thinks io.ReadAll streams rather than buffering everything
- Relies on a read deadline to bound memory
- Believes the operating system or the garbage collector caps the allocation