skip to content

Why can one Read on a net.Conn return fewer bytes than the peer sent in one Write?

level: middleimportance: must knowfreq 58%

answer

  1. a byte stream, not messages
  2. the io.Reader contract says up to len(p)
  3. one Write is not one Read
  4. the application supplies the boundaries
  5. Scanner for lines, ReadFull for counts

basics

~20 s

A net.Conn is a byte stream, not a message queue. Read returns whatever has arrived, so writes can split across reads or merge into one. The application supplies its own framing, such as newline-delimited lines or a length prefix.

solid answer

~50 s

`net.Conn` implements `io.Reader`, and the `io.Reader` contract only promises that `Read` returns between 0 and `len(p)` bytes with an error — never that it fills the slice or that it lines up with one call to `Write` on the other end. Bytes arrive in pieces, so one 4 KB write can surface as three reads, and three small writes can surface as one. That is why a protocol needs framing you supply yourself. For newline-delimited text, wrap the connection: `sc := bufio.NewScanner(conn)` and loop on `sc.Scan()`, which buffers partial data and hands you exactly one line per iteration; check `sc.Err()` afterwards, because Scan returns false both at clean end-of-stream and on failure. When you need a fixed number of bytes — a length prefix, say — use `io.ReadFull(conn, buf)`, which loops until the slice is full or fails with `io.ErrUnexpectedEOF`.

code

go · 12 lines
go
// Raw: n may be anything from 1 to len(buf), whatever the peer wrote.
buf := make([]byte, 4096)
n, err := conn.Read(buf)

// Framed: one \n-terminated line per iteration, buffering handled for you.
sc := bufio.NewScanner(conn)
for sc.Scan() {
	fmt.Fprintf(conn, "ok %s\n", sc.Text())
}
if err := sc.Err(); err != nil {
	log.Printf("read: %v", err)
}

go deeper

for a junior

Remember the one-line rule: a connection is a stream of bytes with no message boundaries. Reach for bufio.NewScanner when the protocol is newline-delimited text rather than parsing raw Read results yourself.

for a middle

Be ready to quote the io.Reader contract — up to len(p) bytes, short counts are legal — and to name both framing tools, bufio.Scanner for delimiters and io.ReadFull for fixed sizes.

for a senior

Show the hardening instincts: the Scanner's 64 KiB token ceiling as a defence against an unbounded peer, capping any length prefix before allocating, and distinguishing io.EOF from io.ErrUnexpectedEOF in a decoder.

for a principal

Own the framing decision for a protocol other teams will implement against: delimited text is debuggable by hand but needs escaping, length-prefixed binary is cheap but opaque, and the choice fixes what every client library must handle.

## The contract you are actually programming against `net.Conn` embeds `io.Reader`: ```go Read(p []byte) (n int, err error) ``` The documented promise is narrow. `Read` reads *up to* `len(p)` bytes and returns how many. It may return a short count with a nil error, and it is allowed to return `0, nil` (though callers should treat that as a no-op and keep going). It says nothing whatsoever about matching the peer's `Write` calls, because at the transport level there is no record of where one write ended. So all four of these are legal for a peer that wrote `"PING\n"` and then `"PONG\n"`: - one Read returning `"PING\nPONG\n"` - two Reads returning `"PING\n"` and `"PONG\n"` - three Reads returning `"PI"`, `"NG\nPON"`, `"G\n"` - a Read returning `"PING\nPO"` followed by a Read that blocks for a second Code that assumes the second case works perfectly on localhost with small payloads and then fails in production once packets are large enough to be split, or the peer's writes are coalesced. ## Framing is your job Since the stream carries no boundaries, the protocol must supply them. The two common shapes are: **Delimiter framing** — messages end at a byte such as `\n`. Read until the delimiter appears. **Length framing** — each message is preceded by a fixed-size length, so you read the header, then exactly that many bytes. Go's standard library has a ready tool for each. ### Delimiter framing with `bufio.Scanner` ```go sc := bufio.NewScanner(conn) for sc.Scan() { line := sc.Text() // one line, newline already stripped fmt.Fprintf(conn, "ok %s\n", line) } if err := sc.Err(); err != nil { log.Printf("read: %v", err) } ``` The Scanner keeps an internal buffer, calls `Read` as often as needed, and only returns from `Scan` when a full token is available. Three things about it are worth knowing. First, its default split function is `bufio.ScanLines`, which splits on `\n` and also strips a trailing `\r` — friendly to peers that send CRLF. Second, `Scan` returning false is ambiguous on its own: it means either a clean end of stream or an error, and `sc.Err()` is what distinguishes them. Err returns nil at a clean end, because Scanner deliberately does not report `io.EOF` as a failure. Third, there is a size ceiling. A Scanner will not buffer a token longer than `bufio.MaxScanTokenSize` (64 KiB) unless you raise it with `sc.Buffer(buf, max)`. A peer that sends a megabyte with no newline stops the loop with `bufio.ErrTooLong`. That ceiling is a feature on a network connection — it stops a hostile or broken client from making you allocate without bound — and code that raises it should raise it to a considered number, not to `math.MaxInt`. ### Length framing with `io.ReadFull` ```go var hdr [4]byte if _, err := io.ReadFull(conn, hdr[:]); err != nil { return err } n := binary.BigEndian.Uint32(hdr[:]) body := make([]byte, n) // check n against a cap first if _, err := io.ReadFull(conn, body); err != nil { return err } ``` `io.ReadFull` loops over `Read` until the slice is exactly full. If the stream ends after zero bytes it returns `io.EOF`; if it ends part-way it returns `io.ErrUnexpectedEOF`, which is the distinction you want in a decoder. Note the comment: sizing a slice from a number the peer sent is an unbounded allocation waiting to happen, so cap it before calling `make`. ## Writes have a mirror-image rule `Write` on a `net.Conn` returns `n` and an error, and the `io.Writer` contract requires it to return a non-nil error whenever `n < len(p)` — so a partial write always comes with an error and you do not need a retry loop the way you do for reads. What you do need is to check the error at all, including on `Close`, and to buffer small writes through a `bufio.Writer` if you are emitting many tiny pieces, remembering to `Flush`. ## The shape of the answer in an interview Say the contract (up to `len(p)` bytes), say that framing is the application's responsibility, and name the two tools: `bufio.Scanner` for delimited text, `io.ReadFull` for fixed-size chunks. Mentioning the 64 KiB Scanner ceiling shows you have run this against a real peer.

  • bufio.Scanner's Scan returns false. How do you tell a clean end of stream from a failure?
    Call `sc.Err()`. It returns nil when the stream ended cleanly — Scanner deliberately does not surface `io.EOF` as an error — and returns the underlying failure otherwise, such as `bufio.ErrTooLong` for an over-long token or a network error from the connection. Code that loops on Scan and never checks Err silently treats broken connections as normal terminations.
  • What happens when a client sends a 1 MB line with no newline to a default bufio.Scanner?
    Scan stops and `Err` reports `bufio.ErrTooLong`: the default ceiling is `bufio.MaxScanTokenSize`, 64 KiB. On a network connection that ceiling is protective, since it bounds how much a peer can make you buffer. Raise it deliberately with `sc.Buffer(make([]byte, 0, n), max)` to a size your protocol actually needs, not to an unlimited value.
  • Does the same short-count problem exist on Write?
    No, in practice. The `io.Writer` contract requires a non-nil error whenever fewer than `len(p)` bytes are written, so you check the error rather than looping. The write-side concern is different: many tiny writes each cost a syscall, so wrap the connection in a `bufio.Writer` and Flush, and always check the error returned by Flush and by Close.

It is a water pipe, not a parcel service. You can pour two buckets in and catch three cups out; if you want parcels back you have to tie a knot between them yourself.

saying these in an interview costs you the question

  • Assumes one Write on the peer equals one Read locally
  • Reads into a big buffer once and parses the whole message
  • Loops on Scan without ever checking Scanner.Err
  • Treats a short read with a nil error as an error
  • Allocates a buffer sized from a peer-supplied length with no cap