skip to content

Why does reading from the net.Conn returned by Hijack lose bytes the client already sent?

level: seniorimportance: nice to knowfreq 26%

answer

  1. the server read more than the request
  2. those bytes are already off the socket
  3. two values came back from Hijack, not one
  4. check Reader.Buffered right after hijacking

basics

~20 s

net/http reads requests through a buffered reader and can read past the end of the request head. Those bytes are already off the socket, so a read on the net.Conn skips them. Read through the *bufio.ReadWriter Hijack returned instead.

solid answer

~50 s

The server parses requests through a `*bufio.Reader` on the connection, and a single TCP read can pull in more than the request head — a client that writes its upgrade request and its first protocol bytes together leaves those trailing bytes sitting in the server's buffer. `Hijack` hands you that same buffer as the reader half of the returned `*bufio.ReadWriter`, precisely so nothing is lost. If you ignore it and call `conn.Read`, the kernel has no copy of those bytes any more and you silently start the session mid-stream. The symptom is nasty: it works on a loopback client that pauses after the handshake and fails against a client that batches its writes, so it looks like flakiness rather than a bug. Log `brw.Reader.Buffered()` immediately after hijacking and read through `brw` for the life of the connection.

code

go · 13 lines
go
conn, brw, err := hj.Hijack()
if err != nil {
	return
}
defer conn.Close()

// Note the qualification: both embedded halves define Buffered.
log.Printf("already buffered at hijack: %d bytes", brw.Reader.Buffered())

buf := make([]byte, 4096)
n, err := brw.Read(buf) // drains the buffer first, then the socket
_, _ = n, err
// conn.Read(buf) here would silently skip whatever brw still holds.

go deeper

for a junior

Remember that Hijack returns two useful values and the buffered one is not optional: read through it, because the server may already have pulled some of the client's bytes off the socket.

for a middle

Explain the mechanism — a bufio read fills from the socket in bulk and can capture bytes past the request head — and name the correct read path through the returned reader.

for a senior

Show the diagnosis: a first message that vanishes only against clients that batch writes, confirmed by logging the reader's buffered count at hijack time, plus the rule for draining before handing the connection elsewhere.

for a principal

Recognise this class of defect as an argument about who writes protocol code: it passes review, passes tests, and fails in production, which is a reason to keep hand-rolled upgrades in one audited place rather than spread across teams.

## Where the missing bytes went Nothing in `net/http` reads a request byte at a time. Each connection is served through a `*bufio.Reader`, and every fill of that buffer takes whatever the socket currently holds — up to the buffer size — not just the part that belongs to the request head. If the peer wrote the upgrade request and the first bytes of the new protocol in the same burst, the kernel may hand both to that one read. The request parser consumes the head and stops; the rest stays in the bufio buffer. So at the moment `Hijack` returns, the connection's data lives in two places: - whatever the buffered reader is still holding, already drained from the kernel; - whatever the client sends from now on, still in the socket. That is exactly why the signature is `Hijack() (net.Conn, *bufio.ReadWriter, error)` and not just `(net.Conn, error)`. The `*bufio.ReadWriter` is not a convenience wrapper — it is the only handle on the first group. ## The bug ```go conn, brw, err := hj.Hijack() // ... write the handshake ... n, err := conn.Read(buf) // WRONG ``` This reads the socket directly and therefore begins with whatever the client sends *next*. Anything already buffered is skipped, forever. Depending on the protocol you are speaking, the result is a corrupt first frame, a desynchronised parser, or a session that simply stalls waiting for a message it already received. The correct form reads through the returned reader, which drains its buffer first and then falls through to the socket: ```go n, err := brw.Read(buf) // correct ``` Once you are reading through `brw`, keep doing so for the whole life of the connection; do not alternate between `brw` and `conn` on the read side. ## Why it hides in testing Hand-written test clients tend to write the request, read the 101, and only then write protocol data — three separate syscalls with the server's reply in between, which guarantees the server's buffer is empty at hijack time. Real clients pipeline: they write the request and their first message together to save a round trip, and TLS makes it worse by delivering whole records at once. So this defect ships green and fails against the first efficient client, usually as "the session works but the very first message is lost". ## Confirming it `brw.Reader.Buffered()` returns the number of bytes currently held in the reader. Logging it immediately after `Hijack` tells you unambiguously whether the connection arrived with data in hand. Note the qualification: `*bufio.ReadWriter` embeds both a `*bufio.Reader` and a `*bufio.Writer`, and **both** types have `Buffered`, `Reset` and `Size` methods, so the selector `brw.Buffered()` is ambiguous at the same embedding depth and does not compile. You must say `brw.Reader.Buffered()` or `brw.Writer.Buffered()`. Method names that appear on only one half — `Flush`, `WriteString`, `Read`, `ReadString`, `Peek` — promote normally and can be called on `brw` directly. ## The write side is different On the write side there is no equivalent trap: the server has nothing pending for you, and writing to `conn` directly is legitimate. The only rule is not to interleave `conn` writes with buffered `brw` writes without flushing in between, since the buffered bytes would then arrive after the direct ones. ## The rule of thumb Treat the `net.Conn` from `Hijack` as the thing you close and configure, and the `*bufio.ReadWriter` as the thing you read. If you truly want to hand a plain `net.Conn` to some other library, drain the buffered reader first — check `Buffered()` and copy what it holds — rather than assuming it is empty because it usually is.

  • Why is brw.Buffered() a compile error on a *bufio.ReadWriter?
    `bufio.ReadWriter` embeds both `*bufio.Reader` and `*bufio.Writer`, and both types declare `Buffered` — as well as `Reset` and `Size`. Two promoted methods at the same depth make the selector ambiguous, so the compiler refuses it. Qualify the half you mean: `brw.Reader.Buffered()`.
  • Is it safe to write directly to the net.Conn while reading through the ReadWriter?
    Yes on the write side, as long as you do not mix it with unflushed buffered writes, which would then arrive out of order. The read side is the asymmetric one: bytes can already be in the reader's buffer, so reads must go through it — or you must drain it explicitly before handing the raw connection to anything else.
  • How would you hand the hijacked connection to code that only accepts a net.Conn?
    Drain the buffered reader first. Check `brw.Reader.Buffered()`, read that many bytes out, and either replay them into the consumer or wrap the connection so those bytes are returned before any socket read. Passing the bare net.Conn while the buffer is non-empty loses data silently.

saying these in an interview costs you the question

  • Assumes the socket still holds everything the client sent
  • Treats the returned *bufio.ReadWriter as an optional convenience
  • Alternates reads between the conn and the buffered reader
  • Calls the bug flaky because a local test client never triggers it
  • Hands the raw net.Conn onward without draining the buffer