skip to content

Why does io.LimitReader in front of a JSON decoder turn an over-sized request into a parse error?

level: middleimportance: nice to knowfreq 38%

answer

  1. one signals the end, one signals a failure
  2. EOF is not an error in Go's IO contract
  3. the decoder only knows the value stopped
  4. 413 versus 400 hangs on which wrapper
  5. only one of them takes a ResponseWriter

basics

~20 s

io.LimitReader reports io.EOF once its byte budget is spent, which looks exactly like the stream ending. The decoder sees a value cut off mid-way and reports truncated input, so the handler cannot tell an over-sized body from malformed JSON.

solid answer

~50 s

`io.LimitReader(r, n)` hands out at most `n` bytes and then behaves like a reader that has reached its end: it returns `io.EOF`. A JSON decoder reading through it gets a value that stops in the middle of an array or object and reports truncated input — an unexpected end of the value — which is indistinguishable from a client that genuinely sent a half-written payload. The handler therefore answers 400 for a request that should be 413, and the caller has no way to learn the size limit. `http.MaxBytesReader` exists for this: the over-limit read fails with a `*http.MaxBytesError` you can match with `errors.As`, and it also signals the server not to keep draining and reusing the connection. Use `io.LimitReader` where you own the error mapping — a file, a pipe, a non-HTTP stream — and `http.MaxBytesReader` inside an HTTP handler.

code

go · 14 lines
go
// Bounded, but the failure looks like malformed input.
if err := json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&v); err != nil {
	// err reports the value ended unexpectedly; size is not visible here
}

// Bounded, and the failure is attributable.
r.Body = http.MaxBytesReader(w, r.Body, 1<<20)
if err := json.NewDecoder(r.Body).Decode(&v); err != nil {
	var tooLarge *http.MaxBytesError
	if errors.As(err, &tooLarge) {
		http.Error(w, "body too large", http.StatusRequestEntityTooLarge)
		return
	}
}

go deeper

for a junior

Know that io.LimitReader stops by reporting the end of the stream, not by failing, and that a decoder reading through it will complain that the value was cut short.

for a middle

Explain the contract difference in Go's IO terms — io.EOF is a normal end signal — and show the resulting status-code bug in a handler that used the wrong wrapper.

for a senior

Argue the operational side: an over-sized body reported as malformed input corrupts both the caller's debugging and your own traffic metrics, and say which wrapper belongs on which side of a service.

for a principal

Set the convention so nobody has to choose per handler: one shared middleware caps every inbound body with the typed error and a documented 413 body, and io.LimitReader is reserved for non-request streams.

## Two limiters, two different contracts Both wrappers stop an untrusted source from handing you unbounded bytes, but they disagree about what "stopped" means, and the disagreement is the whole answer. **`io.LimitReader(r io.Reader, n int64) io.Reader`** returns a reader that reads from `r` but stops with `io.EOF` after `n` bytes. `io.EOF` is not an error condition in Go's IO contract — it is the normal, expected signal that a stream is over. The limiter is deliberately indistinguishable from a shorter source. **`http.MaxBytesReader(w http.ResponseWriter, r io.ReadCloser, n int64) io.ReadCloser`** returns a reader whose over-limit read fails with a real error, of the concrete type `*http.MaxBytesError` carrying a `Limit` field. It also takes the `ResponseWriter` so it can tell the server the request was over-sized, which stops the server from trying to drain the remainder of a flood before replying and from holding the connection open for reuse. ## What the decoder sees Consider a handler that does `json.NewDecoder(io.LimitReader(r.Body, 1<<20)).Decode(&v)` and receives a 4 MB body. The decoder consumes a megabyte of a JSON value that is still open — say, halfway through the third element of a large array — and then the reader says EOF. From inside the decoder there is no such thing as "the source was cut off by a policy": there is only a value that ended before it was complete, which is a truncated-input error (a `Decoder` surfaces this as `io.ErrUnexpectedEOF`; `json.Unmarshal` over a truncated buffer reports a syntax error about input ending unexpectedly). So the handler's error branch sees the same thing it would see if a client's connection dropped mid-upload or if someone posted `{"a":` by hand. The consequences compound: - The response is 400 when it should be 413, so a caller with an over-sized payload is told their JSON is malformed. They will go and check their JSON, which is fine. - Nothing in the response names the limit, so there is no path to "send smaller batches". - Your metrics and logs cannot separate "clients are hammering us with huge bodies" from "clients are sending broken payloads", which are two very different operational stories. ## The one thing io.LimitReader does buy you It still bounds the allocation, which is the primary job. A handler with `io.LimitReader` is far safer than a handler with no cap. The complaint is about diagnosability, not about protection — and about the fact that on the server side the better tool is one line away. There is a known trick when you must use `io.LimitReader` and still want to detect the overflow: give it `n+1` bytes of budget and check afterwards whether you consumed more than `n`. If you did, the source had more to give. It works, but inside an HTTP handler it is strictly more code for a worse result than `http.MaxBytesReader`. ## Where each belongs Use `http.MaxBytesReader` (or `http.MaxBytesHandler` to wrap a whole handler) in server handlers, because you want the typed error, the correct status code, and the connection hint. Use `io.LimitReader` for untrusted streams that are not an HTTP server request and where you control the surrounding error mapping: bounding how much of a file or a subprocess's output you will buffer, capping a response body you are reading as a *client*, or truncating input in a tool where a short read is an acceptable outcome. In those places the EOF-shaped contract is a feature, because it composes with every reader-consuming API in the standard library without anyone needing to know about a special error type. ## The reviewable rule In an HTTP handler, a cap that produces a syntax error is a cap that lies about who was at fault. Wrap `r.Body` with `http.MaxBytesReader`, match `*http.MaxBytesError` with `errors.As`, and return 413 with the limit in the message. Everywhere else, `io.LimitReader` is the right tool and its EOF is the right contract.

  • If io.LimitReader still bounds the allocation, is the difference only cosmetic?
    No. The allocation is bounded either way, but the response status, the message the caller gets, and your ability to separate "over-sized" from "malformed" in logs and metrics all change. On a public endpoint those are operational facts, not cosmetics.
  • Is there any way to detect the overflow when you must use io.LimitReader?
    Give it a budget of `n+1` bytes and check afterwards whether more than `n` were consumed; if so the source had more to give. It works for non-HTTP streams, but in a handler it is more code than `http.MaxBytesReader` for a worse result.
  • Where is io.LimitReader the better choice?
    Anywhere the EOF contract is what you want and you own the error mapping: bounding how much of a file or subprocess output you buffer, capping a response body you read as a client, or truncating input in a tool. It composes with every reader-consuming API without a special error type.

io.LimitReader hangs up mid-sentence; http.MaxBytesReader says out loud that you have gone over the word count. Both stop you, but only one leaves the listener able to explain what happened.

saying these in an interview costs you the question

  • Says io.LimitReader returns an error at its limit
  • Returns 400 for an over-sized body and calls it correct
  • Treats io.EOF from a limiter as proof the client sent everything
  • Thinks the two wrappers are interchangeable in a handler
  • Cannot say why one of them needs the ResponseWriter