Retries, Backoff and Idempotency

## The body is a stream, not a value `http.Request` carries its payload in a field of type `io.ReadCloser`: ```go type Request struct { Body io.ReadCloser // ... } ``` An `io.Reader` is a source you pull bytes from until it reports `io.EOF`. It has no rewind, no length and no memory of what it already produced. When you write `bytes.NewReader(payload)` you get a `*bytes.Reader` positioned at offset zero; every `Read` moves that offset forward. Once it has yielded the last byte, further reads return `0, io.EOF` forever unless something explicitly seeks it back. That is the whole explanation for the empty second attempt. `http.Client.Do` hands the request to the `Transport`, which streams `req.Body` onto the connection and then closes it. The net/http documentation is explicit that the Transport closes the request body — including when the request fails — so by the time `Do` returns to you, whether it succeeded or not, that body is consumed and closed. Calling `Do` again with the same `*http.Request` presents an already-drained, already-closed reader. ## What you usually see instead of an empty POST When `http.NewRequest` or `http.NewRequestWithContext` is given a `*bytes.Buffer`, `*bytes.Reader` or `*strings.Reader`, it also records `req.ContentLength` from that value's length. On the retry, the Transport still believes it must write, say, 214 bytes, but the reader hands it zero. Rather than send a truncated request it fails the write with an error along the lines of `http: ContentLength=214 with Body length 0`. So the symptom in the wild is often a confusing client-side error on attempt two, not a mysterious blank payload at the server — though a body of unknown length can indeed arrive empty. ## The shape that gets this wrong A webhook delivery worker draining a queue is the classic case. It pops a job, marshals the event to JSON, builds one request, and loops: ```go req, _ := http.NewRequestWithContext(ctx, http.MethodPost, endpoint, bytes.NewReader(payload)) for attempt := 0; attempt < 3; attempt++ { resp, err := client.Do(req) // attempt 2 has no bytes left // ... } ``` The request object looks reusable because it is a plain struct with a URL and headers, and nothing about the code says "consumed". The Transport may also have mutated the request on the way out (host, transfer encoding, the closed body itself), so reuse is unsound beyond the body too. ## The two correct fixes **Rebuild the request per attempt.** Hold the payload as a `[]byte` for the life of the retry loop and construct a new `*http.Request` each time around, wrapping the same slice in a new `bytes.NewReader`. This is the version to reach for by default: it is obviously correct, it re-derives `ContentLength` every time, and it gives you a clean place to attach a per-attempt context. **Use `Request.GetBody`.** That field is a `func() (io.ReadCloser, error)` whose job is to produce a fresh reader over the same bytes. `http.NewRequestWithContext` populates it for you when the body you passed was a `*bytes.Buffer`, `*bytes.Reader` or `*strings.Reader`. A retry loop can therefore call `body, err := req.GetBody()` and assign the result to a cloned request's `Body` before each attempt. If you built the body from some other reader, `GetBody` is nil and you must set it (and `ContentLength`) yourself, or accept that the request is not replayable. ## What replayability costs Both fixes have the same price: the payload must stay in memory for as long as you might retry. Streaming a large upload straight from an `io.Reader` is cheaper, but a stream that cannot be re-opened cannot be replayed — that is a real design tradeoff, not an oversight. For a delivery worker, holding the JSON payload is usually fine, but remember that the memory is per in-flight delivery: concurrency times payload size. ## Proving it in a test Stand up an `httptest.NewServer` whose handler counts calls, fails the first two with a 500, then succeeds, and have it read and record the body it received on every call. Point your worker at `srv.URL`, run one delivery, and assert that all three recorded bodies are byte-identical. The broken version fails that assertion (or errors out) on the second call, and the assertion keeps failing loudly if someone later "optimises" the loop by hoisting the request construction out of it. ## The mental model to keep A response body is something you read once and close; a request body is something the client reads once and closes. Retrying is not re-sending an object, it is performing the write again — and a write needs its bytes again.