skip to content

Why do open upstream connections pile up when a Node.js upload proxy forwards request bodies to object storage with req.pipe(upload) and clients disconnect mid-upload?

level: seniorimportance: should knowfreq 45%

answer

  1. what a dropped client does to req
  2. pipe() ends the destination only on 'end'
  3. nobody destroys the upstream request
  4. one promise, one AbortSignal

basics

~20 s

When a client disconnects, Node destroys req, which closes without ever emitting 'end'. pipe() only ends its destination on 'end', so each upstream upload is left half-written and open. await pipeline(req, upload, { signal }) destroys both sides on any failure.

solid answer

~50 s

When the client goes away mid-body, Node destroys the incoming request; it emits `'close'` but never `'end'`. `req.pipe(upload)` only calls `upload.end()` on `'end'`, and it does not close the destination when the source fails, so every aborted upload leaves an outgoing request to storage open and half-written until some timeout on either side frees it. The reverse failure is also bad: if storage resets the upload, `pipe()` just unpipes, leaving `req` paused and unread, and an unlistened `'error'` on `upload` can crash the process. The fix is `await pipeline(req, upload, { signal: AbortSignal.timeout(ms) })` from `node:stream/promises` inside `try`/`catch`: any failure, a premature close or the deadline destroys both streams, and the handler then answers with a 502 or 504 if the client is still there. Because `res` is not inside the pipeline, it is still usable for that answer.

code

javascript · 17 lines
javascript
import http from 'node:http';
import { pipeline } from 'node:stream/promises';

http.createServer(async (req, res) => {
  const upload = http.request('http://127.0.0.1:9000/uploads/blob', { method: 'PUT' });
  upload.on('response', (upRes) => {
    upRes.resume(); // discard the storage response body
    if (!res.headersSent) res.writeHead(upRes.statusCode < 300 ? 201 : 502).end();
  });
  try {
    await pipeline(req, upload, { signal: AbortSignal.timeout(120_000) });
  } catch (err) {
    // req and upload are already destroyed; res is outside the pipeline.
    console.error('upload failed:', err.name, err.code ?? '');
    if (!res.headersSent) res.writeHead(err.name === 'AbortError' ? 504 : 502).end();
  }
}).listen(8080);

go deeper

for a junior

Recall that a client disconnect stops the request stream and that pipe() does not close the stream it writes into when that happens.

for a middle

Explain why req closes without 'end', why pipe() only ends its destination on 'end', and what pipeline destroys instead.

for a senior

Diagnose the leak from socket counts and error codes, fix it with the promise form of pipeline plus a deadline, and keep res outside the pipeline so you can still answer.

for a principal

Decide the proxy's failure budget: one end-to-end deadline per upload, which status each failure maps to, and an alert on rising premature-close rates.

## The setup An **upload proxy** receives a file in an HTTP request body and forwards it to object storage without writing it to local disk. In Node.js the incoming body `req` is a **Readable** stream and the outgoing upload (for example an `http.request()` with method `PUT`) is a **Writable**. The naive code connects them with `req.pipe(upload)` and responds when the upload finishes. Under normal traffic this works, and it even respects **backpressure**: when storage accepts data slowly, `pipe()` pauses `req`, TCP flow control slows the client, and memory stays flat. The trouble is entirely in the failure paths. ## What happens when the client disconnects 1. The client's connection drops mid-body (a closed laptop lid, a mobile network change, a cancelled upload). 2. Node's HTTP server destroys the in-flight `req` with an `'aborted'` error whose code is `ECONNRESET`. Node only emits that error as an `'error'` event when `req` has an `'error'` listener; otherwise `req` simply closes. 3. Either way, `req` emits `'close'` and never `'end'`. 4. `pipe()` ends its destination only when the source emits `'end'`, and Node documents that a destination is **not closed automatically** when the source errors. 5. So `upload` is never ended and never destroyed. The outgoing connection to storage stays open, holding a socket, buffers and whatever server-side state the half-finished upload holds, until an idle timeout on one side fires. Under steady traffic with a share of aborted uploads, those orphaned connections accumulate. Symptoms are a climbing count of open sockets and file descriptors, memory held by half-finished uploads, and eventually `EMFILE` errors or memory pressure. ## What happens when storage fails The opposite direction leaks too: - If `upload` errors (storage resets the connection), `pipe()` only **unpipes**. `req` is left paused, nobody reads it, and the client's upload stalls until the client or a server timeout gives up. - If nothing listens for `'error'` on `upload`, the error is an unhandled `'error'` event and **crashes the process**, taking every other in-flight request with it. ## The fix: one pipeline with a deadline ```js await pipeline(req, upload, { signal: AbortSignal.timeout(120_000) }); ``` `pipeline` from `node:stream/promises` changes every failure path: - it listens for errors and premature close on both streams, so a dropped client rejects the promise, typically with the `ECONNRESET` error or `ERR_STREAM_PREMATURE_CLOSE`; - on any failure it calls `destroy(err)` on each stream that has not already finished, so the upstream connection is torn down at once; - the `signal` bounds the whole transfer: when it aborts, `pipeline` destroys the chain and rejects with an `AbortError` whose `cause` is the signal's reason; - the promise settles once, so the handler has exactly one place to decide the response. ## Responding after the pipeline Keep `res` **out** of the pipeline. `pipeline` destroys every stream it owns on failure, so if `res` were the last stage, a storage error would close the client's socket before you could send a status. With `res` outside it: | Outcome | What the handler does | |---|---| | Pipeline resolves and storage answers 2xx | Reply 201 to the client | | Pipeline resolves, storage answers an error status | Reply 502 | | Rejects with an `AbortError` (deadline) | Reply 504 if `res` is still writable | | Rejects because the client disconnected | Log it; there is nobody to answer | Also consume or discard the storage response body (`upRes.resume()`), or that connection stays occupied too. ## Why not buffer the body instead Reading the whole upload into memory and sending it afterwards would sidestep the stream lifecycle, but it trades one problem for worse ones: - memory per upload grows to the file size, multiplied by concurrent uploads in one heap; - storage receives nothing until the client has finished, doubling the end-to-end time; - a disconnect still has to be detected and the half-read body discarded. Streaming with `pipeline` keeps memory near the streams' `highWaterMark` values and handles the disconnect for you. ## How to confirm it in production - Count open sockets or file descriptors per process over time and correlate with aborted requests. - Log every rejected pipeline with its error code: a burst of `ECONNRESET` or `ERR_STREAM_PREMATURE_CLOSE` matches the disconnect pattern, `AbortError` matches the deadline. - Reproduce locally by starting an upload with a rate-limited client and killing it halfway; with `pipe()` the upstream connection stays open, with `pipeline` it closes immediately.

  • Why not add a req.on('close') handler that calls upload.destroy() and keep pipe()?
    It covers one direction only. You would also need an `'error'` listener on `upload` that destroys `req`, a check for `'close'` before `'end'`, care not to destroy after a normal finish, and a deadline. That is a hand-written `stream.pipeline`; using the real one gets the edge cases right and gives one promise to handle.
  • What does the AbortSignal passed to pipeline do that a socket timeout on the upstream request does not?
    It bounds the whole transfer from start to finish, including a client that trickles bytes slowly enough never to trip an idle timeout. When it aborts, `pipeline` destroys every stream and rejects with an `AbortError`, so one deadline covers both sides.
  • Does pipeline keep backpressure working between the client and storage?
    Yes. When the upstream request's `write()` returns `false`, `pipeline` stops reading `req` until `'drain'`, and because `req` is not read, TCP flow control slows the client. Memory per upload stays near the two streams' `highWaterMark` values rather than the file size.

  • pipe() closes the destination when the source errors, so nothing can leak
  • A disconnected client makes req emit 'end', so the upload finishes normally
  • Putting res as the last stage of the pipeline is fine for sending a 502 afterwards
  • An idle timeout on the upstream request is enough to bound a slow upload
  • Buffering the whole body before uploading avoids the problem at no real cost
  • An 'error' on the upstream request without a listener is harmless