Why do open upstream connections pile up when a Node.js upload proxy forwards request bodies to object storage with req.pipe(upload) and clients disconnect mid-upload?
answer
- what a dropped client does to req
- pipe() ends the destination only on 'end'
- nobody destroys the upstream request
- one promise, one AbortSignal
basics
~20 sWhen a client disconnects, Node destroys req, which closes without ever emitting 'end'. pipe() only ends its destination on 'end', so each upstream upload is left half-written and open. await pipeline(req, upload, { signal }) destroys both sides on any failure.
solid answer
~50 sWhen the client goes away mid-body, Node destroys the incoming request; it emits `'close'` but never `'end'`. `req.pipe(upload)` only calls `upload.end()` on `'end'`, and it does not close the destination when the source fails, so every aborted upload leaves an outgoing request to storage open and half-written until some timeout on either side frees it. The reverse failure is also bad: if storage resets the upload, `pipe()` just unpipes, leaving `req` paused and unread, and an unlistened `'error'` on `upload` can crash the process. The fix is `await pipeline(req, upload, { signal: AbortSignal.timeout(ms) })` from `node:stream/promises` inside `try`/`catch`: any failure, a premature close or the deadline destroys both streams, and the handler then answers with a 502 or 504 if the client is still there. Because `res` is not inside the pipeline, it is still usable for that answer.
code
javascript · 17 linesimport http from 'node:http';
import { pipeline } from 'node:stream/promises';
http.createServer(async (req, res) => {
const upload = http.request('http://127.0.0.1:9000/uploads/blob', { method: 'PUT' });
upload.on('response', (upRes) => {
upRes.resume(); // discard the storage response body
if (!res.headersSent) res.writeHead(upRes.statusCode < 300 ? 201 : 502).end();
});
try {
await pipeline(req, upload, { signal: AbortSignal.timeout(120_000) });
} catch (err) {
// req and upload are already destroyed; res is outside the pipeline.
console.error('upload failed:', err.name, err.code ?? '');
if (!res.headersSent) res.writeHead(err.name === 'AbortError' ? 504 : 502).end();
}
}).listen(8080);go deeper
Recall that a client disconnect stops the request stream and that pipe() does not close the stream it writes into when that happens.
Explain why req closes without 'end', why pipe() only ends its destination on 'end', and what pipeline destroys instead.
Diagnose the leak from socket counts and error codes, fix it with the promise form of pipeline plus a deadline, and keep res outside the pipeline so you can still answer.
Decide the proxy's failure budget: one end-to-end deadline per upload, which status each failure maps to, and an alert on rising premature-close rates.
## The setup An **upload proxy** receives a file in an HTTP request body and forwards it to object storage without writing it to local disk. In Node.js the incoming body `req` is a **Readable** stream and the outgoing upload (for example an `http.request()` with method `PUT`) is a **Writable**. The naive code connects them with `req.pipe(upload)` and responds when the upload finishes. Under normal traffic this works, and it even respects **backpressure**: when storage accepts data slowly, `pipe()` pauses `req`, TCP flow control slows the client, and memory stays flat. The trouble is entirely in the failure paths. ## What happens when the client disconnects 1. The client's connection drops mid-body (a closed laptop lid, a mobile network change, a cancelled upload). 2. Node's HTTP server destroys the in-flight `req` with an `'aborted'` error whose code is `ECONNRESET`. Node only emits that error as an `'error'` event when `req` has an `'error'` listener; otherwise `req` simply closes. 3. Either way, `req` emits `'close'` and never `'end'`. 4. `pipe()` ends its destination only when the source emits `'end'`, and Node documents that a destination is **not closed automatically** when the source errors. 5. So `upload` is never ended and never destroyed. The outgoing connection to storage stays open, holding a socket, buffers and whatever server-side state the half-finished upload holds, until an idle timeout on one side fires. Under steady traffic with a share of aborted uploads, those orphaned connections accumulate. Symptoms are a climbing count of open sockets and file descriptors, memory held by half-finished uploads, and eventually `EMFILE` errors or memory pressure. ## What happens when storage fails The opposite direction leaks too: - If `upload` errors (storage resets the connection), `pipe()` only **unpipes**. `req` is left paused, nobody reads it, and the client's upload stalls until the client or a server timeout gives up. - If nothing listens for `'error'` on `upload`, the error is an unhandled `'error'` event and **crashes the process**, taking every other in-flight request with it. ## The fix: one pipeline with a deadline ```js await pipeline(req, upload, { signal: AbortSignal.timeout(120_000) }); ``` `pipeline` from `node:stream/promises` changes every failure path: - it listens for errors and premature close on both streams, so a dropped client rejects the promise, typically with the `ECONNRESET` error or `ERR_STREAM_PREMATURE_CLOSE`; - on any failure it calls `destroy(err)` on each stream that has not already finished, so the upstream connection is torn down at once; - the `signal` bounds the whole transfer: when it aborts, `pipeline` destroys the chain and rejects with an `AbortError` whose `cause` is the signal's reason; - the promise settles once, so the handler has exactly one place to decide the response. ## Responding after the pipeline Keep `res` **out** of the pipeline. `pipeline` destroys every stream it owns on failure, so if `res` were the last stage, a storage error would close the client's socket before you could send a status. With `res` outside it: | Outcome | What the handler does | |---|---| | Pipeline resolves and storage answers 2xx | Reply 201 to the client | | Pipeline resolves, storage answers an error status | Reply 502 | | Rejects with an `AbortError` (deadline) | Reply 504 if `res` is still writable | | Rejects because the client disconnected | Log it; there is nobody to answer | Also consume or discard the storage response body (`upRes.resume()`), or that connection stays occupied too. ## Why not buffer the body instead Reading the whole upload into memory and sending it afterwards would sidestep the stream lifecycle, but it trades one problem for worse ones: - memory per upload grows to the file size, multiplied by concurrent uploads in one heap; - storage receives nothing until the client has finished, doubling the end-to-end time; - a disconnect still has to be detected and the half-read body discarded. Streaming with `pipeline` keeps memory near the streams' `highWaterMark` values and handles the disconnect for you. ## How to confirm it in production - Count open sockets or file descriptors per process over time and correlate with aborted requests. - Log every rejected pipeline with its error code: a burst of `ECONNRESET` or `ERR_STREAM_PREMATURE_CLOSE` matches the disconnect pattern, `AbortError` matches the deadline. - Reproduce locally by starting an upload with a rate-limited client and killing it halfway; with `pipe()` the upstream connection stays open, with `pipeline` it closes immediately.
- Why not add a req.on('close') handler that calls upload.destroy() and keep pipe()?It covers one direction only. You would also need an `'error'` listener on `upload` that destroys `req`, a check for `'close'` before `'end'`, care not to destroy after a normal finish, and a deadline. That is a hand-written `stream.pipeline`; using the real one gets the edge cases right and gives one promise to handle.
- What does the AbortSignal passed to pipeline do that a socket timeout on the upstream request does not?It bounds the whole transfer from start to finish, including a client that trickles bytes slowly enough never to trip an idle timeout. When it aborts, `pipeline` destroys every stream and rejects with an `AbortError`, so one deadline covers both sides.
- Does pipeline keep backpressure working between the client and storage?Yes. When the upstream request's `write()` returns `false`, `pipeline` stops reading `req` until `'drain'`, and because `req` is not read, TCP flow control slows the client. Memory per upload stays near the two streams' `highWaterMark` values rather than the file size.
- pipe() closes the destination when the source errors, so nothing can leak
- A disconnected client makes req emit 'end', so the upload finishes normally
- Putting res as the last stage of the pipeline is fine for sending a 502 afterwards
- An idle timeout on the upstream request is enough to bound a slow upload
- Buffering the whole body before uploading avoids the problem at no real cost
- An 'error' on the upstream request without a listener is harmless