During a rolling deploy, a Node.js API calls server.close() on SIGTERM, yet it hangs for the whole grace period and some clients see connection resets; why?
answer
- busy at close, reused afterwards
- no automatic Connection: close
- other handles still hold the loop
- the reset comes from SIGKILL
- ask the process what is alive
basics
~20 sKeep-alive connections busy at close time stay open and keep carrying requests, and pools, consumers or intervals keep the event loop alive, so the process runs until SIGKILL cuts whatever is in flight and clients see resets.
solid answer
~40 s`server.close()` only closes connections that are idle at that instant. A keep-alive socket that was mid-request finishes its response and stays open, because Node does not add `Connection: close` just because the server is closing, and a busy load balancer or pooled client keeps reusing it, so the close callback never fires. Even once the server is closed, referenced handles such as database pool sockets, a queue consumer or a `setInterval` keep the event loop alive. The platform then sends `SIGKILL`, and every request still running is reset. The fix is to send `Connection: close` while draining, close pools and consumers after the server's callback, clear intervals, and arm an unref'd hard-exit timer below the grace period, using `process.getActiveResourcesInfo()` to see what still holds the loop.
code
javascript · 26 linesimport http from 'node:http';
import process from 'node:process';
let draining = false;
const server = http.createServer((req, res) => {
if (draining) res.setHeader('Connection', 'close');
res.end('ok');
});
server.listen(3000);
const refresh = setInterval(() => { /* refresh a cache */ }, 30_000);
process.once('SIGTERM', () => {
draining = true;
clearInterval(refresh);
const killTimer = setTimeout(() => {
console.error('still alive:', process.getActiveResourcesInfo());
server.closeAllConnections();
process.exit(1);
}, 20_000);
killTimer.unref();
server.close(() => {
console.log('http server closed');
// close pools and consumers here; the loop then empties
});
});go deeper
Know that a Node process exits only when nothing keeps its event loop alive, and that open sockets and intervals count.
Explain why a keep-alive connection busy at close time can keep receiving requests, and how Connection: close ends it.
Diagnose from evidence: log whether the close callback fires, dump getActiveResourcesInfo() at the deadline, and fix each holder in order.
Make drain duration and forced-exit counts a deploy health signal, so a service that always uses its full budget gets fixed before it drops traffic.
## The symptom A team adds a `SIGTERM` handler that calls `server.close()`. Deploys still go badly: old instances sit in a terminating state for the whole grace period, then die, and a slice of requests at every deploy fails with connection resets or 502s at the load balancer. Two separate mechanisms are usually at work, and they compound. ## Cause 1: busy keep-alive sockets outlive `close()` HTTP/1.1 clients and load balancers reuse connections with **keep-alive**: one TCP socket carries request after request. In Node 24, `server.close()` stops the listening socket and closes connections that are **idle at that instant**. Any connection that is mid-request when the signal arrives is left to finish, and here is the trap: - when its response completes, Node treats it as an ordinary keep-alive socket again; it does **not** add `Connection: close` because the server is closing; - the server arms the keep-alive timeout, `keepAliveTimeout` (default 5000 ms) plus `keepAliveTimeoutBuffer` (default 1000 ms, added in 24.6.0), and waits for the next request; - a busy upstream sends another request on that socket well inside six seconds, and the server serves it. Under steady traffic those sockets never go idle long enough to time out, so the `'close'` event and the close callback never fire, and none of the shutdown steps queued behind them ever run. ## Cause 2: other handles keep the event loop alive Even when the HTTP server does close, a Node process exits only when **no referenced handle** remains. Common survivors: 1. database or cache pool sockets that nobody closed; 2. a queue consumer's connection, still subscribed or polling; 3. a `setInterval` for metrics, cache refresh or heartbeats; 4. an `fs.watch` watcher or a child process left running. Any one of them keeps the process up until the platform escalates. ## Where the resets come from When the grace period ends the platform sends `SIGKILL`, which no listener can catch. Every socket is torn down by the kernel at once, so any request still being served is reset. A second, smaller source is the race inherent to keep-alive: a server closing an idle socket at the same moment a client sends a new request on it, which the client sees as `ECONNRESET`. That is why idle sockets should be closed by `Connection: close` on the last response, which tells the client not to reuse the socket, rather than only by destroying them. ## Diagnosing it - Log `process.getActiveResourcesInfo()` from the hard-exit timer before it calls `process.exit(1)`. It returns the type names of the resources keeping the loop alive, for example `'Timeout'` for a pending timer or a socket wrapper type for an open TCP connection, which points straight at the leftover handle. It is stable in Node 24. - Log when `server.close()`'s callback fires. If it never does, the cause is open HTTP connections; if it does and the process still lingers, the cause is something else. - Count forced exits at every deploy. A healthy service drains in seconds; one that always uses the full budget has a leak of this kind. ## The fix | Problem | Change | |---|---| | Busy keep-alive sockets reused | set `Connection: close` on every response while draining | | Close never finishes under load | at a deadline, `server.closeAllConnections()` | | Pools and consumers hold the loop | close them after the server's callback, in that order | | Intervals hold the loop | `clearInterval` them, or create them with `.unref()` | | Drain runs into `SIGKILL` | hard-exit timer, `.unref()`'d, a few seconds below the grace period | With those in place, a deploy drains in roughly the length of the slowest in-flight request, the process exits with a code you chose, and the only resets left are requests longer than the budget. Bounding those is a separate decision: the HTTP server's own timeouts and per-route limits decide how long a request may legally run. How the platform stops routing to the instance, including the race between endpoint removal and the signal, belongs to the orchestrator's termination material; the readiness flag and the drain are what the Node process contributes.
- The close callback fires within a second, but the Node process still lingers until SIGKILL. What do you check next?The HTTP server is no longer the cause, so something else holds a referenced handle. `process.getActiveResourcesInfo()` lists the resource types still alive: a `Timeout` entry points at an interval or timer, and a TCP socket type at an unclosed pool or consumer connection. Close each in the shutdown sequence, or `unref()` background timers that should never block exit.
- Why prefer sending Connection: close over calling closeIdleConnections() repeatedly during the drain?`Connection: close` on the final response tells the client not to reuse the socket, so it never sends a request into a socket the server is about to destroy. Destroying idle sockets from the server side can race a request the client has just sent, which the client sees as `ECONNRESET`. The header ends the conversation by agreement; the sweep ends it unilaterally.
- server.close() cuts keep-alive sockets as soon as their response ends
- Once server.close() is called no new request can reach the handler
- A closed HTTP server guarantees that the process exits
- Raising keepAliveTimeout makes the shutdown finish faster
- The resets come from server.close() itself, not from the final SIGKILL
Assumes Node 24: server.close() closes idle sockets since 19, keepAliveTimeoutBuffer (default 1000 ms) was added in 24.6.0, and process.getActiveResourcesInfo() became stable in 24.0.0.