skip to content

Draining Before Exit

Turning a kill signal into a clean exit: stop accepting work, finish what is in flight, close pools and consumers, and bail out if draining stalls. Asked because rolling deploys drop requests.

on this pageshow

explore

questions

5

How do you implement graceful shutdown on SIGTERM in a Node.js HTTP service, from refusing new work to the final exit?

level: middleimportance: must knowfreq 78%

answer

  1. order matters: inputs first, outputs last
  2. stop accepting, then finish in flight
  3. server.close callback before the pool
  4. an unref'd timer as the backstop

basics

~10 s

Catch SIGTERM, stop taking new work with server.close() and by pausing queue consumers, let in-flight requests finish, then close database pools, and arm an unref'd timer that calls process.exit(1) if draining stalls.

solid answer

~40 s

I register `process.once('SIGTERM', shutdown)`, and SIGINT for local runs, with a guard so it runs once. The handler first flips a `draining` flag so the readiness endpoint returns 503, and arms a hard-exit timer, `setTimeout(() => process.exit(1), budget).unref()`. Then it calls `server.close()`, which stops accepting connections and, in Node 24, also closes idle keep-alive connections; while draining, responses carry `Connection: close` so busy sockets end after their current response. Queue consumers stop fetching and finish the messages they hold. Only after the server's close callback fires and consumers are idle do I close database pools, because in-flight handlers still need them. Then I set `process.exitCode` and let the event loop empty.

code

javascript · 46 lines
javascript
import http from 'node:http';
import process from 'node:process';
import { setTimeout as sleep } from 'node:timers/promises';

const BUDGET_MS = 25_000;
const READINESS_DELAY_MS = 5_000;
let draining = false;

const server = http.createServer((req, res) => {
  if (draining) res.setHeader('Connection', 'close');
  if (req.url === '/ready') {
    res.statusCode = draining ? 503 : 200;
    res.end(draining ? 'draining' : 'ready');
    return;
  }
  res.end('hello');
});
server.listen(3000);

async function stopConsumers() { /* stop fetching, await in-flight handlers */ }
async function closePools() { /* end database and cache pools */ }

async function shutdown(signal) {
  if (draining) return;
  draining = true;
  console.log(`${signal}: draining`);
  const killTimer = setTimeout(() => {
    console.error('drain exceeded budget, forcing exit');
    process.exit(1);
  }, BUDGET_MS);
  killTimer.unref();
  try {
    await sleep(READINESS_DELAY_MS);
    await new Promise((resolve, reject) =>
      server.close((err) => (err ? reject(err) : resolve())));
    await stopConsumers();
    await closePools();
    process.exitCode = 0;
  } catch (err) {
    console.error('shutdown failed', err);
    process.exitCode = 1;
  }
}

process.once('SIGTERM', shutdown);
process.once('SIGINT', shutdown);

go deeper

for a junior

Know that SIGTERM needs a handler that calls server.close() and waits for it, rather than exiting straight away.

for a middle

Explain the order: stop inputs, drain in-flight work, close pools last, and why the hard-exit timer is unref'd.

for a senior

Cover the cases that hang or drop work: busy keep-alive sockets, consumers and intervals holding the loop open, and requests longer than the budget.

for a principal

Treat the sequence as a shared library with a fixed hook order so every service drains the same way and forced exits are logged and counted.

## Why the order matters A **graceful shutdown** turns a termination signal into a clean exit: no request is cut mid-response, no message is half-processed, and no connection to a database is dropped while a query is running. In a rolling deploy the platform sends `SIGTERM`, waits for a **grace period**, then sends `SIGKILL`. Everything below has to fit inside that window. The rule that drives the order is simple: **close inputs before outputs**. First stop new work arriving, then let the work already in flight finish, and only then close the resources that in-flight work depends on. ## The sequence, step by step 1. **Install the handler.** Node's default `SIGTERM` handler exits immediately, so you replace it with `process.once('SIGTERM', shutdown)` (and `SIGINT` for Ctrl+C during development). Installing a listener removes the default exit, so from here on the handler owns the process lifetime. 2. **Arm a hard-exit timer.** `setTimeout(() => { log; process.exit(1); }, budget)` with `.unref()`. The timer is the backstop if anything below stalls; `unref()` makes sure the timer itself is not what keeps the process alive after a successful drain. 3. **Fail readiness first (optional but common).** Flip a `draining` flag so the health endpoint the load balancer polls returns 503, and keep serving for a few seconds so the balancer stops routing new connections before the socket closes. How long the platform needs is a platform question; the Node side is the flag and the delay. 4. **Stop accepting HTTP connections.** `server.close(callback)` stops the listening socket. Since Node 19 it also closes **idle** connections, those not sending a request or awaiting a response. Connections that are busy stay open until they finish, and the callback fires only when every connection has ended. 5. **End busy keep-alive sockets.** Node does not add `Connection: close` to responses just because the server is closing. Setting it yourself while `draining` is true makes each busy socket close after its current response instead of waiting for more requests. 6. **Drain queue consumers.** Stop fetching or subscribing, let in-flight message handlers finish and acknowledge, and close the consumer connection. 7. **Close outbound resources.** After the server's callback and the consumers are done, close database pools, cache clients and telemetry exporters, flushing anything buffered. 8. **Exit.** Clear any `setInterval` you own, set `process.exitCode`, and let the loop empty. If something still holds it open, the hard-exit timer fires. ## What each step protects | Step | Skip it and you get | |---|---| | Replace the default handler | every deploy cuts in-flight requests at once | | Hard-exit timer | a stuck drain runs to `SIGKILL` with no log of why | | `server.close()` before the pool | new requests arrive while you tear things down | | Pool after the close callback | in-flight handlers fail with "pool is closed" style errors | | `Connection: close` while draining | busy keep-alive sockets keep the server open | ## A minimal implementation ```js import http from 'node:http'; import process from 'node:process'; import { setTimeout as sleep } from 'node:timers/promises'; let draining = false; const server = http.createServer((req, res) => { if (draining) res.setHeader('Connection', 'close'); res.end('ok'); }); async function shutdown() { draining = true; setTimeout(() => process.exit(1), 25_000).unref(); await sleep(5_000); // let the load balancer see 503 on readiness await new Promise((resolve, reject) => server.close((err) => (err ? reject(err) : resolve()))); // then: stop consumers, close pools, flush telemetry } ``` ## Common mistakes - **Calling `process.exit()` right after `server.close()`.** `close()` returns immediately; the drain happens later, so this cuts every in-flight request, and it can also truncate log output still being written. - **Closing the pool first.** Requests still running on open connections fail. - **A ref'd hard-exit timer.** Without `unref()`, a drain that finishes in two seconds still waits the full budget before exiting. - **Forgetting background timers and consumers.** A `setInterval` or an open consumer connection keeps the event loop alive after the server has closed, so the process hangs until `SIGKILL`. The platform's own side of this, such as removing the pod from endpoints or a pre-stop delay, belongs to the orchestrator material; what is described here is what the Node process itself does.

  • In this Node.js shutdown sequence, why is the hard-exit timer unref'd instead of being cleared at the end?
    An unref'd timer never keeps the event loop alive, so once the server, consumers and pools are closed the process exits naturally even though the timer is still scheduled. Clearing it would also work, but only on the paths you remembered; `unref()` is correct on every path, including a drain that finishes through an unexpected route.
  • Why does the handler wait for the server.close() callback rather than closing the database pool right after calling server.close()?
    `server.close()` returns immediately; it only stops new connections. Requests already running are still executing and still need their database connections. The callback fires when every connection has ended, so it is the first point at which no request handler can still be using the pool.
  • What should the shutdown handler do with a request that is still running when the hard-exit timer fires?
    Nothing more can be saved at that point: `process.exit(1)` ends the process and the client sees a reset. The fix is upstream: bound request duration with `server.requestTimeout` or per-route timeouts so the longest legal request fits in the budget, and log the forced exit so it shows up as an incident rather than silence.

Closing a restaurant for the night: lock the front door, let the seated diners finish, and only then switch off the kitchen, while the manager's watch sets a last call after which everyone is asked to leave.

  • Call process.exit right after server.close to finish quickly
  • server.close waits for in-flight requests before it returns
  • Close the database pool first so no new queries start
  • A hard-exit timer is unnecessary because server.close always finishes
  • Node adds Connection: close to responses once the server is closing

Assumes Node 24; since Node 19, server.close() also closes idle keep-alive connections, which earlier releases needed closeIdleConnections() for.

open as a page

In Node.js, what happens when the process receives SIGTERM or SIGINT with no listener, and what changes once you call process.on('SIGTERM')?

level: juniorimportance: should knowfreq 52%

basics

~20 s

With no listener, Node's default handler exits at once with code 128 plus the signal number, cutting off in-flight work. Installing a listener removes that default: Node stops exiting, so the handler must drain and end the process itself.

open as a page

In a Node 24 HTTP server, what does server.close() wait for, and how do closeIdleConnections() and closeAllConnections() differ from it?

level: middleimportance: should knowfreq 55%

basics

~10 s

server.close() stops new connections, closes idle ones, and calls back only when every connection has ended. closeIdleConnections() destroys only idle connections; closeAllConnections() destroys active ones too, except upgraded sockets such as WebSocket.

open as a page

During a rolling deploy, a Node.js API calls server.close() on SIGTERM, yet it hangs for the whole grace period and some clients see connection resets; why?

level: seniorimportance: should knowfreq 58%

basics

~20 s

Keep-alive connections busy at close time stay open and keep carrying requests, and pools, consumers or intervals keep the event loop alive, so the process runs until SIGKILL cuts whatever is in flight and clients see resets.

open as a page

As the lead for several Node.js HTTP services, which shutdown defaults would you standardise so each one drains within its deploy budget, and what does each choice cost?

level: principalimportance: should knowfreq 30%

basics

~20 s

Standardise one shutdown module: a hard-exit timer below the platform's grace period, a fixed close order, Connection: close while draining, request time limits that fit the budget, and a rule for long-lived connections. Each trades deploy speed against dropped work.

open as a page