skip to content

With Node's cluster module, how should the primary replace crashed workers and roll all workers onto new code without refusing connections?

level: seniorimportance: should knowfreq 48%

answer

  1. Node respawns nothing
  2. the 'exit' event in the primary
  3. exitedAfterDisconnect: planned or crash
  4. fork first, then retire one
  5. disconnect() plus a kill timer

basics

~20 s

In cluster's 'exit' handler, fork a replacement unless worker.exitedAfterDisconnect is true, and stop after repeated crashes. To roll, fork a new worker, wait for 'listening', then disconnect() one old worker with a kill() timer, one at a time.

solid answer

~40 s

Node does not manage the worker count, so the primary does. In `cluster.on('exit', (worker, code, signal) => ...)` check `worker.exitedAfterDisconnect`: `true` means a planned `disconnect()`, anything else is a crash worth logging and replacing with `cluster.fork()`, behind a crash-loop limit. For a rolling restart, take one old worker at a time: fork its replacement, which loads the current code from disk, wait for the new worker's `'listening'` event, send the old one a shutdown message, then call `old.disconnect()`. That closes its servers, so it gets no new connections, and lets existing ones finish. Long-lived WebSockets can stall it, so a timer calls `old.kill()`. The primary itself cannot be rolled, and if it dies its workers exit immediately.

code

javascript · 48 lines
javascript
import cluster from 'node:cluster';
import http from 'node:http';
import { availableParallelism } from 'node:os';
import process from 'node:process';

if (cluster.isPrimary) {
  let crashes = [];
  for (let i = 0; i < availableParallelism(); i++) cluster.fork();

  cluster.on('exit', (worker, code, signal) => {
    if (worker.exitedAfterDisconnect) return; // planned: disconnect() was called
    const now = Date.now();
    crashes = crashes.filter((t) => now - t < 60_000);
    crashes.push(now);
    if (crashes.length > 5) {
      console.error('crash loop: not respawning');
      return;
    }
    console.error(`worker ${worker.id} died (${signal || code}), forking`);
    cluster.fork();
  });

  async function rollingRestart() {
    for (const old of Object.values(cluster.workers)) {
      if (!old.isConnected()) continue; // already leaving or gone
      const fresh = cluster.fork();
      await new Promise((resolve) => fresh.once('listening', resolve));
      old.send({ cmd: 'shutdown' }); // let the app close long-lived sockets
      old.disconnect(); // stop new connections, wait for open ones
      const timer = setTimeout(() => old.kill(), 10_000);
      await new Promise((resolve) => old.once('exit', resolve));
      clearTimeout(timer);
    }
  }

  process.on('SIGUSR2', () => {
    rollingRestart().catch((err) => console.error(err));
  });
} else {
  http
    .createServer((req, res) => res.end(`worker ${cluster.worker.id}\n`))
    .listen(8000);
  process.on('message', (msg) => {
    if (msg?.cmd === 'shutdown') {
      // close WebSocket connections here with a "reconnect" frame
    }
  });
}

go deeper

for a junior

Remember that the primary must listen for 'exit' and call cluster.fork() itself; nothing comes back on its own.

for a middle

Explain what exitedAfterDisconnect records, why the replacement is forked and listening before the old worker disconnects, and how disconnect() differs from kill().

for a senior

Show the production details: a crash-loop guard, a kill timer for WebSockets, and draining workers when the primary is told to stop.

for a principal

Decide whether in-process cluster supervision is worth owning, given a primary that cannot roll itself and is a single point of failure.

## Node supervises nothing by itself The cluster module gives the primary the tools to supervise workers, but the docs are explicit that **Node.js does not automatically manage the number of workers**. If a worker crashes and the primary has no handler, the cluster simply runs one worker short. If all of them die, existing connections are dropped and new ones are refused while the primary carries on. Restarting and redeploying are code you write in the primary. ## The lifecycle events the primary sees | Event on `cluster` | Means | |---|---| | `'fork'` | `cluster.fork()` created the worker process | | `'online'` | the worker is running and reported in | | `'listening'` | the worker's server called `listen()` and is ready for connections | | `'disconnect'` | the worker's IPC channel closed | | `'exit'` | the worker process ended; receives `(worker, code, signal)` | The same events exist on each `Worker` object, for example `worker.once('listening', ...)`. A gap between `'disconnect'` and `'exit'` is a useful signal that a worker is stuck in cleanup or holding long-lived connections. ## Telling a crash from a planned exit In the `'exit'` handler, `worker.exitedAfterDisconnect` is `true` when the worker left because `worker.disconnect()` was called, and `false` when it exited any other way. That is the flag to branch on: respawn on a crash, do nothing on a planned retirement. The exit `code` alone cannot tell you this, because a worker killed by a signal reports the `signal` instead, and a worker whose IPC channel closes unexpectedly exits with code 0. In Node 24's primary, the flag is set by `disconnect()`; calling `worker.kill()` on its own sends the signal without setting it, so call `disconnect()` first whenever you want an exit read as planned. A restart policy needs two more things: - **A crash-loop guard.** A worker that throws during startup will die as fast as you fork it. Count recent exits and stop respawning, or back off, past a threshold. - **Diagnostics.** Log `worker.id`, `code` and `signal` on every unplanned exit so a crash is visible rather than silently healed. ## A rolling restart, step by step `cluster.fork()` starts a brand-new process from `cluster.settings.exec`, so a new worker loads whatever code is on disk now. That makes a rolling deploy possible without dropping the port: 1. **Fork the replacement first** with `cluster.fork()`. 2. **Wait for its `'listening'` event**, so it is accepting connections before capacity is removed. 3. **Tell the old worker it is leaving**, for example `old.send({ cmd: 'shutdown' })`, so the application can close long-lived connections politely. 4. **Call `old.disconnect()`.** In the worker this closes all its servers, waits for their `'close'` events, then disconnects the IPC channel. The primary stops handing it connections at once; existing connections are allowed to finish. 5. **Start a kill timer.** If the old worker has not exited within your budget, `old.kill()` sends `SIGTERM` (the default signal) without waiting. 6. **Wait for `'exit'`, clear the timer, move to the next worker.** One at a time keeps capacity at N or N+1 throughout. Triggering this from a signal handler in the primary, such as `process.on('SIGUSR2', ...)`, is common. Avoid `SIGUSR1`, which Node reserves for starting the debugger. ## Long-lived connections `worker.disconnect()` waits for server connections to close, and only server connections: client connections the worker opened are not closed for you. An HTTP server's `close()` closes idle keep-alive connections in current Node, but a WebSocket is never idle in that sense, so a chat worker can sit in `disconnect()` indefinitely. The docs recommend exactly the pattern above: send the worker a message so it can end those connections, and kill it if `'disconnect'` has not arrived after a timeout. The finer points of draining an HTTP server belong to the lesson on graceful shutdown. ## What a rolling restart cannot do - **Reload the primary.** The primary is the process running the restart; changes to primary code need a full restart. - **Survive the primary's death.** If the primary exits, each worker sees its IPC channel close without a planned disconnect and calls `process.exit(0)` immediately, cutting in-flight requests. Handle the shutdown signal in the primary and drain workers with `cluster.disconnect()` first. - **Change settings for running workers.** `cluster.setupPrimary()` only affects future forks. - **Fix shared state.** Two code versions serve traffic side by side during the roll, so whatever they share must be readable by both.

  • What happens to the workers if the primary process itself dies?
    Each worker's IPC channel closes. Because no planned disconnect was requested, the worker calls `process.exit(0)` straight away, cutting its in-flight requests, and with no workers left the port stops accepting. Handle the shutdown signal in the primary and call `cluster.disconnect()` so workers drain before the primary exits.
  • Why does a rolling restart pick up new code in workers but not in the primary?
    Each `cluster.fork()` starts a fresh Node process that loads the entry file from disk at that moment, so replacements run the new code. The primary has already loaded its modules and is the process running the restart. A change to primary logic, or to anything it loaded, needs a full restart.
  • Why can worker.disconnect() never finish on a WebSocket server, and what do you do?
    `disconnect()` waits for the worker's servers to close, and a server does not close while connections remain. WebSockets are long-lived, so the worker can wait forever. Send the worker a message first so it can close them with a reconnect hint, and set a timer that calls `worker.kill()` if the worker has not gone in time.

  • Node respawns crashed cluster workers automatically.
  • worker.kill() waits for open connections to finish before exiting.
  • worker.disconnect() drops every open connection immediately.
  • Killing all old workers at once and forking new ones causes no downtime.
  • A cluster rolling restart also reloads the primary's own code.
  • Respawning instantly on every exit is safe even for startup crashes.