skip to content

In PostgreSQL, why does kill -9 on one backend process disconnect every other session on the server?

level: middleimportance: must knowfreq 55%

answer

  1. shared state can no longer be trusted
  2. exit status other than 0 or 1
  3. SIGQUIT to every other child
  4. reinitialise shared memory, replay WAL
  5. SIGTERM ends one session cleanly

basics

~20 s

A backend killed by SIGKILL may have died holding a lock or halfway through changing shared memory, so the postmaster treats it as a crash: it stops every other process, reinitialises shared memory and runs crash recovery.

solid answer

~40 s

The postmaster judges each child by its exit status: 0 is a normal exit and 1 a FATAL end of a session, but anything else, including death by SIGKILL, counts as a crash. A killed backend may have held a spinlock or lightweight lock, or been halfway through changing a buffer or the lock table, so nothing in shared memory can be trusted. The postmaster logs `terminating any other active server processes`, sends the other children SIGQUIT (SIGKILL if they have not exited after 5 seconds), reinitialises shared memory and starts crash recovery; every other client is disconnected with SQLSTATE 57P02 and must reconnect. `restart_after_crash` (default `on`) is what makes the restart automatic. To end one session safely, use `pg_terminate_backend(pid)`, which sends SIGTERM and exits cleanly.

code

console · 7 lines
console
LOG:  client backend (PID 48211) was terminated by signal 9: Killed
DETAIL:  Failed process was running: SELECT hotel_id, sum(nights) FROM bookings GROUP BY hotel_id;
LOG:  terminating any other active server processes
LOG:  all server processes terminated; reinitializing
LOG:  database system was interrupted; last known up at 2026-10-05 02:10:41 UTC
LOG:  database system was not properly shut down; automatic recovery in progress
LOG:  database system is ready to accept connections

go deeper

for a junior

Remember that kill -9 on any backend resets the whole server, and that pg_cancel_backend and pg_terminate_backend are the tools for a single session.

for a middle

Explain the exit-status rule, why shared memory cannot be trusted after an abnormal exit, and each step of the reset through crash recovery.

for a senior

Link resets to OOM-killer incidents and extension crashes, read the log sequence during an outage, and know when restart_after_crash = off suits an HA manager.

for a principal

Treat shared memory as a blast-radius decision: one faulty extension or memory spike takes every session down, which shapes extension vetting and per-host memory limits.

## How the postmaster reads a child's exit PostgreSQL runs every client session in its own **backend** process, forked by the **postmaster**. When any child exits, the postmaster inspects its exit status: - **exit status 0**: a normal exit, for example a client that disconnected; - **exit status 1**: a FATAL error that ended the session cleanly, for example after `pg_terminate_backend()`; - **anything else**: a crash. Death by a signal such as SIGKILL (`kill -9`), SIGSEGV from a buggy extension, or SIGABRT from a failed assertion all land here. For the first two, the postmaster just releases the child's slot and other sessions never notice. For the third, it assumes the worst about shared memory. ## Why one dead backend poisons all of shared memory All server processes attach the same **shared memory**: the buffer pool, the lock table, the array of backend slots, WAL buffers. A backend updates these structures under spinlocks and lightweight locks. A process killed by SIGKILL gets no chance to run cleanup code, so it may have: - died while holding a spinlock or lightweight lock that nobody will ever release; - left a buffer header or a lock-table entry half updated; - been in the middle of copying a WAL record into the WAL buffers. No other process can tell which of these happened. The only safe course is to discard shared memory entirely and rebuild it from what is durable on disk: the data files plus the WAL. This is why the postmaster deliberately never touches shared memory itself; it has to stay healthy to perform the reset. ## The reset sequence 1. The postmaster logs the child's death, for example `client backend (PID 48211) was terminated by signal 9: Killed`, then `terminating any other active server processes`. 2. It sends SIGQUIT to every other child. Each backend aborts its transaction and its client receives the warning `terminating connection because of crash of another server process` with SQLSTATE `57P02` (`crash_shutdown`). Children still alive after 5 seconds get SIGKILL. 3. Once all children are gone it logs `all server processes terminated; reinitializing` and recreates shared memory. 4. A `startup` process runs **crash recovery**, replaying WAL from the last checkpoint; new connections are refused with `the database system is in recovery mode` meanwhile. 5. The server logs `database system is ready to accept connections`. Committed transactions survive, because their WAL was flushed; every in-flight transaction on the server is rolled back, not just the killed one. On a busy hotel-booking system that means every open booking attempt fails at once, and the outage lasts as long as replay takes, which depends on how much WAL has accumulated since the last checkpoint. Connection pools in the application then reconnect in a burst, which is worth expecting in the runbook. ## What triggers it besides kill -9 - The **Linux OOM killer** ends its victim with SIGKILL, so a backend it picks causes exactly this reset. - A segmentation fault in a C extension or a PL with native code. - An operator who sends SIGKILL to a stuck autovacuum worker or another helper; most child types are handled the same way. ## Ending one session safely | Tool | Signal | Effect | |---|---|---| | `pg_cancel_backend(pid)` | SIGINT | Cancels the current query; the session stays connected | | `pg_terminate_backend(pid)` | SIGTERM | Ends that session with SQLSTATE `57P01` and exit status 1; nobody else is affected | | `kill -9 <pid>` | SIGKILL | Treated as a crash: all sessions are reset and crash recovery runs | If a backend does not react to SIGTERM immediately, it is usually inside a section that holds off interrupts; waiting, or finding what it is blocked on, is still far cheaper than a server-wide reset. Finding the right PID in `pg_stat_activity` is the monitoring topic's ground. ## restart_after_crash and clusterware `restart_after_crash` defaults to `on` and can be changed with a reload. With it on, the sequence above runs automatically. With it off, the postmaster stops the other children, logs `shutting down because "restart_after_crash" is off` and exits instead of reinitialising; the manual suggests this when PostgreSQL runs under clusterware, so the HA manager decides whether to restart in place or fail over. A separate developer option, `send_abort_for_crash`, sends SIGABRT instead of SIGQUIT so that each child leaves a core file for debugging; it does not stop the restart.

  • What does PostgreSQL do after a backend crash when restart_after_crash is off?
    It still stops all the other children, because shared memory is just as untrustworthy, but it then exits instead of reinitialising. The manual suggests this when PostgreSQL runs under clusterware, so the HA manager can decide whether to restart the server or fail over to a standby. The setting changes with a reload.
  • Why can the Linux OOM killer cause the same server-wide outage as an operator's kill -9?
    The OOM killer ends its victim with SIGKILL, so the postmaster sees a backend that died from a signal and runs the full reset. The usual direction is to stop the kernel from overcommitting memory, keep the postmaster itself out of the OOM killer's reach, and size per-backend memory so the host is not exhausted; the kernel and sizing details are their own topics.
  • Does pg_terminate_backend() ever trigger PostgreSQL's crash reset?
    No. It sends SIGTERM; the backend aborts its transaction, reports FATAL with SQLSTATE 57P01 and exits with status 1, which the postmaster treats as a normal end of session. Only that session is affected. It can take a moment when the backend is in a section that holds off interrupts, which is why impatient operators reach for kill -9, the one tool that hurts everyone.

  • kill -9 on a backend is the quickest safe way to stop one runaway query.
  • Only the killed session's transaction is rolled back; other sessions carry on.
  • The postmaster simply restarts the one backend that was killed.
  • pg_terminate_backend and kill -9 are equivalent; both send SIGKILL.
  • No recovery is needed after the reset because shared memory is rebuilt from scratch.