When `pg_ctl stop -m smart` on a self-managed PostgreSQL inventory server reports "server does not shut down" during a maintenance window while idle pooled sessions stay connected, what is happening and how do you finish safely?
answer
- idle still counts as a session
- the client gave up, the server did not
- nobody new can get in
- a stronger signal upgrades it
basics
~20 sSmart mode waits for every session, idle pooled ones included, to disconnect; pg_ctl gave up after its 60-second wait but the server is still shutting down and refusing new connections. Escalate with pg_ctl stop -m fast.
solid answer
~40 sSmart shutdown (`SIGTERM`) refuses new client connections and waits until every existing session ends **on its own**. The application's pool keeps idle connections open, so that never happens. `pg_ctl` waited its default 60 seconds (`-t`), printed `server does not shut down` with a hint about `-m fast`, and exited 1, but that did **not** cancel anything: the postmaster is still in smart shutdown, serving the old sessions and refusing new ones with `FATAL: the database system is shutting down` (57P03). Finish with `pg_ctl stop -m fast`, which upgrades the shutdown in progress: open transactions roll back, sessions are terminated, a shutdown checkpoint is written and the next start is clean. Do not reach for `-m immediate` or `kill -9`. Next time, stop or drain the application first and use fast mode.
code
console · 10 lines$ pg_ctl stop -D /srv/pg/inventory -m smart
waiting for server to shut down............................................................ failed
pg_ctl: server does not shut down
HINT: The "-m fast" option immediately disconnects sessions rather than
waiting for session-initiated disconnection.
$ psql -h inv-db01 -d inventory -c 'SELECT 1'
psql: error: connection to server at "inv-db01" (10.0.4.12), port 5432 failed: FATAL: the database system is shutting down
$ pg_ctl stop -D /srv/pg/inventory -m fast
waiting for server to shut down.... done
server stoppedgo deeper
Recall that smart mode waits for every session to leave, and that fast mode, the default, disconnects them instead.
Explain why idle pooled connections stall smart mode, what pg_ctl's timeout means, and why new connections fail with 57P03 during the wait.
Demonstrate the recovery path: escalate to fast without kill -9, account for checkpoint and walsender time, and rewrite the runbook to drain clients first.
Design maintenance so the database stop is the last, boring step: application drained, pooler paused, checkpoint pre-run, and timeouts sized from measured stop times.
## What smart mode is waiting for In PostgreSQL, `pg_ctl stop -m smart` sends `SIGTERM` to the postmaster, which logs `received smart shutdown request`, marks `postmaster.pid` as `stopping`, and stops admitting ordinary client connections. It then waits until **every existing client session has disconnected by itself**. The manual's wording is that it "lets existing sessions end their work normally", and an idle session has no work to end, so it simply stays. Application connection pools are designed to keep connections open while idle, so on a typical inventory service the wait is unbounded. Only after the last session leaves does the server continue with the normal shutdown sequence and its checkpoint. ## What pg_ctl's failure does and does not mean `pg_ctl` waits by default and polls `postmaster.pid`. After the timeout (`-t`, 60 seconds unless `PGCTLTIMEOUT` says otherwise) it prints: - `pg_ctl: server does not shut down` - a hint that `"-m fast"` disconnects sessions instead of waiting for them, and exits with status 1. That exit is only `pg_ctl` giving up on **waiting**. The server is still in smart shutdown, and the state is the worst of both worlds: | What | State | |---|---| | Existing pooled sessions | still served normally | | New connections, including your own psql | refused: `the database system is shutting down`, SQLSTATE `57P03` | | Pool replacing a broken connection | refused, so capacity drains away | | Maintenance window | stalled until something changes | You can confirm the state from the shell: `pg_ctl status` still reports a running server, and the last line of `postmaster.pid` reads `stopping`, the status word the postmaster writes when any shutdown begins. The refusal has a practical twist: you cannot open a fresh session to run `pg_terminate_backend()` on the idle sessions. Only a session that was already open before the shutdown began can do that. ## Finishing the stop safely 1. **Escalate to fast.** Run `pg_ctl stop -D /srv/pg/inventory -m fast`. The postmaster upgrades a shutdown in progress to a stronger mode; it logs `received fast shutdown request` and `aborting any active transactions`, terminates the sessions, rolls back whatever they had open and writes a shutdown checkpoint. The next start needs no recovery. 2. **Or remove the clients.** Stopping the application or its pool lets the smart shutdown complete by itself. You can also end individual sessions from the operating system with `kill -TERM <backend pid>`, which the manual documents as the way to terminate one session while others continue. 3. **Keep immediate in reserve.** `-m immediate` would also end the wait, but it skips the checkpoint and forces crash recovery at the next start, truncating unlogged tables and resetting the statistics counters. It is for a fast shutdown that is itself stuck. 4. **Never `kill -9` the postmaster.** It cannot pass the signal on, leaves backends and shared memory behind, and the next start then fails until the orphans are killed. ## Why even the fast stop can take a while Fast mode does not wait for clients, but it still does real work before it exits: - the **shutdown checkpoint** writes every dirty buffer at full speed, and with `log_checkpoints` on by default the log shows `checkpoint starting: shutdown immediate` followed by its completion line; - the server does not fully stop until **walsenders** have sent the outstanding WAL to the standbys that are still connected. If `pg_ctl` times out again here, the shutdown is progressing, not stuck; give it a larger `-t`. ## Preventing it next time - Use **fast** mode, the default, in maintenance scripts; remove an explicit `-m smart` from old scripts. - **Drain the application first**: stop the service or pause the pooler, so no session is mid-transaction when the database goes down. - Run a manual `CHECKPOINT` a minute before the stop so the shutdown checkpoint has little left to write. - Pass a realistic `-t` so the script's own timeout reflects how long a stop actually takes on this server. - Check that the log shows `database system is shut down` and that `postmaster.pid` is gone before touching the data directory.
- Why can't you simply connect and run pg_terminate_backend on the idle sessions once the smart stop has begun?Smart shutdown refuses every new ordinary connection with `FATAL: the database system is shutting down` (57P03), including yours. Only a session opened before the shutdown started can call `pg_terminate_backend()`. Otherwise terminate the backends from the OS with `kill -TERM <pid>`, or escalate with `pg_ctl stop -m fast`.
- The escalated fast stop also takes several minutes. What is it doing?Writing the shutdown checkpoint, which flushes every dirty buffer at full speed, and waiting for walsenders to send outstanding WAL to connected standbys. The log shows `checkpoint starting: shutdown immediate`. A manual `CHECKPOINT` shortly before the stop and a larger `pg_ctl -t` make this predictable.
- What does a smart shutdown do on a hot standby?It refuses new connections and waits for the read-only sessions to end, and recovery and streaming replication keep running until they do. The standby therefore keeps replaying WAL during the wait; it stops only after the last regular session has gone.
- pg_ctl exiting after 60 seconds means the shutdown was cancelled.
- Escalating from smart to fast requires killing the postmaster first.
- Smart mode only waits for running queries, so idle sessions do not matter.
- kill -9 on the postmaster is the quick, safe fix for a hung stop.
- pg_ctl reload makes the server drop idle sessions so the stop can finish.