A `systemctl reload haproxy` runs `haproxy -f haproxy.cfg -p /run/haproxy.pid -sf $(cat /run/haproxy.pid)`. What does the `-sf` flag do to the old process and its in-flight connections, and what can still be lost across that reload?

## What actually happens on reload HAProxy has no in-place config re-read. A "reload" is a *new process* started from the config on disk, plus an orderly handover from the old one. The flags decide how brutal the handover is: ```bash haproxy -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid -sf $(cat /run/haproxy.pid) ``` - **`-sf <pids>`** — *soft finish*. The new process signals the listed PIDs to stop accepting new connections and to finish the sessions they already have, then exit on their own. - **`-st <pids>`** — *stop*. The listed processes are terminated immediately, dropping whatever they were serving. Reserve this for when a config emergency outweighs the in-flight requests. Because the old workers linger until their sessions end, you will normally see two (or more) HAProxy processes for a while after a reload. That is expected, not a leak — up to a point, discussed below. ## Why no connection is refused The risky moment is between "old process stops listening" and "new process is listening". Two mechanisms close that window. In **master-worker** mode (`master-worker` in `global`, or `-W`) with `expose-fd listeners` declared on the stats socket, the new process asks the old one for its listening file descriptors and inherits them directly. The socket is never closed, so the kernel's accept queue is continuous and clients see nothing at all. ```haproxy global master-worker stats socket /var/run/haproxy/admin.sock mode 660 level admin expose-fd listeners hard-stop-after 60s ``` Even without descriptor transfer, HAProxy does not simply close and reopen. On Linux it *pauses* the listeners — the socket stays bound, so incoming connections accumulate in the backlog rather than being refused — and if the new process fails to start, it un-pauses them and carries on. A config syntax error therefore does not take the site down, which is why `haproxy -c -f haproxy.cfg` before reloading is a cheap and worthwhile check. ## What is lost anyway The handover moves sockets and lets sessions drain. It does not move memory. Three things routinely surprise people: **1. Runtime API state.** A server you drained with `set server web/app1 state drain` is an ordinary configured server as far as the new process is concerned; it rejoins the pool the instant the reload lands. Mid-deploy this is exactly wrong. **2. Health-check history.** A freshly started process has never checked anything. Servers begin in their configured state and are eligible for traffic before the first probe result arrives, so a reload during an incident can send requests to a machine the previous process had already evicted. This is the mechanism behind "we get a burst of 5xx every time we reload". Both are solved by the server-state file: ```bash echo "show servers state" | socat stdio /var/run/haproxy/admin.sock > /var/lib/haproxy/server-state ``` ```haproxy global server-state-file /var/lib/haproxy/server-state defaults load-server-state-from-file global ``` The dump belongs in the service unit's `ExecReload`/`ExecStop` so it is never forgotten. With it, administrative state, operational UP/DOWN status and weights carry across. **3. Anything else counted in memory.** In-memory tables and counters held by the old workers are not merged into the new process; whatever is not persisted or replicated is simply gone with the old PIDs. ## Old processes that will not leave `-sf` waits for sessions to end — and some sessions do not end. WebSocket connections, server-sent event streams and long-polling clients can hold an old worker for hours. Deploy every twenty minutes and you accumulate a stack of old processes, each with its own memory footprint and each still serving the *previous* configuration. ```haproxy global hard-stop-after 60s ``` `hard-stop-after` sets the maximum time a worker may keep running after being told to stop; when it expires, remaining sessions are cut. Choosing the value is the usual trade — long enough for normal requests to finish, short enough that reload frequency does not produce process pile-up. Frontends serving long-lived streams should expect their clients to reconnect, and clients should be written to do so. ## The checklist worth reciting Validate with `-c` first; reload with `-sf`; run master-worker with `expose-fd listeners` so the socket handover is seamless; dump and load the server-state file so drains and health status survive; set `hard-stop-after` so old workers cannot accumulate; and expect long-lived connections to be the thing that makes a "hitless" reload visible to somebody.