When should a Go service supervise a crashed worker in-process, and when should it exit and let the platform restart it?

## Two restart machines, one service Every long-running Go service that supervises workers has two restart mechanisms available, and the interesting decision is where the boundary between them sits. **In-process supervision.** A supervisor goroutine relaunches the worker function. Cost: microseconds. Blast radius: one worker. Everything else in the process — the other device sessions, warm connection pools, in-memory routing state, the admin surface — keeps running untouched. **Process restart.** The process exits and whatever started it starts it again. Cost: seconds, plus whatever it takes to rebuild every session. Blast radius: everything. But it is the only mechanism that reclaims anything the process has irreversibly damaged. Neither is a default you can apply everywhere. Choosing per failure class is the job. ## When in-process wins The failure is **local and identified**: this device is refusing connections, this stream sent a frame the parser rejects. Nothing about it says the rest of the process is unsound. The process holds **expensive state**. A session manager with several thousand live sockets is the clearest case: restarting the process means every peer reconnects at once, and that reconnect storm can be worse for the fleet than the original fault — especially if the peers themselves retry aggressively. Keeping the process alive is what protects the ninety-nine percent that were fine. The restart is **cheap and complete**: relaunching the worker function genuinely re-establishes it, with no residue. That is a claim you should be able to defend, not assume. ## When exiting wins The **residue accumulates**. Each crashed worker leaves something behind — a goroutine blocked on a socket that was never closed, a file descriptor, an entry in a map keyed by session. Restarting in-process a thousand times leaves a thousand copies. If the goroutine count climbs in lockstep with the restart counter, in-process restarting is converting a fault into a slow leak, and only a fresh process reclaims it. The **damage is not provably local**. Shared mutable state that a worker may have left half-updated, a global cache the failure could have poisoned, an initialisation the code cannot redo idempotently. A process restart is a very cheap way to get back to a state you can reason about; "restart the goroutine and hope" is not. The fault is **environmental and only fixed by starting over**: stale configuration read at startup, a rotated credential, a dependency whose address changed. Retrying inside the same process cannot pick up any of those. The **budget is spent**. Past a cap of consecutive fast failures, in-process retry has stopped being a recovery strategy and become a way of hiding a fault. ## The organisational half, which is what makes this a judgment call A supervisor that retries forever is a decision to make a fault invisible. From the outside the process is up; its health endpoint answers; the platform's restart count is zero; no page fires. Degradation quietly becomes the steady state and nobody is accountable for it. That is why an unbounded retry loop is a governance problem, not just a technical one. The mirror image is equally real: a process that exits on the first error hands every fault to a restart policy tuned by a different team, with a different backoff, and a single flaky peer can drive a restart cycle that drops all the healthy sessions repeatedly. So the posture I argue for, and the one that survives being overruled: 1. **Bound in-process restarts.** A cap on consecutive fast failures per worker, with capped backoff between attempts. Named constants, not literals buried in a loop. 2. **Always export the restart rate.** Per worker, alerted on rate rather than total, alongside current-run uptime and the last exit error. If a crash loop can run without a page, the design is not finished. 3. **Escalate deliberately when the cap is spent.** Either mark that worker dead and keep serving the rest — an explicit, visible, alerted state — or return the error up so `main` exits non-zero. Both are defensible; drifting into infinite retry is not. 4. **Write the handoff down.** The platform team owns process-level restart pacing and the paging policy; the service team owns per-worker restart pacing. The cap is where those two meet, and it belongs in a runbook with the alert threshold beside it, because the on-call engineer needs to know which machine is currently restarting things and how to stop it. ## How I would present the tradeoff Framing it as a comparison of two numbers usually settles the argument fast: the cost of one process restart, measured in reconnects and seconds of unavailability across every session, against the cost of tolerating one degraded worker for the time it takes a human to respond. When the first number is large and the fault is local, supervise. When the first number is small, or the fault could be anywhere, exit — the fresh process is more trustworthy than any recovery you write by hand.