For a Go service hosting other teams' handlers, do you recover panics into 500s or let the process crash?

## The question behind the question This is not "is recover good". It is: when your process discovers that one of its own invariants was violated, do you keep serving or do you stop? On a service that hosts handler code you did not write and cannot fix, the answer is a policy, and the person carrying the pager has a legitimate veto. ## Start from Go's default, because it is already a choice `net/http` recovers panics at the connection layer. So a Go HTTP service **already** survives handler panics with no code from you. "Let it crash" is therefore not the lazy option; it is an active decision you implement by re-panicking out of your recoverer, or by calling `os.Exit` after logging. State this early — candidates who think they are choosing between crashing and recovering have the baseline wrong. ## The case for recovering into a 500 - **Blast radius.** A panic is normally one request meeting one unhandled shape — a nil map write on an unusual payload, an index computed from untrusted input. Crashing converts a one-request bug into an outage for every in-flight request of every other team on the box. - **Poison requests.** If a client retries the request that panicked, crash-only turns into a crashloop, and an orchestrator's backoff makes the outage longer than the bug deserves. - **Observability.** Recovering lets you attach the route, the request id and the owning package to the failure. A crash gives you a traceback on stderr and nothing else correlated. - **The client gets an answer.** A 500 is a status your callers, your load balancer and your SLO maths can all reason about. A dropped connection looks like a network fault and pollutes someone else's error budget. ## The case for crashing - **A panic means you were wrong about something.** The unwinding skipped every cleanup that was not deferred. A `sync.Mutex` unlocked on the happy path only is now locked forever, so the next request touching it hangs — a far worse failure than a restart, and much harder to diagnose. - **Shared mutable state.** If the panicking handler was midway through mutating a process-wide cache, an in-memory index or a file, continuing can serve wrong answers silently. Wrong answers beat down trust faster than 500s do. - **Restarting is cheap and honest.** With a fast start-up, a readiness probe and connection draining, an orchestrator restart is measured in seconds and gives you a known-good state. - **Recovering makes panics free.** If a panic costs nobody anything, the contributing team never fixes it, and the recoverer quietly becomes the error-handling strategy. ## The posture I would actually own 1. **Recover by default, at the edge, exactly once.** One recoverer for the whole platform, not one per team, so the behaviour is uniform and auditable. 2. **Make every panic expensive.** A structured high-severity event with route, method, panic value, stack, and the package that appears first in the trace — which names the owning team. A panic without an owner is a platform bug by default. 3. **Alert on rate, not on instances.** One panic per week is a ticket; a route panicking on 5% of requests is a page. Give panics an error budget like any other failure class. 4. **Shed before you crash.** If a single route is panicking above a threshold, disable that route and return 503 for it. That contains a broken tenant without punishing the rest, and it is a much better lever than process restart. 5. **Crash-only for a named list.** Handlers holding process-wide mutable state whose consistency you cannot verify after an abort. Write down which ones, and why, so the decision is reviewable rather than folklore. 6. **Accept what you cannot recover.** Fatal runtime errors — concurrent map writes, the deadlock detector, out of memory — end the process no matter what you decide, and panics in goroutines a handler spawns do too. Your policy must cover being restarted anyway: fast startup, drain on shutdown, no work lost that was not already durable. 7. **Agree the operational contract.** What happens after N panics per minute? Does the instance fail readiness and get rotated — which gets you a clean restart *and* graceful draining, the best of both — or does it page a human? That is SRE's call as much as yours, and it should be in the runbook, not in a middleware nobody has read. ## What a weak answer sounds like "Always recover, a 500 is fine" — ignores that the process may be unsound. "Crash-only is correct, that is how Erlang does it" — borrows a supervision model Go does not have; Go has no per-goroutine supervisor and no isolated heaps, so a crash is process-wide, not actor-scoped. "It depends" with no criteria — the criteria are blast radius, state soundness, restart cost and who is paged. ## The closing line Recover so that one request's bug stays one request's bug; make the recovery loud enough that it gets fixed; and reserve crashing for the cases where continuing would mean serving answers you cannot defend.