How do you stop one subscriber's panicking callback from killing a webhook dispatcher process?

## The situation A dispatcher holds a registry of callbacks supplied by other packages — one per subscriber — and runs them when an event arrives. It starts a goroutine per dispatch so a slow subscriber does not hold up the rest. Then one subscriber's callback dereferences a nil pointer and the entire dispatcher process dies, dropping every delivery in flight for every subscriber. The crash log has one panic in it; the platform records a restart. Because a panic anywhere terminates the process, and because a recover only works on the goroutine that panicked, the fix must live inside the goroutine body. ## The wrapper ```go type result struct { sub string err error } func runOne(sub string, h func(), out chan<- result) { defer func() { if r := recover(); r != nil { out <- result{sub: sub, err: fmt.Errorf("handler panicked: %v", r)} } }() h() out <- result{sub: sub} } ``` Three properties make this work: 1. **The deferred function is registered by the goroutine that will run `h`.** `go runOne(sub, h, out)` puts the recover on the right stack. Nothing the dispatcher writes around the `go` statement could do this. 2. **The recovered value becomes an ordinary value.** A panic cannot be returned; a value can. The dispatcher learns about the failure the same way it learns about a successful delivery. 3. **Exactly one send happens on every path.** If `h` panics, the trailing send is skipped and the deferred send happens instead. If a collector is counting one result per subscriber, that invariant is what stops it from blocking forever or double-counting. ## Placement: per unit of work, not per worker The most common half-fix is a single deferred recover at the top of a long-lived worker goroutine that loops over callbacks: ```go go func() { defer func() { _ = recover() }() // wrong granularity for job := range jobs { job.handler() } }() ``` This prevents the process from dying, but deferred functions run when the function **returns**, and a panic unwinds the whole frame including the loop. The recover therefore fires as the worker exits, and the worker is gone: every remaining callback goes unrun and the queue backs up silently. The recover must be inside a function that covers **one** callback — either a wrapper called per iteration, or the goroutine body when the goroutine handles a single dispatch. ## What the wrapper does not buy you - **It does not repair state.** If the callback took a mutex and panicked before releasing it (no `defer` on the unlock), that mutex stays locked forever and the next dispatch deadlocks. If it half-updated a shared structure, the wrapper cannot tell. - **It does not cover goroutines the callback starts.** If the subscriber's code writes its own `go`, a panic in that goroutine has its own stack, no recover on it, and it kills the process regardless of your wrapper. - **It does not make the failure benign.** A recovered panic is still a bug in someone's code, and if the dispatcher hides it at debug level, the subscriber never learns and the defect lives forever. So the contract for a recovered panic is: attribute it (which subscriber, which event), log it at error severity, count it as a metric, and treat a rising rate as an incident. That is what keeps containment honest rather than turning it into a silencer. ## Reporting the failure back Sending a `result` on a channel is the plain version. Two variants come up: - The dispatcher waits on a `sync.WaitGroup` and each wrapper writes its outcome into a preallocated slot; no channel capacity to size, and the recover path just assigns to the slot. - The dispatcher uses `golang.org/x/sync/errgroup` to collect the first error. Note that `errgroup.Group.Go` does **not** recover panics for you, and neither does `sync.WaitGroup.Go`; both run your function on a fresh goroutine, so a panic in it still kills the process. The deferred recover is still yours to write inside the function you hand them. Whichever you use, make sure the wrapper cannot leave the collector waiting: an unbuffered results channel with a collector that has already given up will block the recover path's send and leak the goroutine, so pair the send with buffering or with a select on the dispatcher's cancellation channel. ## What to verify in review For each `go` statement that runs foreign code: is the launched function a wrapper? Is `defer` + `recover` its first statement? Does every exit path — normal, panic, early return — send or record exactly one outcome? Is the recovered value logged with the subscriber's identity? If all four hold, one bad callback costs one delivery instead of the whole process.