A Go queue worker retries every non-nil error and is overloading a failing downstream. How do you fix its classification?

## How the storm forms The worker's handler returns an error and the loop does the obvious thing: ```go if err := handle(ctx, msg); err != nil { return msg.Nack() // try again later } ``` This is correct for exactly one class of failure and wrong for every other. When a downstream starts failing, each message now produces N attempts instead of one, so the offered load on the sick dependency rises by the retry factor at precisely the moment it needs less. Messages that can never succeed — a payload that does not unmarshal, a record that was deleted — are re-attempted forever, occupying worker slots that healthy messages need. And if the failure was cancellation from a rolling restart, the worker re-attempts work nobody wants. All three effects compound, which is why a retry-everything worker turns a downstream blip into a self-sustaining outage that survives the original cause. ## The fix: one classifier, three dispositions The first move is to stop making the decision at call sites. Scattered `if strings.Contains(...)` and ad-hoc `errors.Is` checks drift apart; the retry policy should be readable in one function. ```go type disposition int const ( terminal disposition = iota retryable abandoned ) func classify(err error) disposition { var r interface{ Retryable() bool } switch { case errors.Is(err, context.Canceled): return abandoned case errors.Is(err, ErrBadMessage): return terminal case errors.As(err, &r) && r.Retryable(): return retryable default: return terminal } } ``` Three things about this shape matter more than the specific cases. **It reads the unwrapped chain.** `errors.Is` and `errors.As` both walk `Unwrap`, so the classifier keeps working when a handler adds context with `%w`. Any check that inspects only the top-level value breaks the first time someone improves an error message. **The default is terminal, not retryable.** This is the deliberate inversion of the buggy worker. An unrecognised error is one you have not reasoned about; sending it to the dead-letter queue costs you one message and a page, while retrying it costs you an amplification factor against a dependency you do not understand. Dead-lettered messages are recoverable — you can replay them once you have classified the new failure mode. A storm is not recoverable in the same sense. **Abandoned is its own outcome.** Cancellation means the attempt never reached a verdict, so it should neither consume the retry budget nor mark the message bad. Leave it unacknowledged for redelivery. ## The budget you keep even when the classifier is wrong Classification will be wrong sometimes — a new dependency version returns a failure shape you have never seen. So the envelope's delivery count is the backstop: after N deliveries the message is dead-lettered regardless of how it was classified. That converts "retried forever" into "retried N times", which is a bounded, survivable mistake. Keep the cap in the worker, not in each handler, and record the attempt number on the dead-lettered message so the replay has context. The cap also protects against the subtler version of the bug: an error that is genuinely transient in principle but not in practice, such as a downstream that will be down for six hours. Classification says retryable and it is not lying; the budget is what stops you from proving it. ## Making it visible from the 3am chair A single `errors_total` counter cannot answer the question you actually have at 3am, which is *what kind* of failure and *which attempt*. Break the counter out by two labels: the class the classifier assigned (or a short symbol for the matched sentinel), and the attempt number from the envelope. That gives you three readings instantly: - Volume concentrated at attempt 1 with low classes of retryable: a genuine downstream incident, the classifier is behaving. - Volume concentrated at attempts 5-10 with the same class: a storm — the same small set of messages is cycling, and you should stop retrying that class. - A rising count in the terminal bucket after a deploy: a new failure shape hitting the default branch, which is your signal to classify it properly rather than to widen the retryable set. Without the attempt-number dimension, a storm and real traffic look identical on the dashboard, which is exactly why these incidents run long. ## The immediate mitigation Before the code change ships, the lever is the classifier's inputs: drop the retryable set to only the classes you are sure about, or set the attempt cap to 1, so the worker drains rather than amplifies. Then classify the new failure mode, replay the dead-letter queue, and add the class to the retryable set deliberately. ## What an interviewer is listening for That you can name the amplification mechanism, that you centralise the decision and read the unwrapped chain rather than error text, that your unknown-error default is the safe one, and that you know what you would have to see on a dashboard to distinguish a storm from an incident.