How would you design a consistent error-handling strategy across a multi-service system so failures are diagnosable, retryable where safe, and never silent?

## Why a *strategy* rather than local good manners In one class, clean error handling is a readability concern. Across a system it becomes an operational contract: whether a failure is retried, whether it wakes someone at 3am, whether the user sees something actionable, and whether an engineer can reconstruct what happened. Left to individual judgement, every service invents its own conventions and the composite behaviour becomes unpredictable — one service retries a business rejection forever, another swallows a timeout and returns a 200 with an empty body. ## 1. A shared taxonomy that drives behaviour The categories are only useful if each implies a different action: | Category | Cause | Retry? | Transport | Who is woken | |---|---|---|---|---| | Invalid input | Malformed/failing validation | No — deterministic | 400 / INVALID_ARGUMENT | Nobody | | Business rejection | Insufficient funds, out of stock, forbidden | No — needs a decision | 409/422/403 | Nobody | | Transient infrastructure | Timeout, connection reset, 503, deadlock | Yes, bounded, with backoff | 503 / UNAVAILABLE | On rate/SLO breach | | Programmer error / invariant broken | Null deref, impossible state | No — fix it | 500 / INTERNAL | Yes | | Dependency degraded | Downstream over budget | Maybe: degrade or shed | 503 + Retry-After | On SLO breach | The two attributes worth encoding explicitly on every error are **retryable** and **client-visible**. Everything else is diagnostics. ## 2. Stable error codes as part of the contract Clients must be able to branch programmatically. That means a stable, documented code (`PAYMENT_DECLINED`, `RATE_LIMITED`) — not an HTTP status alone (too coarse), not a message string (localised, reworded, breaks silently), not an internal exception class name (leaks structure and changes on refactor). Practical rules: codes are additive-only within a major version; a client seeing an unknown code must have defined default behaviour (usually: treat as terminal, surface generic message); document each code with cause, retryability and recommended client action. Reserve a machine-readable payload for details (which field, which limit, when to retry). ## 3. One translation point per boundary Each service should have exactly one place that turns internal failures into an outward representation: an exception-mapping handler / middleware / interceptor. It: - maps exception or Result-error type → status + code + safe message; - attaches the correlation id so the client can quote it in a support ticket; - emits **one** structured log entry at the appropriate level, with the full cause chain; - strips anything sensitive — stack traces, SQL, internal hostnames, PII, secrets. Verbose error output is a real reconnaissance vector, and stack traces to clients are a standard finding in security reviews. Symmetrically, at the *inbound* edge of each adapter, third-party and transport errors are wrapped into domain terms so the core never sees `SQLException` or an SDK type. ## 4. Retries only where they're safe "Retryable" is a claim about *side effects*, not about the error alone. A timeout is the hard case: the request may have succeeded and you lost the response. Retrying a non-idempotent write duplicates the charge. So the strategy must pair retryability with **idempotency keys** (client-supplied id the server deduplicates on) or naturally idempotent operations (`PUT`, conditional updates with expected-version). Then: - exponential backoff **with jitter**, or synchronised clients retry in lockstep and create a thundering herd; - a bounded attempt count and a total time budget derived from the caller's deadline (retrying past the caller's timeout is pure waste and amplifies load); - **retry at one layer only** — retries at client, gateway, and service multiply (3×3×3 = 27 downstream calls) and are a classic cause of retry-storm outages; - a **circuit breaker** so a sustained failure stops generating traffic and fails fast instead; - **bulkheads** (separate pools/queues per dependency) so one slow dependency can't consume all threads; - **load shedding** and deadline propagation so an overloaded service rejects early instead of timing everything out. ## 5. Asynchronous paths need their own model Queues and event consumers change the shape: failures are not returned to a caller. You need a redelivery policy, a maximum attempt count, a **dead-letter queue** with enough context to replay, poison-message handling, and — because at-least-once delivery is the norm — idempotent consumers. Silent dead-lettering is the async equivalent of an empty catch block, so DLQ depth must be alarmed and owned. ## 6. Diagnosability - **Correlation/trace id** generated at the edge, propagated on every hop and every log line, so one identifier reconstructs the whole request path. - **Log once.** Log-and-rethrow at each layer multiplies noise and hides the real handler. Wrap with context as it rises; log where it is finally handled. - **Structured logs** (fields, not sentences) so error rates can be aggregated by code and dependency. - **Errors as metrics**: rate by code, by dependency, by client — alerting on SLO burn rather than on individual exceptions. - **Preserve the cause chain** end to end; a code with no provenance is only half the story. ## 7. Failure of the *system*, not the request Decide deliberately, per dependency, between fail-fast and degrade: is a recommendation service optional (render the page without it) or essential (fail the request)? Write it down — the default of "whatever the code happens to do" produces user-visible outages from optional features. Graceful degradation, sensible defaults, cached fallbacks, and read-only modes are design decisions, not accidents. ## 8. Making silence impossible - Lint/static analysis rules that fail the build on empty catch blocks, swallowed errors, ignored Results, and bare broad catches. - Tests that exercise failure paths; fault injection or chaos experiments for timeouts, 500s, and slow dependencies. - Runbooks keyed by error code so an alert has an action attached. - Review checklist item: for every new failure mode, what is the code, is it retryable, who is paged, what does the user see? ## Trade-offs to state out loud A rigid global taxonomy can be over-engineering for a small system, and forcing every team through one shared error library creates a coupling point and a release bottleneck. The stable compromise: standardise the **contract** (codes, retryability, transport mapping, correlation id, log-once) and let each service implement it with its own idioms. Standardise semantics, not code.