A service intermittently fails with the database's deadlock error, which rolls the transaction back entirely. Design the application-side handling for that error: where the retry lives, how many attempts, what must not be inside the retried block, and how you distinguish it from errors that must not be retried.

## Why retry is the correct response — and only half the answer A deadlock error carries a specific contract: your transaction was chosen to break a wait cycle, it was rolled back **completely**, and the conditions that caused it were timing-dependent, so running again is likely to succeed. That makes retry correct. But retry is a *tolerance* mechanism, not a fix; a service whose deadlock rate is climbing needs its access ordering or transaction scope repaired, and the retry layer exists to keep the occasional collision from reaching a user. ## Where the retry lives At the **transaction boundary**, wrapping the whole unit of work: begin, read, decide, write, commit. This is non-negotiable, and it is where most implementations go wrong. Retrying an individual statement inside a transaction that no longer exists is meaningless. Retrying at the boundary but reusing values loaded before the failure is worse — the surviving transaction has since changed exactly the rows you were fighting over, so replaying a decision computed from stale reads can produce a wrong result rather than an error. Each attempt must re-read. Structurally, this argues for one shared helper — a `withTransactionRetry { … }` wrapper, an interceptor around service methods, or the framework's declarative equivalent — rather than per-call-site try/catch. A single implementation means the classification, the bounds, the backoff, and the metrics are consistent, and new code inherits them. ## Classifying the error Retry must be driven by the engine's **specific error class**, not by string matching on a message and not by catching a broad exception type. Retryable: deadlock detected, serialization failure, and — with more care — lock-wait timeout, which is retryable but often signals a slow holder rather than a cycle. Not retryable: unique or foreign-key constraint violations, check-constraint failures, syntax or type errors, permission denials, and any domain-level rejection. These are deterministic; a second attempt produces the same failure while doubling the load and burying the real bug. A subtlety worth raising: some frameworks wrap driver exceptions, and a careless mapping can collapse a deadlock and a constraint violation into the same generic type. Verify what your data-access layer actually throws, and assert it in a test. ## Bounding and spacing the attempts **Bound the attempts** — three to five is typical. Beyond that, either the contention is structural or the workload is saturated, and continuing to retry converts a localised problem into a system-wide one: connections stay checked out, thread pools fill, upstream timeouts fire, and load amplification pushes the deadlock rate higher still. When the cap is reached, fail the request with a clear error the caller can act on. **Back off exponentially with jitter.** Two transactions that deadlocked will, if retried immediately and in lockstep, collide again in exactly the same way — the classic path from deadlock to livelock, where work is repeatedly done and thrown away and nothing progresses. Randomised delay decorrelates the retries so one attempt wins. ## What must not be inside the retried block Anything the database rollback cannot undo: - **External calls** — payment authorisations, emails, SMS, webhooks, third-party APIs. If the transaction is retried, these run again; a duplicate charge is far worse than a failed request. - **Message publishes** — use the transactional outbox pattern: write the intent to a table inside the transaction, and let a separate relay publish it after commit. - **In-memory or cache mutations** — updating a shared cache, a counter, or an object's state inside the block leaves stale mutations behind when the attempt is rolled back. - **Non-idempotent business logic** — anything that consumes a nonce, generates a sequence value with a visible side effect, or writes files. The general rule: the block must be safe to execute *n* times with only the last one committing. If some step cannot be made re-runnable, move it after commit, or make it idempotent with a natural key or an idempotency token. ## Interaction with the rest of the stack Retries consume the request's latency budget, so the total worst case — attempts times per-attempt latency plus backoff — must fit inside the caller's timeout, or the caller gives up while the database keeps working. Retries also hold a connection for the whole sequence, so a burst of contention can exhaust the pool; some designs return the connection between attempts precisely to avoid that. And retries interact with upstream retries: a client that retries on a 500 while the service retries internally multiplies attempts, so the layers must be coordinated rather than each defending itself. ## Observability Emit a counter per operation for deadlock errors and a histogram for attempts-until-success, and alert on rate rather than on individual events. Log the engine's deadlock report or its identifiers, because that is what names the conflicting statements and their access order. Two patterns should trigger investigation rather than tuning: a specific pair of operations that always appear together in reports (an ordering bug), and one operation that is always the victim (a starvation pattern, usually a small transaction colliding with a large batch). ## Then fix the cause With retries in place, use the reports to remove the deadlock: order the access to shared rows consistently across all code paths, sort keys before batch updates, shorten transactions so locks are held briefly, avoid user think-time inside a transaction, and take the strongest lock you will need up front instead of upgrading a shared lock to exclusive mid-transaction.