skip to content

A component keeps producing deadlocks because its operations acquire several locks. What redesigns remove the possibility entirely rather than managing it, and what does each one cost?

level: principalimportance: should knowfreq 36%

answer

  1. one coarse lock first, measure second
  2. shard by key so one op = one lock
  3. immutable / copy-on-write kills exclusion for readers
  4. single owner + messages: still deadlocks if handlers block
  5. CAS removes deadlock, adds livelock

basics

~20 s

Collapse to one coarse lock; partition state so each operation touches one shard; make data immutable or copy-on-write; give state a single owner and use messages; or use lock-free atomic updates. Each trades throughput, memory, latency or complexity for the guarantee.

solid answer

~60 s

Ranked roughly by how much they buy and cost: - **One coarse lock.** Removes circular wait inside the component outright. Costs throughput under contention and still deadlocks if it can be held across a call into another locked subsystem. - **Partition by key.** Each operation touches exactly one shard, so no two locks are co-held. Costs a design answer for the rare cross-shard operation, which needs an ordering or a single serializer. - **Immutability / copy-on-write.** Readers need no exclusion at all, so mutual exclusion disappears for the read path. Costs allocation and makes writes coarse. - **Single owner plus messages (actor or channel).** State is mutated by one participant; others send requests. Costs latency and requires that handlers never block awaiting a reply, or two actors deadlock exactly like two mutexes. - **Lock-free updates via compare-and-swap, or optimistic versioning with retry.** No mutual exclusion, so no deadlock; livelock becomes the hazard and correctness is hard. The rule of thumb: prefer designs where the invariant cannot be violated by a stranger's code path, not designs that require everyone to remember a rule.

code

text · 8 lines
text
before:  op(x, y) -> lock(x); lock(y)          # order matters, cycle possible

after:   shard = shardOf(key)
         lock(shard); apply(op); release(shard)

cross-shard op (rare):
         acquire shards in ascending shard index, or
         hand the whole operation to one serializer

go deeper

for a junior

Know that using one lock instead of several removes the multi-lock deadlock, and that immutable data needs no locking at all.

for a middle

Compare coarse locking, sharding and immutability, and state the throughput cost of each.

for a senior

Add message passing and optimistic or lock-free updates, and be explicit that message passing still deadlocks when handlers block on replies.

for a principal

Sequence the decision - simplest correct design first, measure contention, then pay for partitioning or ownership changes - and make the invariant structural and local so no future caller can break it.

## Reframing the problem Recurring deadlocks in one component are a design signal, not a series of bugs. Every mitigation discussed elsewhere - ordering, bounded attempts, shorter sections - depends on humans continuing to obey a rule as the code grows. A redesign that removes the multi-lock requirement removes the class of defect and stops consuming review attention forever. The engineering question is which guarantee you want and what you are willing to pay. ## Option 1: collapse to a single lock Guard the whole component with one lock. Circular wait becomes impossible internally, because a thread never holds two of your locks at once. This is the cheapest correct answer, and it is dramatically undervalued: a component that is not on the hot path rarely notices the serialization, and the simplification often makes other bugs disappear too. Cost: all operations serialize, so throughput is capped by the critical section length times the arrival rate. Measure before assuming that matters. Two residual risks remain - if the single lock can be held while calling into another locked subsystem, cycles across components are still possible, and one coarse lock makes a slow operation block fast ones, turning a throughput issue into a latency issue. ## Option 2: partition the state Shard state by key so any single operation touches exactly one shard and needs exactly one lock. Concurrency scales with shard count, and no operation co-holds locks, so no cycle can form. Cost: cross-shard operations. If some operation legitimately spans two shards - a transfer between accounts in different shards - you are back to two locks and must apply an ordering there, or route all cross-shard work through a single serializing path. Choosing a shard key that makes cross-shard operations rare is the real design work, and skew can leave one hot shard doing most of the traffic. ## Option 3: immutability and copy-on-write If a structure is never mutated after publication, readers need no exclusion at all - mutual exclusion, the first Coffman condition, simply does not apply to them. Updates create a new version and swap a single reference atomically, so only writers coordinate, often with just one atomic operation. Cost: allocation and copying, which is fine for read-mostly configuration or routing tables and poor for large, write-heavy state. Writes become coarse-grained (whole-structure swaps), and readers may observe a slightly stale version, which must be acceptable semantically. Persistent (structurally shared) data structures reduce the copying cost substantially. ## Option 4: single ownership and message passing Give each piece of state exactly one owner - an actor, a task, a single-threaded event loop - and let everyone else send messages. No shared mutable state means no locks, so lock-based deadlock is gone, and the design also sidesteps the publication hazards that shared-memory models have, because state never crosses a boundary while mutable. Cost, and this is the part candidates miss: **circular waits survive**. If actor A sends a request to B and blocks awaiting the reply while B does the same toward A, they are deadlocked exactly as two mutexes are. Message-passing systems stay safe by making handlers non-blocking (send and continue, handle the reply as another message), by giving the call graph a direction so requests only flow one way, and by putting timeouts on every await. Additional costs are latency per hop, mailbox growth under overload, and the fact that a single owner is a throughput ceiling for its state. ## Option 5: lock-free and optimistic techniques Update shared state with atomic compare-and-swap: read the current value, compute the new one, swap only if it has not changed, retry if it has. With no mutual exclusion there is no deadlock, and a lock-free algorithm guarantees that *some* participant always makes progress. Cost: individual threads can retry indefinitely under heavy contention (livelock or starvation for the unlucky), correctness is genuinely difficult for anything beyond a single word, and the same idea at coarse grain - optimistic versioning with a version check and a retry, as databases do - trades wasted work under contention for freedom from blocking. Use published, tested primitives rather than hand-rolling structures. ## Choosing Ask three questions. **Where is the contention actually?** Measure before paying for anything more complex than a single lock. **What does the operation genuinely need to be atomic over?** Multi-lock designs usually come from an atomicity requirement that was never stated; write it down and often it shrinks. **Who else can violate the invariant?** A rule that depends on every future caller remembering the acquisition order is weaker than a structure where the second lock does not exist. The strongest answer is usually staged: collapse to one lock first to make it correct and simple, measure, then partition or move to single-ownership only where the numbers justify it - and keep the component's locks entirely private, never held across a call to another subsystem, so the guarantee is local and cannot be broken from outside.

  • Your component now uses a single lock, yet it still deadlocks against another subsystem. What went wrong?
    The single lock removed cycles inside the component but not across components: some path holds your lock while calling into the other subsystem, which takes its own lock, and a reverse path exists elsewhere. The guarantee must be made local - your lock is never held across a call that leaves the component, and callers never invoke you while holding a lock you might need to call back into. If cross-component atomicity is genuinely required, define one ordering across the two subsystems and enforce it with lock levels.
  • Does moving to an actor or channel-based design make deadlock impossible?
    No. It removes data races and lock-based deadlock, since state is owned by a single participant and never shared mutably, but a blocking wait for a reply is still hold-and-wait on a non-preemptible resource. Two actors that each await the other's response are deadlocked in exactly the same shape as two mutexes taken in opposite orders. Safety comes from non-blocking handlers, a directed request graph, and timeouts on every await.
  • How do you decide between a single coarse lock and a partitioned design?
    By measurement, not by intuition. Look at lock wait time, hold time and arrival rate: if the coarse lock's queueing delay is invisible against end-to-end latency, keep it, because simplicity has real ongoing value. Partition when contention is demonstrably the bottleneck and the workload has a key that spreads evenly and keeps cross-shard operations rare; otherwise sharding buys you skew, cross-shard complexity, and the multi-lock problem back again.

saying these in an interview costs you the question

  • Reaching for lock-free structures first, when a single coarse lock would be correct, simple and fast enough.
  • Assuming actors or channels make deadlock impossible, ignoring blocking request-response cycles between them.
  • Partitioning without a plan for cross-shard operations, which reintroduces two co-held locks.
  • Treating fine-grained locking as automatically faster, without measuring contention or accounting for the added acquisition overhead.
  • Believing a single component-wide lock guarantees safety even when it is held across calls into other locked subsystems.

context