How would you decide whether a sharded workload should use cross-shard MongoDB transactions or be remodelled to avoid them?

## What changes when a transaction crosses shards On a replica set, a transaction is local: one primary stages the writes, one commit makes them visible. In a sharded cluster, `mongos` routes each operation to the shards that own the documents. If every operation in the transaction lands on the same shard, the transaction is effectively a single-shard transaction and behaves like the replica-set case. When the transaction writes to more than one shard, it becomes a **distributed transaction**. MongoDB elects one of the participating shards as the transaction coordinator, and the commit runs as a two-phase commit: the coordinator asks each participant to prepare, each participant durably records that it is prepared and holds its locks, and only when all have voted does the coordinator record the decision and tell everyone to commit. The coordinator's state is persisted so the decision survives a failure — a prepared transaction is not something a crash can quietly forget. ## Where the cost actually lands - **Latency.** The commit is no longer one acknowledgement; it is a prepare round plus a decision round across every participant, each majority-acknowledged. Commit latency is set by the slowest participant, so p99 gets worse as the number of participants grows. - **Blocking on prepared documents.** Between prepare and decision, the documents a participant has prepared are in limbo. Other operations that need to see them may have to wait for the decision. A brief coordinator hiccup therefore radiates into unrelated traffic touching those documents. - **Failure surface.** More participants means more ways to hit a transient error and re-run the whole transaction, and more chance of hitting the transaction lifetime limit. Cluster activity such as a chunk migration involving the data can also abort the transaction. - **Operational opacity.** Diagnosing a slow distributed commit means correlating across several shards plus the coordinator, which is a materially harder investigation than one slow primary. None of this makes cross-shard transactions unusable. They are a supported, correct feature, and for a low-rate, high-value operation — settling a payment across two accounts on different shards — paying that cost is the right call. The judgment question is whether your *hot path* is paying it thousands of times a second. ## The decision framework **1. Measure the split.** How many transactions per second are single-shard versus multi-shard, and how many shards do the multi-shard ones touch? A workload where 2% of transactions are cross-shard has a very different answer from one where 60% are. Compare p99 commit latency for each class, and track the abort/retry rate. **2. Fix the shard key first.** This is the highest-leverage change and the hardest to reverse. If the transactional unit is "everything belonging to one tenant" or "one account and its ledger entries", a shard key prefixed by that identifier colocates the whole unit on one shard, and the common transaction becomes single-shard by construction. The classic failure is a shard key chosen purely for write distribution that scatters co-changing documents across every shard, turning every business operation into a distributed commit. **3. Then look at the model.** Many invariants that appear to need a transaction fit in one document — order plus line items, entity plus its counters — where atomicity is free and shard placement is irrelevant. Others can be restructured so that the cross-boundary part is a derived value that tolerates a short lag rather than a hard invariant. **4. Then consider a workflow instead of a transaction.** Where the boundary is genuinely a service or domain boundary, a transaction is often the wrong shape: write the local change and an outbox record in one atomic transaction on one shard, and drive the remote effect from the outbox with compensation on failure. You trade linearizable simplicity for availability and independence — a trade worth making when the two halves have different owners or availability requirements. **5. Keep the residual transactions small and boring.** Whatever cross-shard transactions survive should touch few documents, hold no external calls, run well under the lifetime limit, commit with an explicit write concern, and be instrumented separately so their cost stays visible. ## What would change my mind If the cross-shard rate is low and stable and the p99 is within budget, remodelling is not worth the migration risk — a resharding effort is expensive and disruptive, and correctness you already have is worth a lot. If the rate is growing with traffic, or commit latency dominates the request budget, or the abort rate is climbing under contention, then the shard key is the problem and no amount of tuning fixes it. ## The failure mode to name explicitly The worst version is a team that reaches for transactions to paper over a model that scattered co-changing data, then raises the transaction lifetime limit when the transactions start timing out. That converts a design problem into an operational one: longer-held prepared state, more blocking, more lag — and a much harder diagnosis when it finally breaks.