What are the concrete performance and availability costs of using XA/2PC, and how do they show up in production?

## Why 2PC is inherently more expensive A local commit is one call: `connection.commit()`. A global 2PC commit is, per transaction: 1. **Prepare** call to each enlisted resource (network + each resource fsyncs its prepared state). 2. **Coordinator tx-log write** of the decision — a durable fsync, the point of no return. 3. **Commit** call to each resource (another network round-trip + fsync). So instead of one round-trip and one fsync you have roughly `2 × (number of resources)` round-trips plus multiple fsyncs. Even with 2 resources that's a large constant multiplier on commit cost. ## The three cost dimensions ### 1. Latency Extra round-trips and disk syncs raise per-transaction latency. Under high commit rates the coordinator's tx-log fsync becomes a serialization point / throughput ceiling. Optimizations reduce it in special cases — **1PC** when only one resource is enlisted (skips prepare and the log write), **read-only** vote for resources that only read, **presumed abort** to save log writes on rollback — but the general multi-resource write path pays full price. ### 2. Lock duration and the in-doubt window Resources acquire locks during the transaction and **hold them until phase-two commit**. Because 2PC inserts a whole prepare+decision phase before commit, locks live *longer* than in an equivalent local transaction, reducing concurrency and raising lock-wait/deadlock probability. Worse, after a resource votes commit it is **in-doubt**: it *must* keep locks until it hears the decision. If the **coordinator crashes between prepare and commit**, those locks stay held until the coordinator restarts and runs **XA recovery** (`XAResource.recover`) — which may be a scheduled scan running every N seconds. Other transactions touching those rows block or time out in the meantime. ### 3. Availability coupling 2PC can only commit if the coordinator **and every participant** are reachable. A single slow or down resource stalls the whole global transaction and, via held locks, can cascade to unrelated work. This makes availability the *product* of all participants' availability — the opposite of the independence you want in a scaled system. ## How it surfaces in production - **Throughput drop / higher p99 latency** as commit cost and the tx-log fsync bite under load. - **Lock-wait timeouts and deadlocks** from the longer lock-hold window. - **In-doubt transactions** after a coordinator or participant crash — rows/queue entries frozen until recovery. - **Heuristic exceptions** (`XAException.XA_HEUR*`) when an operator/timeout forces an in-doubt resource independently, producing a real inconsistency that needs manual cleanup. - **Recovery storms**: after restart the coordinator scans and resolves a backlog of prepared transactions, a burst of load. - **Deployment friction**: the durable tx log must survive restarts and each node needs a stable unique id (bad fit for ephemeral containers). ## Mitigations / when it's acceptable - Keep the number of XA participants **small** (ideally 2) and **co-located** to minimise round-trip and availability coupling. - Keep transactions **short** to shrink the lock window. - Ensure fast, durable storage for the tx log. - If costs are unacceptable, drop XA for a **transactional outbox** (single-resource atomicity) or a **Saga** (eventual consistency, no cross-resource locks). This performance/availability profile is the main reason distributed systems avoid XA.