In a choreographed checkout flow (order placed, then payment charged, then inventory reserved, then shipment created - each step reacting to the previous step's event), an order is stuck: payment shows charged but no shipment was ever created, and no service logged an error. Why is finding the root cause structurally harder here than in an orchestrator-driven version of the same flow, and what would you build to make it tractable?

## Nothing owns the process The root cause of this pain is structural: choreography deliberately has no component whose job is to track "the process." Each service only knows its own local slice — the payment service knows it charged the card and published `PaymentCharged`; the shipment service knows nothing at all, because it never received (or never successfully processed) that event. From any single service's point of view, everything looks fine: no errors were thrown, no retries exhausted, no alert fired. The "stuckness" only exists at the level of the whole business process, and no component's health check is scoped to the whole business process. ## The symptom is silence, not failure This is different in kind from a service being down. The classic symptom is silence, not failure: the message either - **was never published** (a bug in the payment service's event-publishing code, a transaction that committed the charge but rolled back before the outbox write), - **was published but never delivered** (a broker partition issue, a consumer group that fell behind or was misconfigured), - **or was delivered but silently dropped** (a deserialization error swallowed by a catch-all handler, or routed to a dead-letter queue nobody was monitoring). None of these produce an error in the sense of "a service logged a stack trace" — they produce an absence, and absences don't page anyone by default. ## The two things a cross-service timeline needs Tracking this down requires reconstructing a timeline that spans services, which requires two things that don't exist automatically. 1. **First, a shared correlation ID** (sometimes paired with a causation ID identifying which specific prior event triggered a given one) must be generated once, at the start of the process, and propagated through every subsequent event's headers, so that a query for that ID across every service's logs and message store returns the full, ordered story. If services weren't disciplined about propagating this ID from day one, retrofitting it means the correlation is simply impossible to reconstruct for historical incidents. 2. **Second, distributed tracing infrastructure** (commonly OpenTelemetry-based) needs trace context to flow through the message broker the same way it flows through HTTP headers in a synchronous call chain, which many teams only wire up after their first painful multi-service outage. ## Build the process tracker Even with both in place, choreography still lacks a natural place to ask "show me every order currently stuck for more than 10 minutes." The standard fix is to build a dedicated **read model** — sometimes called a process tracker or saga log — that subscribes to every event in the flow purely to project them into an explicit state per order ("placed → charged → [missing: reserved] → [missing: shipped]"), without that component ever issuing commands or making decisions (which would make it an orchestrator in disguise). This tracker becomes the dashboard on-call engineers query, and it's what turns "silent stall" into "alertable condition" by flagging orders that haven't advanced within an expected SLA. ## Why an orchestrator never has this gap An orchestrator sidesteps this whole class of problem by construction: because it must persist "which step this instance is on" in order to function at all (it needs that state to decide what to call next), the same state that drives the workflow is also, for free, the answer to "where is order 123 right now" — a queryable row rather than a reconstruction exercise. This is the core observability trade-off: choreography's diffuse execution avoids a central bottleneck but requires teams to deliberately build the correlation-ID discipline, tracing, and read-model tooling that orchestration gets as a side effect of its own control-flow bookkeeping. A concrete, widely-cited pattern here is pairing choreographed event flows with an explicit process-manager or saga-tracking read model precisely so teams get orchestration-like visibility while keeping choreography's decoupled execution — a hybrid that trades implementation effort for both benefits at once.