Two of eight parallel worker agents fail — should the orchestrator synthesize, retry, or abort?

## Why this needs a policy, not a judgment call In a fan-out of eight, the probability that all eight succeed is the product of eight independent success rates. At 97% per worker that is about 78% of runs completing cleanly; at 90% it is under half. Partial failure is therefore the *normal* case at any real width, not an exception, and leaving the lead to improvise a response means the behaviour changes run to run. ## Classify the failure first Different failures deserve different handling: - **Transient** — timeout, rate limit, tool error, truncated response. Retry is likely to work; retry with backoff. - **Deterministic** — the source is missing, malformed, or access is denied. Retrying the same dispatch reproduces the failure exactly; either change the approach or record the gap. - **Budget exhaustion** — the worker hit its iteration or token cap mid-task. Often the best return is what it had at exhaustion, which is why workers should be instructed to report partial findings rather than fail silently at the cap. - **Silent failure** — the worst kind: the worker returned something well-formed and confidently empty, or answered a different question. This does not look like a failure at fan-in unless the lead validates returns against the contract. That last case is the argument for **validating returns**: check required fields, non-empty evidence, and that the return references the scope the worker was given. A worker that returns 200 fluent words about the wrong city is a failure the lead will otherwise synthesize straight into the answer. ## The policy ladder 1. **Bounded retry.** Once, maybe twice, with backoff and a cap. Uncapped retries turn a partial failure into a cost incident. Narrowing the scope or downgrading the tool path on the retry often converts a failure into a partial success. 2. **Degrade with a declared gap.** Synthesize from what returned and name what is missing, specifically: which slices, why, and what the reader should not conclude. Generic hedging ("some sources were unavailable") is nearly useless; a named gap lets a human decide whether to care. 3. **Escalate.** Hand the failed slice to a human, to a different tool path, or to a more capable model rather than dropping it. 4. **Abort.** Reserve for tasks where partial coverage is not merely weaker but wrong — anything that answers "is there any instance of X", any compliance or safety sweep, any decision gate. A negative result from an incomplete sweep is a false negative dressed as an answer. ## Timeouts and stragglers A fan-out's latency is the slowest worker's, so the policy also needs a **deadline**: at time T the lead proceeds with what it has and marks the stragglers as gaps. Without one, a single stuck worker can hold an otherwise complete run indefinitely. Pairing a deadline with an instruction to workers to return partial findings on approaching their budget converts most stragglers into usable, if thinner, returns. ## Idempotence and side effects Retries are only safe if the worker's action is repeatable. This is one of the practical reasons the surviving version of this pattern keeps **writes and side effects single-threaded** with the lead or one designated writer: read-only workers can be retried freely, whereas a worker that half-applied a change and then failed leaves state you now have to reason about before retrying. If workers must act, they need idempotency keys or a compensating path. ## What to surface to the user Whatever the policy, the final answer should carry the coverage story: eight slices attempted, six returned, these two missing for this reason. That single line converts an unquantified answer into one a reader can calibrate, and it is the difference between a degraded system and an unreliable one.