Your teams keep losing asynchronous failures in production streams — what visibility standard would you set, and how would you make it stick?

## Standardise the outcome, not the library The failure mode is organisational, not technical: every team knows how to log an error, and failures still vanish. They vanish because nothing in the codebase makes losing one *harder* than losing one. So the standard should describe **what must be observable about every asynchronous run**, and say nothing about which reactive library a team picked — otherwise it stops applying the moment a team makes a different choice, and it reads as a tooling mandate rather than a contract. ## The contract worth mandating Four requirements, deliberately few enough to remember: 1. **Every terminal subscription registers a failure handler.** A subscription with only a value handler is the single largest source of lost failures, and it is mechanically detectable. 2. **The process-wide sink for undeliverable signals is configured at startup.** It counts, logs with whatever stack exists, and tags the event as its own defect class — never a generic error. 3. **Every run records exactly one terminal outcome** — completed, failed or cancelled, with a reason on the last two — so an ending is a positive fact rather than an absence somebody must notice. 4. **Every failure carries identity.** The work item and a correlation identifier go into the failure when it is created, because nothing derived from a thread survives a hop. Two more are worth *recommending* rather than mandating, because their cost is real: named markers at chain boundaries, and the assembly-time diagnostic mode under sampling. ## Enforcement, in order of how well it works | Mechanism | Strength | Cost | |---|---|---| | One sanctioned subscribe helper that supplies the defaults | Strongest — compliance is the default | Building and owning a shared component | | Startup assertion that the sink is configured | Strong for one requirement, fails loudly | Trivial | | Automated check that flags handler-less subscriptions | Good, catches the main case pre-merge | Rule authoring and false positives | | Review checklist | Weak — decays within a quarter | Reviewer time on every change | | A written policy alone | Effectively none | Nothing, which is the problem | The ordering is the point of the answer. A standard with no enforcement path is a preference, and it will be relitigated by every team that finds it inconvenient. ## The trade-offs a lead actually owns - **Should an undeliverable-signal event fail the process?** In test, yes — failing fast is how a defect becomes visible before it ships. In production, no: those workers are shared with unrelated work, and killing the process converts one team's bug into everybody's outage. Count it, page on a threshold, and keep the loud behaviour where it is safe. - **Page or merely count?** An undeliverable failure is a defect, so it should page while the count is low — which requires the count to *be* low first. Rolling it out into a codebase with hundreds of pre-existing events means counting and burning them down before the page is switched on, or nobody will ever trust the alert. - **How much tracing does the platform buy by default?** Always-on assembly-time tracing captures a stack per stage per subscription and is not affordable on a hot path. Buy markers and carried identity as the default; buy deep tracing as a targeted, time-boxed capability for incidents. - **Central helper or team autonomy?** A shared helper is the strongest enforcement available and the most contentious thing on the list. Keep its surface tiny and its escape hatch explicit, or teams will fork it and you will lose the enforcement you were buying. ## How to roll it out without a mutiny 1. Instrument first and mandate second: turn on the sink's counter everywhere, and publish the number of events per team. The argument usually ends there. 2. Ship the helper before the rule, so the rule is satisfiable the day it lands. 3. Fail the build in test environments only, for one quarter, then tighten. 4. Review the alert's precision after a month. An alert that pages on noise is worse than no standard at all, because it teaches people to ignore the class of defect you were trying to surface. ## What separates this from the senior version A senior answer diagnoses one lost failure and fixes it. This answer is about a contract other teams must live with: it accepts that the mandate's *enforcement mechanism* is the design, not the list of requirements, and it prices the two things that are genuinely expensive — always-on tracing and alert noise — instead of asking for everything and getting nothing.