How would you set and enforce a default error-fallback policy across many services so failures stay debuggable without leaking internals?

At one service, the fallback's detail setting is a checklist item. Across dozens, it is a policy problem: the failure mode is not a wrong decision but an undecided one, replicated by every new service from whatever template it copied. ## What the policy actually has to cover - **The default detail level** in each environment, and how that default is arrived at. - **Who may enable the diagnostic rendering**, where, and for how long. - **What the terse response contains** at minimum, so a caller can report a failure usefully. - **What the layers in front do** when they answer instead of the application. - **How it is verified**, and how often. - **What happens on a new service** - whether the safe setting is inherited or must be remembered. The last item decides whether the rest holds. A policy that depends on every team remembering it is a policy that describes your best services and not your worst. ## Defaults must fail safe 1. Choose terse as the *unset* behaviour, so a service that configures nothing is already safe. 2. Never key the choice on the absence of a marker - "verbose unless production is declared" makes every misconfigured, forgotten or newly bootstrapped environment verbose. 3. Make the diagnostic rendering an explicit, visible opt-in, so enabling it is a decision someone made rather than a state something fell into. 4. Treat internal-only deployments the same way. "Behind the boundary" is a claim about today's network, and internal traffic is exactly what an attacker who is already inside can read. ## Where to enforce it | Enforcement point | Strength | Cost | |---|---|---| | Inherited baseline the service starts from | Correct by default, no per-team memory needed | Needs an owner, and drifts as services fork it | | Normalisation at the edge | Covers services you do not own, including ones you forgot | Can mask which service failed; cannot see internal traffic | | Per-service configuration only | Maximum local freedom | Guarantees divergence; audit cost grows with the fleet | | Pipeline check on deploy | Catches regressions at the moment they ship | Only as good as the probe, and needs maintenance | These are not alternatives so much as layers. A realistic arrangement is an inherited default plus a deploy-time probe, with edge normalisation as a backstop for anything outside that lineage. ## Verification, because the fallback is never exercised by success A fallback setting is invisible until something fails, which means review alone cannot confirm it. Make it observable: - provoke the fallback deliberately in every deployed environment - a request guaranteed to take that branch is the cheapest probe there is; - assert on what comes back: no frames, no file paths, the expected content type, a body under a sane size; - run the probe as part of deployment, not as a quarterly audit, so a regression is attributed to the change that caused it; - probe the layers in front too, since their defaults are set by different people on a different cadence. ## The tradeoffs you are actually trading - **Terse versus debuggable.** Every byte removed from the response is a byte someone has to find in a log. Pair the terse body with something the caller can quote, or you trade a disclosure risk for a support cost. - **Uniformity versus autonomy.** A single shape across services is worth real money to clients and to anyone debugging across boundaries; it also means a central decision imposed on teams with different callers. Uniform *floor*, local freedom above it, is the usual settlement. - **Edge normalisation versus attribution.** Rewriting error responses at the edge guarantees the shape and can erase the signal that says which service failed. If you normalise, preserve attribution somewhere operators can still read. - **Policy versus mechanism.** A written rule is cheap and decays; an inherited default and a deploy probe cost real work and hold. Prefer spending the work on the mechanism. ## Migration, when services already differ Inventory what each service actually returns by probing rather than by reading configuration, since the two disagree more often than teams expect. Then move the majority to the inherited default, leave the exceptions documented with their reason, and put the probe in the pipeline first - so the fleet stops getting worse while you are still making it better. ## How to talk about it The strong answer names a default direction and defends it, picks an enforcement point and says what it does not cover, and insists on external verification because nothing else can see the fallback. The weak answer is a policy statement with no mechanism behind it.