In a shared model gateway, what is a fallback chain, and why is it set per route rather than once for the whole company?
answer
- an ordered list of targets
- which failures move the call on
- least disruptive substitute first
- features tolerate different substitutes
basics
~20 sA fallback chain is the ordered list of model targets a gateway tries when the current one errors, stalls or throttles. It is set per route because each feature tolerates different substitutes, latency and cost.
solid answer
~40 sA fallback chain is an ordered list of **targets**, each one model at a specific provider, region or account, that the gateway walks when a call to the current target fails with a server error, stalls or is throttled. A common order runs from least to most disruptive: the same model in another region or account, the same model on another host, then a different model, then a clear error the application handles. It is set per route because features differ: a ticket summary can accept a different model, while a contract extractor whose prompt and parser were tuned on one model may accept only that model elsewhere. Errors the request itself causes, such as a malformed body, should not walk the chain, because another copy of the same model would reject them again.
code
yaml · 18 lines# illustrative gateway route config; every field name is invented
routes:
contract-extraction:
fallback_chain:
- extractor-model @ vendor-a / region-east / account-1
- extractor-model @ vendor-a / region-west / account-2
- extractor-model @ host-b / region-east
fall_back_on: [server_error, overloaded, rate_limited, no_first_byte]
never_fall_back_on: [invalid_request, content_refused]
when_exhausted: return_error
ticket-summary:
fallback_chain:
- summary-model @ vendor-a / region-east / account-1
- summary-model @ host-b / region-east
- other-summary-model @ vendor-c / region-east
fall_back_on: [server_error, overloaded, rate_limited, no_first_byte]
never_fall_back_on: [invalid_request, content_refused]
when_exhausted: return_errorgo deeper
Define a fallback chain as an ordered list of model targets, name the failures that move a call on, and say why a malformed request does not.
Explain the usual order, same model elsewhere before a different model, and why each feature's output contract and prompt tuning make the chain a per-route choice.
Show that you would make chain exhaustion a clear error, keep refusals and caller errors out of failover, and split ownership between platform and feature teams.
Frame the order as a trade between supplier independence and output fidelity, and say which default the platform sets and which choices it leaves to route owners.
## What a fallback chain is A **shared model gateway** is the one internal endpoint every team calls instead of calling model providers directly. A **route** is a named entry on that gateway that one feature calls, such as `ticket-summary` or `contract-extraction`. A **target** is one concrete place a call can go: a particular model, served by a particular provider or host, in a particular region, under a particular account. A **fallback chain** is the ordered list of targets a route may use. The gateway sends the call to the first healthy target; if that call fails in a way another target could fix, it moves to the next entry, and so on until one answers or the chain runs out. ## What moves a call down the chain Not every failure should move a call on. The question to ask is whether the next target could plausibly succeed where this one failed. | Upstream outcome | Walk the chain? | Why | |---|---|---| | Server error or "overloaded" response | Yes | The target is struggling; another target may not be | | HTTP 429 on this API key or account | Yes, to a target with remaining quota | The limit is local to this target | | Connection failure or no first byte in time | Yes | The target is unreachable or stalled | | HTTP 400 for a malformed request | No | A copy of the same model rejects it again | | A content-policy refusal | No | Shopping for a more lenient target defeats the policy | Treating the last two as failover triggers wastes calls, adds latency and, for refusals, quietly turns the chain into a way around a safety decision. ## The usual order: least disruptive first Most teams order a chain by how much the answer could change if that entry replies: 1. **The same model in another region or under another account** of the same provider: the same model and the same request, with separate capacity and failure domain. 2. **The same model on another host**, for example an open-weight checkpoint that several hosts serve, or a closed model resold through a second cloud: nearly the same behaviour, a more independent failure domain. 3. **A different model**: the most independent supplier, and the biggest change in output. 4. **A clear "chain exhausted" error** naming the targets tried, so the application can degrade in its own way. Chains are usually short, two to four entries, because every failed attempt adds latency before the caller hears anything, and every extra entry is one more target that must be kept tested. The order is a trade between independence and fidelity. A different vendor fails independently of the primary, but it also answers differently, which is why it usually sits last rather than first. ## Why per route, not one company-wide list A single company-wide list assumes every feature tolerates the same substitutes. They do not: - **Output contracts differ.** A route that parses structured output or relies on tool calls breaks when a model with different habits answers; a free-text summary rarely does. - **Prompts are tuned per model.** A prompt refined against one model can lose quality on another, so a feature may only accept entries it has evaluated. - **Latency budgets differ.** An autocomplete route may prefer an error over a slow fallback; an overnight report does not care. - **Cost differs.** A fallback to a larger model can cost several times more per call, which a high-volume route may not be allowed to absorb. - **Who knows the consequences differs.** Only the feature team knows what a substitute breaks, so they choose the entries; the platform team owns the mechanism and the list of approved targets. ## What the chain does not cover A fallback chain says *where the call goes next*. How many times to retry the same target first, how long to wait before giving up, and breaker state machines are general resilience patterns covered with retries and circuit breakers. What a feature shows when the whole chain fails is graceful degradation in the application. Making one request shape fit each provider in the chain belongs to the gateway's request translation. The chain itself is the gateway's routing decision: which targets, in what order, for which failures.
- Why should an HTTP 400 for a malformed request not move a call down a fallback chain?A 400 says the request itself is wrong, so another copy of the same model rejects it again, and a different model may accept it only by ignoring the defect. Walking the chain spends calls and latency and hides a caller bug. The gateway should return the error to the caller at once and count it against the caller, not the target.
- What should the gateway do when every entry in a route's chain has failed?Return one clear error that says the chain is exhausted and lists the targets tried and why each failed, rather than looping back to the first entry. The application then applies its own degraded behaviour. Looping turns an outage into a retry storm across every provider at once.
- Who should own a route's fallback chain: the platform team or the feature team?Both, at different layers. The platform team owns the mechanism, the list of approved targets and a sensible default chain. The feature team chooses which approved entries its route accepts, because only it knows whether a different model breaks its parser, its tone or its latency budget.
Rebooking a cancelled flight: the agent first tries the same airline's next departure, then a partner on the same route, and only then a train. A traveller with a tight onward connection may refuse the train, so each traveller's acceptable list differs.
- One company-wide fallback list is good enough for every feature.
- Any error, including a malformed request, should try the next provider.
- The first fallback should always be a different vendor's model.
- When the chain runs out, start again from the top until something answers.
- Fallback targets need no testing until a real outage uses them.