skip to content

Model Gateways

One internal endpoint every team calls instead of each provider, holding keys, budgets, failover and logs in one place. Interviewers ask what it buys and what the extra hop and outage risk cost.

on this pageshow

explore

questions

27

In a shared model gateway, what is a fallback chain, and why is it set per route rather than once for the whole company?

level: juniorimportance: must knowfreq 56%

answer

  1. an ordered list of targets
  2. which failures move the call on
  3. least disruptive substitute first
  4. features tolerate different substitutes

basics

~20 s

A fallback chain is the ordered list of model targets a gateway tries when the current one errors, stalls or throttles. It is set per route because each feature tolerates different substitutes, latency and cost.

solid answer

~40 s

A fallback chain is an ordered list of **targets**, each one model at a specific provider, region or account, that the gateway walks when a call to the current target fails with a server error, stalls or is throttled. A common order runs from least to most disruptive: the same model in another region or account, the same model on another host, then a different model, then a clear error the application handles. It is set per route because features differ: a ticket summary can accept a different model, while a contract extractor whose prompt and parser were tuned on one model may accept only that model elsewhere. Errors the request itself causes, such as a malformed body, should not walk the chain, because another copy of the same model would reject them again.

code

yaml · 18 lines
yaml
# illustrative gateway route config; every field name is invented
routes:
  contract-extraction:
    fallback_chain:
      - extractor-model @ vendor-a / region-east / account-1
      - extractor-model @ vendor-a / region-west / account-2
      - extractor-model @ host-b / region-east
    fall_back_on: [server_error, overloaded, rate_limited, no_first_byte]
    never_fall_back_on: [invalid_request, content_refused]
    when_exhausted: return_error
  ticket-summary:
    fallback_chain:
      - summary-model @ vendor-a / region-east / account-1
      - summary-model @ host-b / region-east
      - other-summary-model @ vendor-c / region-east
    fall_back_on: [server_error, overloaded, rate_limited, no_first_byte]
    never_fall_back_on: [invalid_request, content_refused]
    when_exhausted: return_error

go deeper

for a junior

Define a fallback chain as an ordered list of model targets, name the failures that move a call on, and say why a malformed request does not.

for a middle

Explain the usual order, same model elsewhere before a different model, and why each feature's output contract and prompt tuning make the chain a per-route choice.

for a senior

Show that you would make chain exhaustion a clear error, keep refusals and caller errors out of failover, and split ownership between platform and feature teams.

for a principal

Frame the order as a trade between supplier independence and output fidelity, and say which default the platform sets and which choices it leaves to route owners.

## What a fallback chain is A **shared model gateway** is the one internal endpoint every team calls instead of calling model providers directly. A **route** is a named entry on that gateway that one feature calls, such as `ticket-summary` or `contract-extraction`. A **target** is one concrete place a call can go: a particular model, served by a particular provider or host, in a particular region, under a particular account. A **fallback chain** is the ordered list of targets a route may use. The gateway sends the call to the first healthy target; if that call fails in a way another target could fix, it moves to the next entry, and so on until one answers or the chain runs out. ## What moves a call down the chain Not every failure should move a call on. The question to ask is whether the next target could plausibly succeed where this one failed. | Upstream outcome | Walk the chain? | Why | |---|---|---| | Server error or "overloaded" response | Yes | The target is struggling; another target may not be | | HTTP 429 on this API key or account | Yes, to a target with remaining quota | The limit is local to this target | | Connection failure or no first byte in time | Yes | The target is unreachable or stalled | | HTTP 400 for a malformed request | No | A copy of the same model rejects it again | | A content-policy refusal | No | Shopping for a more lenient target defeats the policy | Treating the last two as failover triggers wastes calls, adds latency and, for refusals, quietly turns the chain into a way around a safety decision. ## The usual order: least disruptive first Most teams order a chain by how much the answer could change if that entry replies: 1. **The same model in another region or under another account** of the same provider: the same model and the same request, with separate capacity and failure domain. 2. **The same model on another host**, for example an open-weight checkpoint that several hosts serve, or a closed model resold through a second cloud: nearly the same behaviour, a more independent failure domain. 3. **A different model**: the most independent supplier, and the biggest change in output. 4. **A clear "chain exhausted" error** naming the targets tried, so the application can degrade in its own way. Chains are usually short, two to four entries, because every failed attempt adds latency before the caller hears anything, and every extra entry is one more target that must be kept tested. The order is a trade between independence and fidelity. A different vendor fails independently of the primary, but it also answers differently, which is why it usually sits last rather than first. ## Why per route, not one company-wide list A single company-wide list assumes every feature tolerates the same substitutes. They do not: - **Output contracts differ.** A route that parses structured output or relies on tool calls breaks when a model with different habits answers; a free-text summary rarely does. - **Prompts are tuned per model.** A prompt refined against one model can lose quality on another, so a feature may only accept entries it has evaluated. - **Latency budgets differ.** An autocomplete route may prefer an error over a slow fallback; an overnight report does not care. - **Cost differs.** A fallback to a larger model can cost several times more per call, which a high-volume route may not be allowed to absorb. - **Who knows the consequences differs.** Only the feature team knows what a substitute breaks, so they choose the entries; the platform team owns the mechanism and the list of approved targets. ## What the chain does not cover A fallback chain says *where the call goes next*. How many times to retry the same target first, how long to wait before giving up, and breaker state machines are general resilience patterns covered with retries and circuit breakers. What a feature shows when the whole chain fails is graceful degradation in the application. Making one request shape fit each provider in the chain belongs to the gateway's request translation. The chain itself is the gateway's routing decision: which targets, in what order, for which failures.

  • Why should an HTTP 400 for a malformed request not move a call down a fallback chain?
    A 400 says the request itself is wrong, so another copy of the same model rejects it again, and a different model may accept it only by ignoring the defect. Walking the chain spends calls and latency and hides a caller bug. The gateway should return the error to the caller at once and count it against the caller, not the target.
  • What should the gateway do when every entry in a route's chain has failed?
    Return one clear error that says the chain is exhausted and lists the targets tried and why each failed, rather than looping back to the first entry. The application then applies its own degraded behaviour. Looping turns an outage into a retry storm across every provider at once.
  • Who should own a route's fallback chain: the platform team or the feature team?
    Both, at different layers. The platform team owns the mechanism, the list of approved targets and a sensible default chain. The feature team chooses which approved entries its route accepts, because only it knows whether a different model breaks its parser, its tone or its latency budget.

Rebooking a cancelled flight: the agent first tries the same airline's next departure, then a partner on the same route, and only then a train. A traveller with a tight onward connection may refuse the train, so each traveller's acceptable list differs.

  • One company-wide fallback list is good enough for every feature.
  • Any error, including a malformed request, should try the next provider.
  • The first fallback should always be a different vendor's model.
  • When the chain runs out, start again from the top until something answers.
  • Fallback targets need no testing until a real outage uses them.
open as a page

Behind a shared model gateway, why should no team hold the provider's API key, and what does a per-team gateway key buy?

level: juniorimportance: must knowfreq 58%

basics

~20 s

A provider key that teams hold directly cannot be capped, attributed or revoked per team. Giving each team its own gateway key keeps the real key in one place and lets the gateway scope, cap, trace and revoke each caller separately.

open as a page

What does a shared model gateway's request log capture that in-app tracing misses, and what does it miss in turn?

level: juniorimportance: must knowfreq 45%

basics

~20 s

A gateway log records every team's model calls in one schema: the exact request sent upstream, the resolved model, response, usage and errors. It misses application context: the user action, where prompt text came from, tool execution and what happened next.

open as a page

How much latency does a model gateway add to each call, and how do you measure it separately from model time?

level: middleimportance: must knowfreq 44%

basics

~20 s

A model gateway usually adds a few milliseconds per call: the extra hop, key and cap checks, and translation. Measure it inside the gateway as time before the upstream call plus time after it, per route, at p50 and p99.

open as a page

For a model gateway fallback entry, what changes with the same model on another host versus a different model, and how is each gated?

level: middleimportance: must knowfreq 52%

basics

~20 s

The same model on another host mostly keeps behaviour but can differ in version, precision and limits; a different model changes format adherence, tool use, refusals and token counts. Gate each entry on the route's own evals.

open as a page

A model gateway checks spend caps before calls but learns costs after: how far can spend overshoot, and how do reservations bound it?

level: middleimportance: must knowfreq 50%

basics

~20 s

Each call admitted under the cap can still spend, so overshoot reaches in-flight calls times their worst cost, plus settlement lag. Holding each call's maximum at admission, then settling the actual, keeps spend within the cap when that maximum is a true ceiling.

open as a page

When a model gateway's target provider lacks a requested parameter or feature, should the gateway drop it, reject the call, or emulate it?

level: middleimportance: must knowfreq 50%

basics

~20 s

Decide per field by what ignoring it costs: reject anything that changes the output's meaning or shape, emulate only what the gateway can reproduce exactly, drop only harmless hints, and report every drop or emulation on the response.

open as a page

Before a model migration you replay a week of gateway-logged requests on the candidate; how do you compare answers when the log holds no labels?

level: seniorimportance: must knowfreq 42%

basics

~20 s

Rerun the incumbent on the same requests to measure its own noise floor, then compare the candidate on label-free signals: format validity, tool-call agreement, refusals, truncation, field-level or semantic diffs, and human review of sampled disagreements.

open as a page

Why should applications calling a shared model gateway name gateway-owned model aliases instead of provider model names?

level: juniorimportance: should knowfreq 40%

basics

~10 s

A gateway-owned alias names a job, not a supplier, so the platform repoints one reviewed config entry instead of every application editing and redeploying; each response still reports which provider and model actually answered.

open as a page

How does a model gateway decide that one provider target is unhealthy enough to eject from a route, and readmit it without flapping?

level: middleimportance: should knowfreq 48%

basics

~20 s

A gateway ejects a target when live calls show its server-error, overload or slow-first-token rate crossing a bar over enough requests, then readmits it after a cool-off on a small, growing share of traffic against a stricter bar.

open as a page

What must a model gateway's logged request contain so it can be replayed faithfully on another model months later?

level: middleimportance: should knowfreq 40%

basics

~20 s

The exact rendered messages, the resolved dated model rather than an alias, every effective parameter including defaults, the full tool definitions and tool mode, any output schema, the prompt version, and the original response as a baseline.

open as a page

What should a model gateway normalise when providers report finish reasons and token usage differently, and what does that mapping lose?

level: middleimportance: should knowfreq 35%

basics

~20 s

Map finish reasons onto a small gateway enum and usage onto fields with defined inclusion rules, keep the raw provider values beside them, and mark missing usage as unknown; distinctions a provider never reports, and cross-provider token comparability, are lost.

open as a page

When your model gateway is down, should applications fail open straight to the provider or fail closed, and what does a bypass lose?

level: seniorimportance: should knowfreq 38%

basics

~20 s

Decide per route. Failing closed keeps key custody, spend caps and logs but loses the feature; a bypass keeps it up but its calls go uncapped and unlogged, so bound it and trigger it only when the gateway is unreachable.

open as a page

How do you run a model gateway so that one bad release or one failed node cannot take every AI feature down at once?

level: seniorimportance: should knowfreq 34%

basics

~10 s

Run the gateway as stateless replicas across zones, sized by open calls, and cap each release's reach: canary with automatic rollback on per-route errors, staged config pushes, drained shutdowns, and independent cells.

open as a page

During a launch, the primary provider's region fails and a model gateway moves all traffic to the fallback provider, which throttles within minutes: what went wrong, and how do you size the fallback?

level: seniorimportance: should knowfreq 45%

basics

~20 s

The fallback held only trickle capacity, its limits were scoped wider than the keys spread over them, and other customers failed over too. Size each protected route's next hop to carry the primary's peak, spread over separate accounts and regions.

open as a page

An intern's agent loop drains a team's monthly model budget over a weekend: what should the gateway do when a budget runs out mid-stream or mid-run, and what should callers see?

level: seniorimportance: should knowfreq 42%

basics

~20 s

Refuse the next call, not the one in flight: admitted calls finish within their reservation, and new ones get a distinct budget-exhausted error that clients treat as terminal. Giving each agent run its own small capped key stops one loop draining the team.

open as a page

A provider API key behind your model gateway turns up in a public repository: how do you replace it without client changes, and stop it recurring?

level: seniorimportance: should knowfreq 45%

basics

~20 s

Mint a new provider key, load it into every gateway replica, confirm traffic succeeds on it, then revoke the leaked one; clients hold only gateway keys, so none changes. Then close the bypass that let a copy exist outside the gateway.

open as a page

How do you keep a bulk replay of gateway-logged traffic from firing side effects, exhausting provider quota or polluting production cost and quality figures?

level: seniorimportance: should knowfreq 30%

basics

~20 s

Treat the replay as a separate tenant: outputs are recorded but never executed or delivered, it runs on its own key and cap at a paced rate, and every request carries a replay tag that dashboards, health checks and future samples exclude.

open as a page

How would you design a model gateway's single stream format over providers that emit text deltas, tool-argument fragments, usage and mid-stream errors differently?

level: seniorimportance: should knowfreq 33%

basics

~20 s

Define the gateway's own event vocabulary with exactly one terminal event, translate each provider stream through a per-stream state machine, merge usage and finish into final events, and report mid-stream failures as in-stream errors, never as a clean finish.

open as a page

How should a model gateway translate tool definitions, tool calls and tool results between providers whose call identifiers, parallel-call support and result placement differ?

level: seniorimportance: should knowfreq 38%

basics

~20 s

Treat definitions, calls and results as three translations: check schemas and tool-choice modes against what the target supports, mint call identifiers the next turn can map back without gateway state, and rebuild where results go, reporting any lossy rewrite.

open as a page

Should a company build its own model gateway, self-run an open-source proxy, buy a managed one or have none, after one gateway release took every AI feature down?

level: principalimportance: should knowfreq 29%

basics

~20 s

Usually keep the gateway and fix release practice; one outage rarely proves the concept wrong. No gateway suits one team on one provider, a self-run proxy a fast start, managed when nobody can run it, built when fit demands.

open as a page

As the lead who owns a company's model gateway, which model substitutions would you let it make silently, and what would you alert on when it substitutes?

level: principalimportance: should knowfreq 38%

basics

~20 s

Allow silent substitution only where answers stay equivalent: the same model in another region, account or approved host. Require route opt-in and a visible answering-model field for a different model, and alert on each route's fallback share, not errors.

open as a page

How would you set nested organisation, team and key budgets on a model gateway, and where should a spend cap be soft rather than hard?

level: principalimportance: should knowfreq 36%

basics

~20 s

A call must fit every budget above its key, so the tightest level binds. Hard caps suit development keys, batch jobs and agent runs; customer-facing features usually get a soft cap that escalates to an owner, backed by a higher hard ceiling.

open as a page

As owner of a shared model gateway, how do you set per-route body logging and read access when legal asks who can read logged prompts?

level: principalimportance: should knowfreq 32%

basics

~20 s

Log metadata on every route and full bodies only on routes whose owners opt in for a stated purpose such as debugging or replay. Store bodies apart, and let only the owning team and named roles read them through an audited path.

open as a page

As the lead of a company model gateway, would you adopt the chat-message request shape most providers accept or define your own, and when would you allow native pass-through?

level: principalimportance: should knowfreq 32%

basics

~20 s

Most companies adopt the widely accepted chat-message shape with namespaced extensions, define their own only when the common core blocks real features, and allow native pass-through per route for provider-only features, keeping key custody and caps but losing portability.

open as a page

A model gateway's upstream stream fails after 200 output tokens are already on the user's screen: can the gateway fail over, and what should the user see?

level: seniorimportance: nice to knowfreq 30%

basics

~20 s

Not invisibly: once tokens reach the client, a fallback cannot seamlessly continue the same answer. Per route, the gateway either ends with a clear error, restarts on the fallback after an explicit reset signal, or asks the fallback to continue.

open as a page

Why does replaying a gateway-logged multi-turn conversation on a new model stop being faithful after the first turn that changes, and what do you replay instead?

level: seniorimportance: nice to knowfreq 22%

basics

~20 s

Each later user message reacts to the old model's reply, so once the candidate answers differently the logged next turn no longer follows. Replay each turn with the logged history as a fixed prefix, and record where the candidate first diverged.

open as a page