When a service decides it can't safely accept a request within its throttling policy, its three main options are to reject it outright, queue it briefly, or serve a degraded response. Walk through the trade-offs of each and how you'd decide which to apply to a given request type.

## Three answers to one decision Once a throttling decision determines that a request exceeds the currently sustainable capacity, the service needs a concrete answer for what to actually do with that specific request, and there are three broad options, each appropriate for a different kind of request and a different kind of caller. | Option | What the service does with the request | What it costs the caller | |---|---|---| | **Rejecting** | returning an explicit failure immediately | the cost of rejection is entirely pushed onto the caller | | **Queueing** | holding the request briefly at the admission boundary | added latency instead of an outright error | | **Degrading** | serving a deliberately cheaper version of the response | costs correctness, freshness, or completeness | ## Rejecting **Rejecting** means returning an explicit failure immediately, without doing further work. Mechanically this is the cheapest option for the service — it spends almost no CPU, memory, or downstream capacity handling a rejected request, which is exactly the point: the whole reason to reject is to avoid spending resources on work the system can't currently sustain. The cost of rejection is entirely pushed onto the caller, who now has to detect the failure and decide what to do — retry later, show the end user an error, or fall back to some other path. Rejection is the right default: - for requests that are safe to retry (idempotent operations) - for callers sophisticated enough to back off sensibly instead of hammering the service again immediately - and for any request where a quick, unambiguous failure is preferable to an uncertain wait — a client waiting on a slow response often has a worse experience than one that gets an immediate, clear error it can act on. ## Queueing **Queueing** means holding the request briefly at the admission boundary rather than immediately answering it, and serving it once capacity frees up, so the caller experiences the overload as added latency instead of an outright error. This is a genuinely different user experience — many callers would rather wait two extra seconds than see an error — but it only works under two conditions: - the queue has to be small and bounded - the arrival rate has to be expected to drop back below the drain rate soon This is deliberately a short-lived, admission-time holding mechanism, distinct from the durable, application-level buffering used to smooth bursts over a longer window (a separate pattern built around a persistent message queue between producer and consumer) — the throttling-level queue exists only to smooth momentary admission spikes, not to durably absorb sustained overload. If the queue isn't bounded, or the overload doesn't actually subside, queueing doesn't prevent overload, it just delays and disguises it — held requests keep accumulating, memory pressure grows, and by the time requests are finally served, their latency may already have breached the caller's own timeout, so the caller experiences a slow failure instead of a fast one, which is often worse than an immediate rejection would have been. ## Degrading **Degrading** means serving a deliberately cheaper version of the response instead of doing the full work — returning cached or slightly stale data instead of a freshly computed answer, skipping a non-critical enrichment step, or serving a simplified result. This preserves both availability and flat latency, since the degraded path is by design cheap to serve, but it costs correctness, freshness, or completeness, so it's only appropriate where an approximate or slightly-out-of-date answer doesn't cause real harm to the caller. A product recommendation list can safely fall back to a generic, cached list under load; a payment authorization amount cannot be 'approximately right.' Deciding which requests are eligible for degradation is a product-level judgment as much as a technical one, and it typically requires having actually built the cheaper fallback path in advance, which is engineering effort spent before the incident, not during it. ## Choosing for a given request type In practice, the choice among the three tracks two questions about the specific request type: 1. how bad is a wrong-but-plausible answer compared to no answer at all 2. and how well can the caller handle each kind of response? - **A background batch job** calling an internal API can usually tolerate rejection with backoff just fine. - **A user-facing checkout total** must be either fully correct or explicitly failed — never approximated — making it a strong candidate for rejection (or, if slightly delayed completion is safe, a short bounded queue) but a poor candidate for degradation. - **A homepage's 'related products' widget** is the opposite: correctness barely matters, so serving a stale cached list under load (degrade) is clearly better than making the user wait or showing an error for a non-critical element. ## Priority-based shedding Systems in production often combine strategies by priority: under load, low-priority, degradable requests (recommendations, non-critical background sync) get degraded or rejected first, freeing capacity to keep serving the high-priority path (checkout, authentication) at full fidelity — a form of load shedding that treats not all requests as equally worth protecting.