A team moves rate limiting from each backend service into the shared API gateway in front of them. What does the gateway need to track to enforce limits correctly, and what breaks if the gateway is horizontally scaled to multiple instances without care?

## Why it moves to the gateway **Rate limiting** is the practice of capping how many requests a given client can make in a period of time, protecting backend capacity from being overwhelmed by a single noisy client, whether that's a buggy retry loop, a scraper, or a deliberate denial-of-service attempt. Offloading it to the gateway, rather than implementing it inside every backend service, is attractive for the same reason authentication and TLS offloading are: it's a cross-cutting concern that behaves identically regardless of which service is being called, so implementing it once avoids inconsistent per-service limits and duplicated logic. ## What the gateway has to track Mechanically, the gateway identifies the caller, usually by API key, authenticated user ID, or source IP as a fallback, and maintains a **counter** tied to that identity plus a time window. Two algorithms dominate in practice. | Algorithm | How it counts | |---|---| | **Token bucket** | assigns each client a bucket that holds up to N tokens; each request consumes one token, and tokens refill at a steady rate over time, which naturally allows short bursts up to the bucket size while enforcing a steady average rate over the long run | | **Sliding window** | counts requests in overlapping time slices to avoid the classic fixed-window problem where a client can send double the intended limit by timing requests to straddle a window boundary (for example, bursting at the end of one minute and the start of the next) | Whichever algorithm is used, once a client exceeds its allotment, the gateway rejects the request immediately, typically with an HTTP `429 Too Many Requests` status and a `Retry-After` header telling the client how long to back off, without the request ever reaching, or costing capacity from, any backend service. ## The scaling problem The scaling problem emerges the moment the gateway itself is no longer a single process. A single gateway instance can track counters cheaply in local memory: fast, no network hop, no extra infrastructure. But production gateways are almost always horizontally scaled behind a load balancer for availability and throughput, meaning a given client's requests get distributed across several gateway instances, often somewhat randomly depending on the load balancing algorithm. If each instance keeps its own local counter, none of them individually see the client's full request volume, so a client sending 1000 requests split evenly across 5 gateway instances looks like only 200 requests to each instance, and a limit set at 500/instance would let 1000 through when the intended limit was 500 total. This **under-counting** is the central failure mode: rate limiting appears to be working (no errors, counters look fine) while the actual aggregate rate silently blows past the configured ceiling, often only discovered when a backend gets overwhelmed anyway despite "working" rate limiting. ## The standard fix and what it costs The standard fix is to move the counter state out of each gateway instance's local memory and into a shared, low-latency store that every instance reads and writes atomically, almost always **Redis**, using atomic increment-and-check operations (or Redis's built-in Lua scripting) to avoid race conditions where two instances both read a stale count and both allow a request that together exceeds the limit. This restores correctness but introduces a new dependency and a small amount of added latency per request, plus a new failure mode of its own: if the shared store becomes slow or unavailable, the gateway has to decide whether to - **fail open** (allow all requests, risking overload), or - **fail closed** (reject all requests, risking a full outage over what should be a soft-limiting feature), and that choice needs to be deliberate, not accidental. Some teams accept an approximate compromise instead: divide the intended global limit by the number of gateway instances and enforce that smaller limit locally per instance, trading precision (the real limit can drift somewhat above or below intent as instance count or traffic distribution changes) for avoiding the shared-store dependency entirely. ## Where it shows up in practice A well-known real-world instance of this pattern is Kong or Envoy's rate-limiting filter backed by a Redis cluster, and cloud API gateways like AWS API Gateway or Azure API Management, which expose managed, distributed rate limiting specifically because customers running these gateways at scale hit exactly this local-counter-undercounts problem when they first tried to DIY it with in-memory counters on auto-scaled gateway fleets.