Rate Limiting & Throttling

## What the mechanism actually does Rate limiting is a mechanism that caps the number of operations a given identity is allowed to perform within a defined time window. That identity may be: - a user - an API key - an IP address - or the system as a whole Mechanically, every incoming request carries or implies a **key**: - the API key in a header - the authenticated user ID - the source IP for anonymous traffic The server, or a dedicated limiter sitting in front of it, looks up a counter or budget tied to that key and compares it against a configured threshold, for example 100 requests per minute. 1. If the request is within budget, the limiter records the usage and lets the request through. 2. If the client has exceeded the threshold, the limiter short-circuits the request before it reaches business logic and responds with HTTP status **429 Too Many Requests**. This is deliberately distinct from a 5xx server error: 429 tells the caller "the server is fine, but you personally must slow down," which is actionable information a generic 500 or a dropped connection is not. ## Why it exists The reason rate limiting exists is **capacity protection and fairness**. Backend capacity is finite — database connection pools, CPU, memory, and quotas on third-party services called on a client's behalf all have hard ceilings. Without a limiter, one misbehaving client (a buggy retry loop, a scraper, a compromised script) can consume all of that capacity and degrade the service for every other client — the classic "noisy neighbor" problem in a multi-tenant system. Rate limiting also defends against abuse patterns such as: - credential-stuffing login attempts - content scraping - brute-force enumeration and it lets a business meter and monetize usage, since many SaaS APIs tie limits directly to pricing tiers. Beyond protecting the provider, a well-designed 429 is also a service to the caller: it converts an ambiguous failure (timeout, connection reset) into an explicit, machine-readable signal the caller can act on immediately rather than discover through trial and error. ## The retry signal The single most important piece of that signal is the `Retry-After` header, which tells the client either a number of seconds to wait or an HTTP-date after which it may try again. Complementary headers most production limiters also return, though none are a single universal standard: | Header | What it carries | |---|---| | `X-RateLimit-Limit` | the ceiling | | `X-RateLimit-Remaining` | budget left in the current window | | `X-RateLimit-Reset` | when the window rolls over | Together these let a well-behaved client throttle itself proactively, spacing requests out before ever hitting a 429, rather than reactively hammering the endpoint and backing off only after rejection. ## The core trade-off The core trade-off is **protecting the server versus rejecting legitimate traffic**. - **Set the threshold too low, or key it too coarsely** — by IP address, say, when many real users sit behind the same corporate NAT or mobile carrier gateway — and false positives appear: real users get 429s during ordinary usage spikes, like a mobile app's retry burst after regaining connectivity. - **Set it too high or key it too finely**, and a single compromised identity can still degrade shared resources before the limiter engages. Choosing the key and the threshold is therefore a product and capacity-planning decision as much as an engineering one, and it usually needs different values per endpoint — a login endpoint needs a much tighter limit than a read-only search endpoint because the cost and abuse profile differ. ## Failure modes Failure modes cluster around **omission of the retry signal** and **inconsistency across the fleet**. 1. **A bare 429 with no `Retry-After`** leaves poorly written clients — including some naive retry libraries — looping immediately, turning a single overloaded moment into a self-inflicted thundering herd that keeps the service pinned at capacity. 2. **A limiter implemented per application instance with in-memory counters behind a load balancer** is another common failure: each instance enforces "100/minute" independently, so a client fanned out across ten instances effectively gets 1,000/minute, defeating the limit's purpose — the classic argument for either sticky routing or a shared, centralized counter store. 3. **Clock skew** between distributed limiter instances or between client and server can also cause a `Retry-After` value to be honored too early or too late. ## Where it shows up A concrete, widely known example: - **GitHub's REST API** returns rate-limit status via X-RateLimit-Limit/Remaining/Reset headers on every response and rejects further calls once exhausted until the reset time. - **Stripe's API** similarly returns 429 with `Retry-After`, and its official client libraries implement exponential backoff that specifically honors that header rather than retrying on a fixed interval — the behavior any well-designed client should replicate.