What is a 'concurrency limit' in a serverless compute platform, and what happens to incoming requests when a function's executions hit that limit at the same moment?

## Concurrency versus throughput Concurrency in a serverless platform is the count of function instances that are **actively executing at a single instant in time** — it is fundamentally different from throughput (requests per second). - A function invoked 1,000 times per second that finishes each invocation in 50ms only needs about 50 concurrent execution slots at any given moment (roughly requests-per-second times average duration, a relationship similar to **Little's Law** from queueing theory). - The same 1,000-requests-per-second rate with a 2-second average duration needs roughly 2,000 concurrent slots. Every serverless platform enforces some ceiling on this concurrent-execution count, and critically, that ceiling is very often shared across an entire account or region rather than allocated per function by default — every function without special configuration draws from the same finite pool. ## Why the ceiling exists This limit exists for two protective reasons. 1. First, it protects the platform's own **physical and orchestration capacity**: unbounded, instantaneous scale-out for every customer simultaneously isn't something any provider's underlying infrastructure can guarantee without some ceiling. 2. Second, and just as important in practice, it protects **whatever your function talks to downstream**. A serverless platform can scale out concurrent executions far faster than a traditional fixed-size application server fleet ever could, so without a ceiling, a burst of traffic could spin up thousands of concurrent function instances almost instantly, each potentially opening its own connection to a database, calling a third-party API, or writing to a shared resource — capacity that those downstream systems were never sized to absorb. ## What happens when the limit is hit What happens when the ceiling is hit differs by invocation type. | Invocation type | Platform behaviour | |---|---| | **Synchronous** — a client making an HTTP request and waiting for a response | Once the concurrency ceiling is reached, new requests are typically rejected immediately with a throttling error (commonly modeled as an HTTP 429 Too Many Requests), and it's the caller's responsibility to retry, ideally with backoff and jitter to avoid making the situation worse. | | **Asynchronous** — a function triggered by a queue message, a storage event, or a scheduled rule | The platform usually doesn't drop the work; instead it buffers the event and retries the invocation automatically after a delay, following a retry policy, and after enough failed retries routes the event to a dead-letter queue or configured failure destination so it isn't silently lost. | ## Failure modes The most important production trade-off is the **shared-pool effect**: because the concurrency ceiling is often account- or region-wide, one function's misbehavior or legitimate success can throttle every other function in that scope, even ones with no logical relationship to the spike. - **A bug that re-invokes itself.** A classic failure mode is a function with a bug that causes it to recursively re-invoke itself (for example, an error handler that republishes to the very queue that triggers it); within minutes it can consume the entire shared concurrency pool, and unrelated production functions — a payment processor, an authentication check — start failing with throttling errors even though nothing about their own code or traffic changed. - **The downstream-connection-storm.** A second common failure mode is the downstream-connection-storm: a burst of concurrent function executions each opens its own connection to a relational database sized for a handful of long-lived connections from a traditional app server; the database's connection pool exhausts almost instantly, and the outage looks like a database problem when its root cause is unmanaged serverless fan-out. ## Where it shows up A concrete real-world scenario is an event-driven pipeline where a serverless function is triggered by messages arriving on a queue after a bulk file upload. If nothing constrains that function's concurrency, a single large upload can drive it to scale out to thousands of simultaneous executions almost immediately, each one hitting the same downstream relational database to record results — a pattern platform teams commonly mitigate by deliberately capping that function's concurrency (via a **reserved concurrency** setting) to a number the database and its connection pool can actually sustain, deliberately trading maximum theoretical throughput for downstream stability, and accepting that some messages will simply take longer to process during a burst rather than taking the database down entirely.