skip to content

Scaling Limits & Throttling

A function fleet scales until a quota or a ramp limit stops it. Interviewers ask how fast it can grow, how you would see throttling, and how you would load-test a spike first.

on this pageshow

questions

5

When a function platform throttles requests, why do the function's own logs stay clean, and where does the throttling show up instead?

level: juniorimportance: must knowfreq 62%

answer

  1. turned away at the front door
  2. a separate metric, not errors
  3. what the caller's status code means
  4. backlog age for queued work

basics

~20 s

A throttled request is rejected before the handler runs, so the function's logs and error count never see it. Throttling shows up in the platform's throttle metric, as too-many-requests errors at callers, and as rising backlog age for queued events.

solid answer

~50 s

Throttling happens at the platform's front door: when no concurrency is available, the request is turned away and no handler code runs, so there is no log line and the function's error metric stays flat. An operator sees it in three places. The platform reports **throttled requests** apart from errors, as a throttle count or a count of too-many-requests rejections. A synchronous caller gets a too-many-requests response, usually an HTTP `429`, which means *no capacity right now*, not a bug, so the caller should retry with exponential backoff and jitter. For asynchronous and queue-driven work nothing fails visibly at first: the platform holds and retries the events, or the messages stay in the queue, so the signal is the **age of the oldest waiting event or message** climbing. A dashboard that charts only invocations, errors and duration can look healthy in the middle of a throttling incident.

go deeper

for a junior

Remember that a throttled request never reaches the handler, so logs and error counts stay clean. Know the throttle metric exists and that a 429 means retry later with backoff.

for a middle

Explain how throttling looks different on the synchronous, asynchronous and queue-driven paths, and why retries need exponential backoff with jitter rather than a fixed interval.

for a senior

Show you can read throttles against concurrency to tell the ceiling from the ramp, and that you alarm on event or message age so quiet asynchronous throttling is caught before deadlines drop work.

for a principal

Decide which throttling signals every user-facing function must alarm on by default, and how gateways pass throttles through so callers keep the retry-later meaning.

## Where throttling happens A function platform runs your code in **instances** (sandboxes that each hold the handler). The number of requests it may run at once is bounded: by a **regional concurrency quota** shared by the functions in one region of an account or project, by the **scale-up rate** that limits how fast new instances appear, and by any **per-function concurrency cap** an operator has set. When a request arrives and none of those allow another execution, the platform **throttles** it: it refuses or defers the request at the routing layer. The key point for an operator is *where* that refusal happens. It happens **before an instance is chosen and before the handler is called**. Nothing in your code runs, so: - there is no log line from the handler, because the handler never started; - the function's **error** metric does not move, because an error is something an invocation returns, and there was no invocation; - the function's **duration** chart may even look better, because the requests that did get through ran on warm instances. That is why "the logs are clean" is not evidence that a spike went well. ## Signal 1: the platform's throttle metric Function platforms generally report throttled requests apart from errors: as a throttle count beside invocations, or as requests rejected with a too-many-requests code. It is the most direct signal and the first one to put on a dashboard and an alarm. Read it beside **concurrent executions**: throttles with concurrency near the regional quota point at the ceiling; throttles with concurrency well below it point at the scale-up rate or at a per-function cap. ## Signal 2: what a synchronous caller sees A **synchronous** invocation is one where the caller waits for the result, such as an HTTP request through a gateway. When it is throttled, the caller gets an error straight away, usually an HTTP `429 Too Many Requests` or the platform's equivalent throttling error. Some platforms first hold the request briefly in the hope that an instance frees up, and only then reject it, so a throttled call can also show up as extra latency before the error. What the error tells the caller: 1. **It is about capacity, not correctness.** The request itself was fine; resending it later may succeed. 2. **The platform will not retry it for you.** On the synchronous path the retry belongs to the caller. 3. **The retry must back off.** Retrying at once, or at a fixed interval, sends every throttled client back at the same moment and makes the next wave worse. Exponential backoff with random **jitter** spreads the retries out. If the gateway in front of the function turns the throttle into a generic `500` or `502`, callers lose that meaning, so it is worth checking how the error is passed through. ## Signal 3: the age of waiting work **Asynchronous** invocations (the caller hands over an event and does not wait) and **queue-driven** ones (the platform polls a queue or stream for you) fail quietly when throttled: | Invocation type | What happens on a throttle | Where the operator sees it | |---|---|---| | Synchronous | Rejected at once (sometimes after a short hold) | Throttle metric, `429`s at the caller, gateway error rate | | Asynchronous | The platform keeps the event and retries it later, up to a maximum event age | Throttle metric, the platform's event-age or pending-events metric | | Queue or stream | The poller slows down; messages stay in the source | Age of the oldest message, queue depth, stream iterator age | In the last two rows no request is lost *yet*, so nothing turns red. The danger is the deadline: an asynchronous event that is still throttled when it reaches its maximum age is dropped or sent to a failure destination, and a queue message that waits past its retention is gone. So the useful alarm is on **age**, set well below those limits, not on errors. ## Building a dashboard that shows it A throttling-aware view of a function fleet puts these side by side: - throttle count per function, with an alarm on any sustained non-zero value for user-facing functions; - concurrent executions against the regional quota, so you can tell the ceiling from the ramp; - the caller's view: `429` rate at the gateway or client; - for event-driven functions, the oldest event or message age. With those four, the classic incident pattern, "errors flat, duration fine, customers complaining", becomes visible in the first minute instead of after the ticket queue fills up.

  • Why should a client that retries a throttled call add random jitter to its backoff?
    Throttling hits many clients at the same moment. If they all wait the same backoff, they all return together and form a second spike that is throttled again. Random jitter spreads the retries across the backoff window, so the platform sees a smoother arrival rate it can serve as new instances come up.
  • Which alarm catches throttling of queue-driven work before any message is lost?
    An alarm on the age of the oldest message in the queue, with a threshold well below the queue's retention period, paired with the function's throttle count. Error alarms stay silent here, because throttled messages simply wait; only their growing age shows the function is not keeping up.

  • Throttled requests show up in the function's own logs with an error and a stack trace.
  • A flat error rate during a traffic spike proves the function kept up.
  • A too-many-requests response means the handler code has a bug.
  • Throttled asynchronous events are dropped the moment they are refused.
  • Retrying a throttled call immediately, in a tight loop, is the right client response.
open as a page

On a function platform that sends many requests to one instance, what changes when you raise the per-instance concurrency setting?

level: middleimportance: should knowfreq 42%

basics

~20 s

Each instance holds more requests at once, so the fleet needs fewer instances, starts fewer cold and often costs less per request, until shared memory runs short or CPU contention stretches tail latency. Pick the value by load-testing one instance.

open as a page

How do you size a per-function concurrency cap that keeps a function from flooding a rate-limited downstream API, and what does the cap fail to guarantee?

level: seniorimportance: should knowfreq 48%

basics

~20 s

Set the cap near the downstream's sustainable rate times the function's average duration. A cap bounds requests in flight, not per second, so a faster function can still breach the rate limit, and the excess lands on callers or a backlog.

open as a page

Before a ticket sale expected to send fifty times normal traffic within a minute, how would you load-test a function-backed API to expose its scaling limits?

level: seniorimportance: should knowfreq 45%

basics

~20 s

Replay the ramp, not just the peak: climb from normal to fifty times inside a minute, starting cold, in an environment with production's quotas but not its shared pool, and watch throttles, cold-start share, latency and every downstream.

open as a page

For a ticket sale that will send fifty times normal traffic to a function-backed API, how do you decide between raising quotas, pre-starting capacity and metering users in?

level: principalimportance: should knowfreq 32%

basics

~20 s

Weigh what a throttled launch costs against what idle capacity and a waiting room cost. A quota raise lifts the ceiling, pre-started capacity skips the ramp but bills while idle, and metering protects every downstream at the price of user waits.

open as a page