skip to content

On a function platform that sends many requests to one instance, what changes when you raise the per-instance concurrency setting?

level: middleimportance: should knowfreq 42%

answer

  1. requests per instance, not instances
  2. fewer instances, fewer cold starts
  3. shared memory and CPU
  4. load-test one instance to the knee

basics

~20 s

Each instance holds more requests at once, so the fleet needs fewer instances, starts fewer cold and often costs less per request, until shared memory runs short or CPU contention stretches tail latency. Pick the value by load-testing one instance.

solid answer

~50 s

On platforms that let one instance serve several requests at once, the **per-instance concurrency** setting is how many in-flight requests the platform routes to an instance before it needs another. Raising it means fewer instances for the same load: fewer cold starts, less of an instance-counted quota used, more of a spike absorbed per new instance, and lower cost where billing follows instance time. The price: requests share the instance's memory and CPU, so memory must fit the peak per-request footprint times the setting, CPU-bound work queues behind itself and tail latency rises, and one heavy request slows its neighbours. Choose by measurement: load-test one instance, raise concurrency until p99 latency or memory crosses its budget, and set the value below that knee with headroom. Handler code that is not safe to run concurrently needs a setting of one.

go deeper

for a junior

Recall that on some platforms one instance serves several requests at once, and that a setting decides how many before another instance is started.

for a middle

Explain what improves as the setting rises (instances, cold starts, cost) and what degrades (shared memory, CPU contention, tail latency), and how to find the knee by testing one instance.

for a senior

Show you tune it from measured p99 and memory per instance, re-test when the code changes, and know how the setting also decides when the autoscaler adds instances.

for a principal

Decide whether teams get a measured per-service value or a conservative shared default, weighing density and cost against isolation and the effort of testing every service.

## What the setting controls Function platforms differ in how many requests one instance handles at a time. On a **microVM-based platform** the classic model is one request per instance: a second simultaneous request gets a second instance. A **container-instance platform** usually routes several requests to the same instance, and some function platforms offer the same as an option. On those **request-concurrent** platforms, the operator sets a **per-instance concurrency** value: the most in-flight requests one instance receives. When every instance is at that value, the platform starts another (or throttles, if a scale limit stops it). The setting is therefore a dial between two extremes: one request per instance (strong isolation, many instances) and many requests per instance (dense packing, few instances). What changes along that dial is the subject of this question, not whether the handler code is written to be concurrent, which is a design question of its own. ## What gets better as it rises - **Fewer instances for the same load.** By Little's law, a workload needs about request rate × duration in-flight requests; divide by the per-instance value to get instances. - **Fewer cold starts.** Fewer instances to start means fewer requests that pay initialisation. - **More spike absorbed per new instance.** If the platform's scale-up rate limits how many instances appear per interval, each instance that appears covers more requests. - **Lower quota use** where the regional quota counts instances rather than requests. - **Lower cost** where billing follows instance time, because overlapping requests share the billed time. Waiting on the network is cheap to overlap. ## What gets worse - **Memory.** All requests share one memory allocation. The instance must hold its baseline plus the peak per-request footprint times the setting; an out-of-memory crash kills every request in flight on it. - **CPU contention.** Requests that compute rather than wait compete for the same CPU. Median latency may barely move while **p99 latency** climbs sharply. - **Noisy neighbours inside one instance.** One slow or pathological request (a huge payload, a runaway loop) slows or crashes the others sharing its instance. - **Correctness.** Handler code that keeps per-request data in shared global state, or uses a library that is not safe to call concurrently, produces wrong answers when the setting is above one. - **Per-instance pools.** Connections and other pooled resources are now shared by more requests; the pool must be sized for the setting, or requests queue inside the instance. | | Low setting (one or a few) | High setting | |---|---|---| | Instances for the same load | Many | Few | | Cold starts | More | Fewer | | Memory per instance | Small, predictable | Must fit many requests at peak | | Tail latency under CPU-bound work | Stable | Rises with contention | | Blast radius of one bad request | One request | Every request on that instance | ## Finding the knee The right value depends on the workload, so it is measured, not guessed: 1. **Pin one instance.** Run a load test against a single instance (or a fleet capped at one) so the platform cannot hide contention by scaling out. 2. **Step the concurrency up.** Raise the number of simultaneous requests in steps, holding each step long enough for latency to settle. 3. **Watch the budget metrics.** Record p50 and p99 latency, CPU utilisation and memory at each step. 4. **Find the knee.** The point where p99 latency or memory leaves its budget is the practical ceiling. 5. **Set the value below it.** Leave headroom for requests heavier than the test mix, then confirm in production with the same metrics per instance. Input-output-bound handlers that mostly wait on other services tolerate high values; CPU-bound or memory-heavy handlers want low ones. ## Interaction with scale limits Many such platforms scale on how full instances are, aiming to keep each one below a target share of the setting, so the value also decides **when** new instances are added. Set it too high and the autoscaler waits until instances are already overloaded; too low and the fleet scales out early and pays for more instances. Re-run the test after any change that alters per-request CPU or memory, because the knee moves with the code.

  • Why can median latency look fine while p99 latency degrades as per-instance concurrency rises?
    Most requests still find spare CPU and finish normally, so the median barely moves. The slowest requests are the ones that landed on an instance when several CPU-heavy requests were running together; they wait for CPU and form the tail. Only percentiles near the top show that contention.
  • When is a per-instance concurrency of one the right choice on a platform that allows more?
    When the handler is not safe to run concurrently (shared mutable globals, a library that is not thread-safe), when each request needs most of the instance's memory or CPU, or when strict isolation between requests matters more than density and cost.

  • Per-instance concurrency changes nothing for memory, since each request gets its own allocation.
  • Raising per-instance concurrency always lowers latency because fewer cold starts happen.
  • The highest value the platform allows is the best default for every service.
  • Handler code with shared mutable state is safe at any concurrency setting.
  • Per-instance concurrency is another name for the regional concurrency quota.