One shared in-memory store serves thirty autoscaling services, each instance holding its own pool - how do you budget its connection ceiling across them?

## The demand is a product, not an observation Nobody exceeds a connection ceiling on purpose. It happens because the number that lands on the tier is a multiplication that no single team performs: **per-instance maximum pool size x that service's maximum instance count, summed over every service, plus everything else that connects.** The last clause is where budgets quietly fail. Operator sessions, metrics collectors, batch and migration jobs, one-off scripts and anything running alongside the application all consume slots, and none of them appear in a service's own sizing discussion. A worked case: a fleet permitted to reach 400 instances, each with a pool capped at 50, demands **20,000 slots**. The same fleet's genuine concurrency, derived from arrival rate times hold time, might be a couple of hundred calls in flight. The gap between the two numbers is pure administrative waste, and it is the gap that reaches the ceiling. ## Why the ceiling binds before work capacity does On this tier a call is microseconds of the **server's service time** plus one round trip, so throughput capacity is enormous relative to the number of callers needed to consume it. The consequence is counterintuitive and worth saying explicitly in a design review: **the store runs out of connection slots long before it runs out of the ability to do work.** A tier sitting at a low fraction of its throughput can be entirely unreachable because every slot is held by an idle pooled connection. There is a second cost in the same direction. Each accepted connection carries per-connection state on the server, including its reply buffer, so a very large standing connection count consumes memory the keyspace wanted. How much varies sharply between stores of this class. ## The contract worth setting | Decision | The number to use | The number teams reach for instead | |---|---|---| | Per-instance pool maximum | That instance's own peak calls in flight, plus a margin | The client library's default | | Fleet multiplier | The autoscaler's configured maximum | The instance count running today | | Reserved share | Operators, collectors, jobs, incident scale-up | Nothing reserved | | Waiting workloads | Counted separately, one slot per parked caller | Blended into the same pool | | Ownership | A published number per tier, with a named owner | Nobody owns it | ## The incident feedback loop The worst case is not steady-state growth; it is a loop that fires precisely when the tier is already unwell: 1. Something slows calls down - a placement change adding a cross-zone hop, one expensive operation, a failover, a partition rebalance. 2. Hold time rises, so each instance's calls in flight rise, so each pool fills. 3. Request latency rises, and an autoscaler that scales on latency **adds instances**. 4. Each new instance opens a fresh pool, and every connection in it is new demand on the ceiling. 5. The ceiling is reached. New instances cannot connect at all, and if the store stops accepting rather than refusing, they hang rather than erroring - so the autoscaler's remedy has become the outage. The defence is arithmetic rather than reflex: the per-instance cap multiplied by the maximum fleet must fit under the ceiling **with the incident scale-up already counted**, not the comfortable steady state. ## Where stores and tiers differ Do not assume one shape for the limit: - Some stores expose an explicit, configurable ceiling; on others the **effective** ceiling is whatever the host's descriptors or per-connection memory allow, so demand shows up as an allocation failure rather than a clean limit. - Where the tier is **hosted**, the ceiling may be fixed by the instance size, in which case the only lever you have is demand. - **Behaviour above the limit differs** - a refusal in one product, a silent failure to accept in another - which decides whether exhaustion arrives as an error rate or as unattributed latency. - Some tiers are reached through an **intermediary that multiplexes many callers onto fewer connections**, which moves the ceiling to the intermediary rather than removing it, and adds its own limit to the budget. - Some **client libraries multiplex several in-flight operations over one connection**, so a service's slot demand can be far below its concurrency; the budget must be computed in the unit the library actually opens. ## What good looks like The tier's ceiling is a **published number with an owner**, each consuming service has an allocated share derived from its own concurrency arithmetic and its autoscaler's maximum, a margin is reserved for operators and for scale-up during an incident, and the current count sits among the tier's watched signals so the budget is approached visibly. The failure mode this prevents is the one that is otherwise guaranteed: a limit nobody owns, reached by a multiplication nobody did.