Which scaling axis does an in-memory tier need when memory, server throughput, or connection capacity is the scarce one?

An in-memory tier runs out of *something* long before it runs out of everything. The three shortages an operator actually meets — room for the data, server time to execute operations, and connections to accept callers — look alike from a distance, because all three reach you as "the tier is slow" or "the tier is failing". They call for three different moves, and two of those moves can make one of the other shortages worse. ## Name the scarce resource, and its frame Before any scaling decision, say which number is at its limit and who measured it: - **Room for the data** — `stored-data size`, the sum the server counts for the entries it holds, approaching the **memory ceiling** the server enforces on itself. This is a different number from the **resident footprint** the operating system sees the process holding, and different again from the host or container limit above both. - **Server time** — the tier's own service time per operation rising as the request rate rises. A high request rate by itself is not a shortage; it is a tier doing its job. - **Connections** — callers refused, or made to wait for a connection, before a request is ever sent. Only the first two are usually about the tier's capacity at all. ## What each axis actually adds | axis | adds | does not add | |---|---|---| | a bigger node | memory; cores the server may or may not use | a shorter round trip; a second failure domain | | more nodes, keyspace partitioned | memory and aggregate execution capacity | a shorter round trip; simpler multi-key work | | more nodes, each a full copy | read capacity, and somewhere to continue from | any room for more data | The third row is where answers usually go wrong. A copy holds the whole keyspace again, so ten copies are ten times the same data. Copies answer availability and read-capacity questions; they never answer "we need to hold more". ## When memory is the scarce one A bigger node is the direct answer and usually the cheaper one to operate: one process, one thing to watch, no change to how callers address keys. It stops being available for two reasons long before the hardware does — the largest instance on offer no longer holds the working data with headroom, or one process has become too large a failure domain to accept. Partitioning the keyspace across nodes is the other answer, and it is a permanent change in the tier's character rather than a bigger version of the same thing: which node owns a key becomes something the deployment has to answer, and work touching several keys at once stops being a local matter. Buy it when you need it, knowing it is not reversible cheaply. ## When server time is the scarce one Here the store's own design decides whether a bigger node helps at all, and this genuinely differs across this class of store. Where a single execution thread runs operations one at a time, extra cores sit outside the path operations take; a bigger machine can offer a faster core and a better network path, but the honest answer is **more serving processes** — several instances, or a partitioned keyspace. Where the server executes on many threads, extra cores do convert into throughput until something else binds. Establish which design you are on before promising anyone a number. Separate saturation from expense as well: one unusually costly operation can occupy a server while there is no shortage of capacity anywhere, and no new node repairs that. ## When connections are the scarce one The server's limit on concurrent connections is rarely reached because the server is busy. It is reached because the caller-side arithmetic is multiplicative: 1. connections held ≈ application instances × pool size per instance; 2. on a partitioned tier, multiply again by the nodes each client keeps open; 3. add anything that opens a connection outside the pool. Scaling the application horizontally multiplies that product, and partitioning the tier multiplies it again — which is why "add nodes" is occasionally the move that causes the incident it was meant to prevent. The repairs live with the caller: fewer processes holding pools, smaller pools with idle connections reclaimed, or a connection-sharing layer in front. Raising the server's limit is legitimate once you know what the number should be, because each accepted connection costs the server memory and attention. ## The move itself is not free Every axis is also a choice of interruption. A bigger node means standing up a replacement instance and moving callers to it. More nodes means moving key ownership while the tier keeps serving, which needs spare room on the nodes that stay. Neither is instant, so the axis is chosen in advance of the shortage, not during it. A complete answer therefore does four things: states the deployment shape and the store's relevant property, names the number that is scarce and its frame, picks the axis that adds that resource, and prices the move that gets you there. ## The right-hand column is the move that looks like capacity and adds none of the resource that is actually short ``` scarce resource | identified by | axis that helps | axis that adds nothing ------------------|------------------------------------------|------------------------------------|------------------------ room for data | stored-data size nearing the ceiling | bigger node; partitioned keyspace | more full copies server time | service time rises with request rate | more serving processes; partitions | more cores, one-thread server connections | callers refused before a request is sent | fewer/smaller caller pools | more nodes (pools multiply) ```