How would you decide how much memory per-thread allocator caches may hold across a fleet of services whose thread counts differ?
answer
- footprint is a product of four numbers
- linear in threads, always
- contention has diminishing returns
- cap the total, not each class
- decay stock from quiet threads
basics
~20 sCached memory scales as threads times size classes times blocks per class, so the decision is a curve: cache enough to keep the shared arena off the fast path, cap the per-thread total, and reclaim idle caches.
solid answer
~50 sStart from the arithmetic, because it is what makes the trade concrete: 64 threads holding 32 blocks in each of 40 size classes at an average 512 bytes park about 40 MiB of free memory that no other thread can use. Then note that the two curves run in opposite directions — contention falls steeply with the first blocks of cache and then flattens, while footprint rises linearly with both cache size and thread count. The policy follows: size caches from the measured arena-visit rate rather than by intuition, cap the total a single thread may hold instead of capping per class, decay or scavenge caches belonging to threads that have gone quiet, and treat a large thread pool as a footprint decision and not only a concurrency one. For services whose lifetimes cross threads heavily, a cache per core rather than per thread bounds the multiplier at the hardware rather than at the pool size.
go deeper
Remember that each thread's private stock is real memory, so more threads means more memory held even when the program is storing exactly the same data.
Do the multiplication out loud: threads times classes times blocks times block size. Being able to produce that figure is what turns a vague worry into a decision.
Show how you would find the knee: arena visit rate and cached bytes measured at several cache sizes on a representative workload, then re-checked whenever the pool changes.
Set a rule rather than a value. Per-thread totals, decay for idle stock, pools sized against cores, and an explicit exception process for services whose lifetimes cross threads and who therefore get less for the same memory.
## Where the memory actually goes A per-thread cache holds free blocks, and the amount is a product of four numbers: ``` cached bytes = threads x size classes x blocks per class x average block size ``` A concrete instance makes the scale obvious. Take 64 threads, 40 size classes, a cap of 32 blocks per class, and an average cached block of 512 bytes: - 64 x 40 = 2,560 populated lists - 2,560 x 32 = 81,920 cached blocks - 81,920 x 512 bytes = 41,943,040 bytes, which is exactly **40 MiB** Forty megabytes of memory that is free, reserved, and invisible to every thread but its owner. Double the pool to 128 threads and it is 80 MiB, with no change in the program's live data. This is the single fact a lead has to hold: **thread count multiplies allocator footprint even when it does not change what the program stores.** In practice the number is usually smaller, because a thread only populates the classes it actually uses, and larger where a service allocates across many classes or where per-class caps are generous. Both directions argue for measuring rather than estimating. ## The two curves that cross | Cache size per thread | Contention | Footprint | |---|---|---| | Zero | Every allocation meets the shared arena | None | | Small | Falls steeply; most allocations served privately | Small, linear in thread count | | Moderate | Falls to a floor set by cross-thread frees and oversized requests | Linear, now visible | | Large | Barely better than moderate | Linear and dominant | Contention has **diminishing returns** and footprint does not. That asymmetry is the whole argument: past the knee, more cache buys almost no throughput and costs memory in proportion to the pool. The defensible policy is to sit just past the knee, and the knee is found by measurement, not by picking a round number. ## The levers, in the order to reach for them 1. **Cap the total per thread, not each class.** A per-class cap multiplies by the number of classes, so a thread using many classes silently costs many times the intended budget. A single total, spent across whichever classes the thread actually uses, is both smaller and more predictable. 2. **Decay idle stock.** A thread that was busy and is now waiting on a queue should not keep holding blocks. Periodically returning a fraction of untouched stock to the shared arena bounds the cost of pools sized for peak. 3. **Retire caches of threads that end.** Short-lived threads are the classic way a cache design turns into retained memory. Their stock must flow back, and any blocks freed by other threads afterwards must still find a valid owner. 4. **Question the thread count itself.** Since footprint is linear in threads, a pool far larger than the core count costs memory for concurrency it cannot use. Fewer, busier threads usually reduce both cached memory and cross-thread traffic. 5. **Consider a cache per core rather than per thread.** This bounds the multiplier at the hardware, at the cost of needing coordination when two threads on the same core interleave. It is a real design point for services with thread counts far above core counts. ## Defending a number A lead is expected to say **how they would know**, not to quote a default: - Measure the **arena visit rate per thread** at several cache sizes on a representative workload; the knee in that curve is the target. - Measure **cached free bytes** at the same points, and multiply out to the pool size the service actually runs. - Check the **latency distribution**, because a big cache makes refills rarer but longer. - Re-measure after any change to the thread pool, since every number here is per thread. ## Policy across a fleet Different services sit in different places on that curve, so one global number is the wrong shape. What generalizes is the **rule** rather than the value: a per-thread total budget scaled to what the process is allowed to use, decay enabled everywhere, thread pools sized to cores rather than to optimism, and a standing expectation that raising a pool size is also a memory change. Services with heavy cross-thread lifetimes get a separate look, because for them the cache is delivering less and costing the same. The honest closing point is that this is a trade with no universal answer. A latency-sensitive service with a modest pool should sit past the knee and pay the memory; a service running many processes each with large pools should sit before it and accept a little contention. Making that call, with numbers attached, is the question.
- Why cap a thread's total cached bytes rather than the number of blocks per size class?Because a per-class cap is multiplied by however many classes a thread touches, so the real cost depends on a workload property nobody set. A single total is spent only on the classes actually used, is predictable across services, and stays meaningful when the number of size classes changes.
- What is the argument for a cache per core instead of per thread?It bounds the multiplier at the hardware rather than at the pool size, which matters most when thread counts far exceed cores. The cost is that the fast path is no longer single-owner, so it needs some coordination against other threads sharing that core, which gives back part of the benefit.
- How would you tell that cached stock, rather than live data, is behind a memory increase?Compare free-but-cached bytes reported by the allocator against the program's live set. A rise that tracks thread count while the live set is flat, and that reverses when caches are scavenged or the pool is shrunk, points at cached stock rather than retention.
saying these in an interview costs you the question
- Quotes one cache size as correct for every service
- Treats raising a thread pool as free in memory terms
- Assumes a bigger cache keeps improving throughput indefinitely
- Overlooks stock held by threads that have gone idle or exited
- Judges allocator memory by the live set alone, ignoring cached free blocks