What happens to throughput and latency when a machine has far more runnable threads than CPU cores, and how would you recognize that state from the outside?

## Runnable is the word that matters A thread is only competing for CPU when it is **runnable**: not blocked on I/O, a lock or a queue. A server with 5,000 threads of which 12 are runnable on 8 cores is mildly oversubscribed; the other 4,988 cost memory and bookkeeping, not CPU. Every statement below is about runnable count versus core count. ## What happens as runnable count rises Let C be cores and R the number of runnable threads. - **R < C**: cores idle. Adding threads adds throughput roughly linearly. - **R ≈ C**: the sweet spot for CPU-bound work. Every core busy, minimal switching. - **R > C**: no extra capacity exists. The scheduler time-slices, so each thread runs at roughly C/R speed. Total useful throughput stays flat *in the ideal model* and per-request latency grows linearly with R. - **R ≫ C**: throughput actively **falls**. This is the part candidates miss. ## Why throughput falls rather than plateaus Four compounding effects: 1. **Switch overhead.** Every preemption costs the switch plus re-warming caches, branch predictors and translation entries. With many runnable threads, each one is descheduled before its working set is fully re-established, so a growing fraction of every slice is spent re-warming. 2. **Cache capacity pressure.** R threads with distinct working sets share one cache hierarchy. As R grows, each thread's share shrinks and its miss rate rises — so each thread is slower *while running*, not just while waiting. 3. **Lock contention and hold time.** A thread preempted while holding a lock keeps holding it for the whole time it is off-CPU. Waiters pile up behind a holder that is not even running — the convoy effect — so contention grows superlinearly with R. 4. **Memory pressure.** Stacks and per-thread structures consume memory that would otherwise be cache or page cache. The useful theoretical framing: simple speedup models with a serial fraction predict an asymptotic ceiling — more workers never *hurt*. The universal scalability model adds a second, negative term for coherence and contention costs that grows faster, producing a curve with a **peak followed by decline**. Oversubscription is that decline made concrete. The practical lesson is that there exists an optimal concurrency level and running past it costs you both throughput and latency. ## Latency, especially the tail With R runnable threads on C cores, a thread's completion is stretched roughly by R/C plus the switch overhead. The median degrades, but the **tail** degrades far more: the unlucky request waits behind more competitors, and variance in wait time compounds across every stage a request passes through. A system at 2× oversubscription rarely shows a doubled median and a doubled 99th percentile — the tail moves much more. ## Recognizing it from outside Signals, roughly in order of usefulness: 1. **Run-queue length** (runnable threads waiting) persistently above core count. This is the direct measurement; load-average style metrics are a rough proxy and on some systems include threads blocked on disk, so read them carefully. 2. **Involuntary context switches rising** while voluntary ones do not. Involuntary means the scheduler took the CPU from a thread that still had work — the signature of CPU competition. Voluntary switches mean blocking, which is a different story. 3. **CPU utilization near saturation while completions per second fall or flatten.** Busy but not productive. 4. **Tail latency degrading faster than the median.** 5. **Rising time-in-lock and queue depths inside the application**, from convoying. A common confusion to call out: high CPU utilization alone does not mean oversubscription — a perfectly-sized CPU-bound service also shows near-100%. The distinguishing evidence is the run queue and the involuntary-switch trend alongside a flat or falling completion rate. ## What you do about it - **Bound concurrency explicitly** — admission control, bounded queues, a cap on concurrently-executing units — so surplus work waits in a queue you control rather than in the OS run queue where it damages everyone. - **Separate the pools.** CPU-bound work sized near core count; blocking work in its own pool where a high thread count is legitimate because those threads are mostly not runnable. - **Reduce work per request** before adding capacity; oversubscription is often a symptom of each request costing more than it should. - **Beware the container mismatch.** A runtime that sizes pools from the machine's core count while the container is limited to a fraction of that is a classic self-inflicted oversubscription: many runnable threads against a small effective quota, plus throttling stalls at every enforcement period. - **Reduce lock hold times**, since preemption inside a critical section is what turns oversubscription into contention collapse. The headline: past one runnable thread per core, extra concurrency is a queue, and a queue in the scheduler is the most expensive place to keep one.