skip to content

An I/O-heavy service uses CompletableFuture.supplyAsync without an explicit executor and sees throughput cap far below expectations. Diagnose how common-pool sizing limits parallelism and what you'd change.

level: principalimportance: should knowfreq 38%

answer

  1. No executor → commonPool, parallelism = cores − 1
  2. Blocking concurrency capped at pool size
  3. Little's Law: concurrency = throughput × latency
  4. I/O wants threads ≫ cores; CPU wants ≈ cores
  5. Fix: non-blocking I/O → virtual threads → dedicated bounded pool

basics

~20 s

The default common pool has only about (cores − 1) threads, so at most that many blocking I/O calls run at once — adding more tasks just queues them. For I/O-bound work you want many more threads than cores, so supply a dedicated, larger (bounded) executor, or use virtual threads.

solid answer

~50 s

Without an explicit executor, supplyAsync uses ForkJoinPool.commonPool(), whose parallelism defaults to availableProcessors() − 1. For CPU-bound work that's appropriate, but for I/O-bound work it's a hard cap: each blocking call holds a thread for its full latency, so the maximum concurrent I/O equals the tiny pool size. Submitting thousands of requests doesn't help — they queue behind ~(cores−1) active threads, and the shared pool is also competing with parallel streams. The classic symptom is throughput plateauing at a small multiple of cores regardless of load, with most time spent waiting on I/O. Fixes, in order of preference: (1) use truly non-blocking I/O so no thread is held; (2) for Java 21+, run blocking work on Executors.newVirtualThreadPerTaskExecutor() — blocked virtual threads unmount, so concurrency scales to the I/O, not the carriers; (3) otherwise supply a dedicated, bounded fixed/cached pool sized for the I/O latency/throughput target (roughly threads ≈ targetConcurrency, bounded for backpressure). Never tune the shared common pool globally as the primary fix.

code

java · 14 lines
java
// PITFALL: I/O-bound work on the small shared common pool caps concurrency at ~cores-1
List<CompletableFuture<Resp>> calls = urls.stream()
    .map(u -> CompletableFuture.supplyAsync(() -> blockingGet(u))) // common pool!
    .toList();

// FIX (Java 21+): virtual threads — blocked tasks unmount, concurrency scales to the I/O
ExecutorService vts = Executors.newVirtualThreadPerTaskExecutor();
List<CompletableFuture<Resp>> scaled = urls.stream()
    .map(u -> CompletableFuture.supplyAsync(() -> blockingGet(u), vts))
    .toList();
CompletableFuture.allOf(scaled.toArray(CompletableFuture[]::new)).join();

// FIX (pre-21): dedicated, bounded I/O pool sized for the wait/service ratio
ExecutorService ioPool = Executors.newFixedThreadPool(64);

go deeper

for a junior

Can say the default pool is small and that blocking limits how many tasks run at once; may not size pools or know virtual threads.

for a middle

Identifies the common pool and its cores−1 size as the cap, and proposes a dedicated larger executor for I/O.

for a senior

Reasons quantitatively (Little's Law, wait/service ratio), prefers non-blocking or virtual threads, sizes/bounds pools for backpressure, and avoids tuning the shared pool.

for a principal

Diagnoses from metrics/thread dumps, sets service-wide executor strategy and isolation, weighs non-blocking vs virtual threads vs bounded pools against operability and downstream load, and bakes backpressure/SLO thinking into the design.

## The scenario A service fans out many calls like `CompletableFuture.supplyAsync(() -> remoteCall())` (no executor argument) and finds that, no matter how much load it receives, throughput saturates at a low, fixed level and CPU sits mostly idle (threads parked in I/O wait). This is a textbook **common-pool sizing** problem. ## Step 1 — Identify the executor No executor argument ⇒ the callbacks run on **`ForkJoinPool.commonPool()`**. Its default **parallelism** is `Runtime.getRuntime().availableProcessors() − 1`. On an 8-vCPU container that's **7 worker threads** — and that pool is **shared JVM-wide** (parallel streams, `Arrays.parallelSort`, other libraries). ## Step 2 — Understand why that caps I/O concurrency For **CPU-bound** work, ~#cores threads is optimal: more threads than cores just context-switch without doing more work. The common pool is tuned for exactly this. For **I/O-bound (blocking)** work, the math is different. A blocking call spends almost all its wall-clock time *waiting*, holding its thread but using ~0% CPU. The number of requests you can have *in flight* equals the number of threads available to block. With ~7 threads, **at most ~7 remote calls run concurrently**; the 8th waits in the queue for a thread to free up. Throughput ≈ (pool size) / (per-call latency). Doubling the offered load doesn't move that ceiling — it just grows the queue and latency. Little's Law makes this concrete: concurrency = throughput × latency, and concurrency is hard-capped by the pool size. (There's a wrinkle: `ForkJoinPool` has a `ManagedBlocker` mechanism and can *temporarily* spawn compensation threads when work signals it's blocking, but ordinary blocking calls don't use it, so in practice you get the fixed parallelism.) ## Step 3 — The fixes, best to worst **(a) Make the I/O non-blocking.** The ideal fix: use an async client that returns a `CompletableFuture` and never holds a thread while waiting (e.g. an async HTTP client, async DB driver, NIO). Then a handful of threads can manage thousands of in-flight calls because none of them blocks. Concurrency is bounded by connections/memory, not threads. **(b) Virtual threads (Java 21+).** Run the blocking work on `Executors.newVirtualThreadPerTaskExecutor()` and pass it to `supplyAsync`. Each task gets its own *virtual* thread; when it blocks on I/O, the JVM **unmounts** it from its carrier platform thread, freeing the carrier for another virtual thread. So you can have, say, 10,000 concurrent blocking calls on a handful of carriers. This makes "one blocking thread per task" cheap and removes the pool-size ceiling for blocking work. **(c) A dedicated, bounded platform-thread pool.** If non-blocking/virtual aren't options, supply your own executor sized for the workload: for I/O you want *more* threads than cores. A common heuristic: `threads ≈ cores × (1 + waitTime/serviceTime)`, i.e. the more a task waits, the more threads pay off. Bound it (e.g. a fixed pool or a cap on a cached pool) so it can't spawn unboundedly under load — the bound provides **backpressure** and protects downstream systems. Pass it explicitly: `supplyAsync(task, ioPool)` and use `*Async(fn, ioPool)` for blocking stages. **What NOT to do as the primary fix:** raising `-Djava.util.concurrent.ForkJoinPool.common.parallelism` to inflate the *shared* common pool. It's global, blunt, affects parallel streams and other libraries, and still mixes blocking and CPU work on one pool. Dedicated executors (or virtual threads) isolate concerns and are tunable per workload. ## Step 4 — Verify Confirm the diagnosis with a thread dump (many common-pool workers parked in socket reads), GC/CPU metrics (low CPU, high latency), and by observing throughput rise after moving to a properly sized executor or virtual threads. Watch the new pool's queue depth and rejected-task count for backpressure tuning. ## Mental model The common pool is a *small, shared, CPU-tuned* resource. Blocking I/O on it caps your concurrency at ~#cores. Either stop holding threads (non-blocking or virtual threads) or give blocking work its own appropriately large, bounded pool.

  • What's a reasonable starting heuristic for sizing a pool dedicated to blocking I/O?
    Roughly threads ≈ cores × (1 + waitTime/serviceTime): the larger the fraction of time a task spends waiting, the more threads help. In practice you size to your target in-flight concurrency (per Little's Law: throughput × latency), then bound it for backpressure and load-test to refine.
  • Why are virtual threads a better answer than a 500-thread platform pool for blocking I/O?
    500 platform threads cost ~MBs of stack each plus OS scheduling overhead and context-switching. Virtual threads are cheap user-mode threads that unmount from a small set of carriers when they block, so you get the same high I/O concurrency at a fraction of the memory and scheduling cost — and your simple blocking code still works.

saying these in an interview costs you the question

  • Suggesting you just submit more tasks to raise throughput
  • Inflating the global common-pool parallelism as the main fix
  • Sizing an I/O pool to #cores as if it were CPU-bound
  • Using an unbounded pool that can exhaust resources with no backpressure
  • Ignoring that the common pool is shared with parallel streams

context

resolving links…