How would you design the executor strategy for a CompletableFuture-based service that mixes CPU-bound transforms and blocking I/O across many stages?
answer
- Classify each stage: CPU vs blocking
- CPU → non-async / common pool
- Blocking → dedicated bounded pool per downstream
- Bulkheads + backpressure + rejection policy
- Virtual threads for I/O; propagate context explicitly
basics
~20 sSeparate the work: keep cheap CPU transforms on non-async or the common pool, and give blocking I/O its own dedicated, bounded executor (or virtual threads). Use one pool per downstream so a slow dependency can't starve the rest.
solid answer
~50 sI classify every stage as CPU-bound or blocking. Cheap CPU transforms stay non-async (run on the completing thread) or on the common pool, whose core-matched sizing fits CPU work. Each blocking dependency — DB, HTTP, cache — gets its own dedicated, bounded ThreadPoolExecutor passed explicitly to the matching *Async overload; this bulkheads failures so one slow downstream can't exhaust threads the others need. Pools are sized to the downstream's concurrency limit (e.g. the connection-pool size), with a bounded queue and an explicit rejection policy for backpressure. On JDK 21+ I prefer a virtual-thread-per-task executor for blocking stages, since blocked virtual threads park cheaply. I make context propagation explicit (no reliance on ThreadLocal across *Async boundaries), name threads for observability, and never run blocking work on the common pool. I document the threading contract and add a lint/review rule to enforce 'blocking ⇒ explicit executor.'
go deeper
Can pass a custom executor to *Async, but typically uses one pool for everything without sizing or isolation reasoning.
Separates CPU from blocking work and supplies a dedicated executor for I/O, knowing the common pool is for CPU-bound tasks.
Sizes pools to downstream limits, adds bounded queues with rejection policies, and keeps blocking off the common pool; handles context propagation.
Architects per-downstream bulkheads, backpressure strategy, virtual-thread adoption, observability, explicit context propagation, and enforceable guardrails ('no blocking on the common pool') across the service.
## Step 1 — Classify every stage The whole design hinges on one distinction per stage: - **CPU-bound** (parsing, mapping, arithmetic, (de)serialization): runs flat-out on a core; optimal thread count ≈ number of cores. - **Blocking / I/O-bound** (JDBC, HTTP, file, cache, locks, latches): the thread mostly **waits**; you can profitably run many more of them than there are cores, because they are idle while waiting. Mixing them on one pool is the root cause of most CompletableFuture pathologies. ## Step 2 — Route CPU work to a core-sized pool For cheap CPU transforms, either: - use the **non-async** operator (it runs on the completing thread, no extra scheduling), or - use the no-arg `*Async` → **common pool**, whose default parallelism (cores − 1) is correct for CPU work. Reserve the common pool **exclusively** for short, non-blocking work. Never put I/O there — it is shared JVM-wide (parallel streams, libraries) and tiny, so blocking it starves unrelated subsystems. ## Step 3 — Give each blocking dependency its own bounded executor (bulkheads) For blocking stages, pass an **explicit executor** to `*Async(fn, executor)`. Use **one pool per downstream** (a DB pool, an HTTP pool, a cache pool): - **Bulkhead isolation:** if the database slows down, only the DB pool's threads fill up; the HTTP path keeps working. A single shared blocking pool would let one sick dependency take down everything. - **Sizing:** match the pool to the downstream's real concurrency ceiling — e.g. the JDBC connection-pool size. More threads than connections just queue at the connection pool. - **Bounded queue + rejection policy:** a bounded `ArrayBlockingQueue` plus a deliberate policy (`CallerRunsPolicy` for backpressure, or fail-fast) turns overload into **backpressure** instead of an OutOfMemoryError from an unbounded queue. - **Naming + metrics:** give threads meaningful names and export queue depth / active count so saturation is observable. ## Step 4 — Prefer virtual threads for blocking (JDK 21+) A `newVirtualThreadPerTaskExecutor()` is often the cleanest target for blocking stages: a blocked virtual thread **unmounts** its carrier, so thousands of concurrent blocking calls don't need thousands of OS threads and don't starve a small carrier pool. You still apply backpressure upstream (e.g. a semaphore) because virtual threads remove the thread limit but not the downstream's capacity limit. ## Step 5 — Make context propagation explicit Because each `*Async` stage may run on a **different** thread, **ThreadLocal-based context** (security principal, trace/MDC, tenant) does **not** automatically flow across boundaries. Options: - thread the context through stage return values / method parameters, - use a context-propagation library that wraps the executor to copy context on task submission, - or capture context explicitly and restore it at the start of each stage. ## Step 6 — Guardrails and contract - Establish the rule **'blocking work ⇒ explicit dedicated executor; common pool is CPU-only'** and enforce it in code review or with a static-analysis lint. - Document the **threading contract** (which pool each stage family uses, sizes, rejection behavior) so the next engineer doesn't reintroduce starvation. - Watch out for **chains of many tiny `*Async` stages** — the per-stage hand-off overhead can dominate; collapse them to non-async where ordering on the same thread is fine. ## The one-sentence policy Classify by CPU-vs-blocking, keep CPU work on the common pool / non-async, isolate each blocking downstream on its own bounded executor (or virtual threads) with backpressure, propagate context explicitly, and enforce 'no blocking on the common pool.'
- Why one executor per downstream rather than a single shared blocking pool?Bulkhead isolation: a slow or failing downstream only saturates its own pool, leaving threads available for the other dependencies, so one sick service can't cause a total outage.
- Why does a bounded queue with a rejection policy matter?It converts overload into backpressure (or fast failure) instead of an unbounded queue silently buffering work until the JVM runs out of memory.
saying these in an interview costs you the question
- One giant shared pool for both CPU and blocking work — couples failures and invites starvation.
- Unbounded executors/queues as the 'fix' — trades starvation for memory exhaustion.
- Assuming MDC/trace/security ThreadLocals flow automatically across *Async stages — they don't without explicit propagation.
- Putting any blocking I/O on the ForkJoinPool common pool.
- Sizing a blocking pool far beyond the downstream's real concurrency limit, expecting more throughput.