How do you choose the splitting threshold (the size at which a Fork/Join task stops splitting and computes directly), and what happens if it's too small or too large?

## What the threshold is A Fork/Join `compute()` is recursive divide-and-conquer: ```java protected Long compute() { if (hi - lo <= THRESHOLD) { // base case: stop splitting return computeDirectly(); // sequential work on this slice } int mid = (lo + hi) >>> 1; // recursive case: split ...fork/compute/join... } ``` The **splitting threshold** (a.k.a. sequential cutoff) is the subproblem size at which the `if` fires and you stop recursing — you solve the slice with a plain loop instead of creating two more subtasks. It is the single most important tuning knob for Fork/Join performance. ## Why there is a cost to splitting Each split creates **task objects**, calls `fork()` (a deque push, memory barriers, possible steal by another thread), and later `join()` (await + result plumbing). That coordination is cheap but **not free** — on the order of tens to hundreds of nanoseconds per task. If a leaf task only does a few nanoseconds of real work, the overhead dominates and you've made things *slower* than a sequential loop, sometimes dramatically. ## Too small a threshold - Explosion of tiny tasks → overhead (allocation, deque ops, steals, GC pressure from millions of short-lived task objects) **outweighs** the useful work. - Cache effects and scheduler churn add up. - Symptom: parallel version no faster than — or slower than — sequential; profiler shows time in pool internals/allocation. ## Too large a threshold - Too **few** tasks → fewer parallel pieces than cores, so some CPUs sit idle; speedup is capped well below the number of cores. - **Poor load balancing**: work-stealing can only redistribute *tasks*. With only a handful of large tasks, one slow/uneven task can't be subdivided, so a single worker becomes the bottleneck while others finish early and idle. - Symptom: low CPU utilization, speedup plateaus far below core count. ## How to choose 1. **Aim for many more tasks than cores.** A common target is generating on the order of hundreds to a few thousand leaf tasks, so the pool has enough granular work to balance via stealing. (Rough guidance: tasks ≈ cores × 8 or more, but it depends.) 2. **Keep each leaf's work >> per-task overhead.** The JDK `ForkJoinTask` documentation suggests a leaf should perform roughly **100 to 10,000 basic computational steps** and avoid indefinite looping — small enough for balancing, large enough to amortize overhead. 3. **Account for per-element cost.** If each element is expensive (e.g. a heavy computation per item), a *smaller* element-count threshold is fine because each task still does plenty of work. If each element is cheap (a single add), you need a *larger* count per leaf to be worth splitting. 4. **Measure, don't guess.** Benchmark (e.g. with JMH) across several threshold values on representative data and the target hardware; plot time vs. threshold and pick the flat "knee." The optimum shifts with core count, data size, and per-element cost. 5. **Consider data-dependent splitting.** For uneven workloads, splitting by *estimated cost* rather than raw count, or splitting until the slice is small, improves balance. `Spliterator` (used by parallel streams) encodes this via `trySplit` and characteristics like `SIZED`. ## Relationship to parallel streams Parallel streams sit on top of Fork/Join and choose splitting via `Spliterator`. If you reach for raw Fork/Join, you're taking manual control of exactly this threshold; often a parallel stream with a well-sized source gives comparable results with less code — so part of "choosing the threshold" is deciding whether to hand-roll Fork/Join at all. ## Summary - Threshold = the base-case size where you stop forking and compute sequentially. - Too small → overhead dominates (slower than sequential). - Too large → too few tasks → idle cores + poor load balancing. - Target: many more tasks than cores, each doing ~100–10,000 ops; then benchmark and tune to the hardware.