skip to content

What is the role of PartitionHandler and TaskExecutorPartitionHandler in partitioning?

level: seniorimportance: should knowfreq 50%

answer

  1. PartitionHandler = how/where workers run
  2. TaskExecutorPartitionHandler = local thread pool
  3. gridSize hint lives on the handler
  4. SyncTaskExecutor = no parallelism
  5. partitions vs threads independent; watch DB pool

basics

~10 s

The PartitionHandler executes the worker StepExecutions produced from the partitions and collects their results. TaskExecutorPartitionHandler is the local implementation: it runs each worker on a TaskExecutor (thread pool) and waits for all to finish.

solid answer

~40 s

`PartitionHandler` is the abstraction that answers 'where and how do the worker StepExecutions run?'. After the `StepExecutionSplitter` (driven by the `Partitioner`) creates one worker StepExecution per partition, the PartitionHandler is handed the manager StepExecution plus the splitter and is responsible for actually executing the workers and returning their finished StepExecutions to the master for aggregation. `TaskExecutorPartitionHandler` is the built-in **local** implementation: you give it the worker `Step` and a `TaskExecutor`, and it submits each worker as a task, running them concurrently on that thread pool. Its `gridSize` property is the hint passed to the Partitioner. Use a bounded pool (`ThreadPoolTaskExecutor`) so partition concurrency doesn't exhaust DB connections; `SyncTaskExecutor` runs them serially (useful for debugging). For scaling across JVMs you swap in `MessageChannelPartitionHandler` (remote), keeping the same Partitioner/worker step.

code

java · 30 lines
java
@Bean
public TaskExecutor partitionTaskExecutor() {
    ThreadPoolTaskExecutor exec = new ThreadPoolTaskExecutor();
    exec.setCorePoolSize(8);
    exec.setMaxPoolSize(8);      // bounded — must fit the DB connection pool
    exec.setQueueCapacity(50);   // extra partitions queue here
    exec.setThreadNamePrefix("part-");
    exec.initialize();
    return exec;
}

@Bean
public TaskExecutorPartitionHandler partitionHandler(Step workerStep,
                                                     TaskExecutor partitionTaskExecutor) {
    TaskExecutorPartitionHandler handler = new TaskExecutorPartitionHandler();
    handler.setStep(workerStep);                 // the worker step run per partition
    handler.setTaskExecutor(partitionTaskExecutor); // real pool => real concurrency
    handler.setGridSize(16);                     // hint passed to Partitioner
    return handler;
}

@Bean
public Step managerStep(JobRepository jobRepository,
                        Partitioner rangePartitioner,
                        TaskExecutorPartitionHandler partitionHandler) {
    return new StepBuilder("managerStep", jobRepository)
            .partitioner("workerStep", rangePartitioner)
            .partitionHandler(partitionHandler)
            .build();
}

go deeper

for a junior

Know that PartitionHandler runs the workers and TaskExecutorPartitionHandler is the local/threaded one.

for a middle

Explain step + taskExecutor + gridSize wiring and that SyncTaskExecutor means serial.

for a senior

Reason about partitions-vs-threads independence, DB pool sizing, and blocking-until-done semantics.

for a principal

Design the executor + connection-pool budget, decide local vs remote handler, and guard against skew/hung partitions.

## Where PartitionHandler sits Partitioning has a clean separation of concerns: - **`Partitioner`** decides *how to divide* the data (produces named ExecutionContexts). - **`StepExecutionSplitter`** turns those into worker **StepExecutions**. - **`PartitionHandler`** decides *how/where to run* those worker StepExecutions and returns them once done. The `PartitionStep` (the master step) orchestrates: it calls the handler, the handler runs the workers, and the master then **aggregates** their exit statuses via a `StepExecutionAggregator` (default rolls up to FAILED if any worker failed, else COMPLETED). ## The interface ```java public interface PartitionHandler { Collection<StepExecution> handle(StepExecutionSplitter splitter, StepExecution managerStepExecution) throws Exception; } ``` It receives the splitter (which knows the Partitioner and gridSize) and the manager's StepExecution, runs the workers, and returns the completed worker StepExecutions. ## `TaskExecutorPartitionHandler` — local execution The default when you use `StepBuilder.partitioner(...)` without specifying a handler. Key properties: - **`step`** — the worker `Step` to run for every partition. - **`taskExecutor`** — the `TaskExecutor` the workers run on. This determines concurrency: - `SyncTaskExecutor` (default if none set) → workers run **serially** on the calling thread — no real parallelism (fine for testing). - `ThreadPoolTaskExecutor` / `SimpleAsyncTaskExecutor` → workers run **concurrently**. - **`gridSize`** — passed to the Partitioner's `partition(gridSize)` as the hint for how many partitions. It submits one task per worker StepExecution, blocks until all complete, then returns them. ## Concurrency vs resources — the critical tuning point Each concurrent partition typically opens its **own DB connection(s)** for reader and writer. If you have 20 partitions on an unbounded `SimpleAsyncTaskExecutor` but a connection pool of 10, you deadlock or starve. Rules of thumb: - Use a **bounded** `ThreadPoolTaskExecutor`; keep max threads ≤ pool capacity headroom. - The number of *partitions* (map size) can exceed pool threads — extra partitions just queue and run as threads free up. gridSize and pool size are independent knobs. ## Local vs remote — same Partitioner, different handler The elegance of the design: the **Partitioner and worker step stay identical** whether local or remote. Only the PartitionHandler changes: - **Local**: `TaskExecutorPartitionHandler` (threads in one JVM). - **Remote**: `MessageChannelPartitionHandler` — sends each partition's request over a `MessagingTemplate`/channel to worker nodes, which run the worker step and report back. (The messaging transport itself is the domain of remote partitioning/chunking, out of scope here — the point is the handler is the swap point.) ## Gotchas - Forgetting to set a real `TaskExecutor` → you 'partitioned' but everything runs serially (`SyncTaskExecutor`), gaining nothing. - Unbounded `SimpleAsyncTaskExecutor` spawns a thread per partition → resource blowup with many partitions. - The handler **blocks** until all workers finish; a single slow/hung partition stalls the master step. - Worker components must be thread-safe or `@StepScope`; sharing a stateful singleton reader/writer across concurrent partitions corrupts state.

  • If you set gridSize=16 but your ThreadPoolTaskExecutor has maxPoolSize=8, what happens?
    The Partitioner still creates ~16 worker StepExecutions (partitions), but only 8 run at a time; the remaining 8 queue and start as threads free up. gridSize controls partition count; pool size controls concurrency — they're independent.
  • How do you switch this job from local to remote partitioning?
    Keep the same Partitioner and worker step, and replace TaskExecutorPartitionHandler with MessageChannelPartitionHandler wired to a messaging channel/broker. The manager sends partition requests to remote workers instead of running them on a local thread pool.

saying these in an interview costs you the question

  • Thinking partitioning is parallel even with the default SyncTaskExecutor
  • Setting gridSize equal to pool threads believing they must match
  • Sharing a stateful singleton reader/writer across concurrent partitions
  • Ignoring DB connection pool size when raising partition concurrency

context

resolving links…