Compare a multithreaded step with partitioning and remote chunking. When would you choose each?

multithreaded: shared reader, 1 JVM, cheapest
partitioning: own reader per slice, restart-per-partition, scale-out
remote chunking: master reads, workers process/write
shared-reader weakness -> partitioning fixes it
match to bottleneck: read vs process/write

basics

~20 s

Multithreaded step: many threads share one reader in one JVM — simplest, but shared state and weak restart. Partitioning: each worker step gets its own reader over a data slice — clean restart, scales locally or across JVMs. Remote chunking: one master reads, workers process/write remotely — for CPU/write-heavy work.

solid answer

~50 s

All three are Spring Batch scaling patterns. A multithreaded step adds a TaskExecutor so chunks run on multiple threads in one JVM; it's the lowest-config option but shares one stateful reader (needs SynchronizedItemStreamReader), loses ordering, and restarts poorly. Partitioning splits input into partitions via a Partitioner and runs a worker step per partition through a PartitionHandler (local TaskExecutorPartitionHandler or remote via messaging); each partition has its own reader over a disjoint range, so there's no shared state, restart works per partition, and it can scale across JVMs — at the cost of needing a way to divide the data. Remote chunking keeps a single reader on the master and ships read items to remote workers that process and write, so it's for cases where processing/writing (not reading) is the bottleneck and reading can't be partitioned. Choose multithreaded for quick I/O-bound wins on one box; partitioning when data divides cleanly and you need restart/scale-out; remote chunking when the read is inherently sequential but processing is heavy.

code

java · 18 lines

java

// Partitioning: master step fans out to worker steps, each with its own reader
@Bean
public Step masterStep(JobRepository repo, Step workerStep,
                       Partitioner partitioner, TaskExecutor exec) {
    TaskExecutorPartitionHandler handler = new TaskExecutorPartitionHandler();
    handler.setStep(workerStep);
    handler.setTaskExecutor(exec);
    handler.setGridSize(8);
    return new StepBuilder("masterStep", repo)
            .partitioner("workerStep", partitioner) // e.g. splits id ranges
            .partitionHandler(handler)
            .build();
}

// Each worker step (below) uses a @StepScope reader bound to its partition's
// {minId,maxId} from the ExecutionContext -> no shared state, clean restart.
// Contrast: a multithreaded step would be one step with .taskExecutor(exec)
// and a single SynchronizedItemStreamReader shared by every thread.

for a junior

Know multithreaded step = threads in one JVM; partitioning = split the data into parallel workers.

for a middle

Contrast shared reader (multithreaded) vs own reader per partition; know local vs remote partition handlers.

for a senior

Explain restart-per-partition, scale-out, and matching remote chunking to process/write bottlenecks.

for a principal

Drive a decision from data-divisibility, restart requirements, bottleneck location, and operational cost; know when combining strategies is (rarely) justified.

The three built-in scaling strategies

Spring Batch offers several ways to go faster; the three relevant here differ in where the reader lives, what's shared, and whether they scale up (threads) or out (machines).

1. Multithreaded step

Mechanism: step.taskExecutor(pool); the TaskExecutorRepeatTemplate runs chunks concurrently on a thread pool.
Reader: one shared instance across all threads → must wrap stateful readers in SynchronizedItemStreamReader, which serializes reads.
Scope: single JVM, scale-up only.
Pros: trivial to enable; good for I/O-bound process/write.
Cons: shared mutable state; no ordering; weak restart; read-bound steps don't benefit; can overwhelm downstream (DB pool).
Best for: quick throughput win on order-insensitive, idempotent, I/O-bound steps.

2. Partitioning

Mechanism: a Partitioner divides the input into a set of ExecutionContexts (partitions) — e.g. by primary-key range, file, or date. A master step uses a PartitionHandler to run a worker step per partition. TaskExecutorPartitionHandler runs partitions on local threads; a MessageChannelPartitionHandler runs them on remote workers over messaging (e.g. Spring Integration/AMQP).
Reader: each worker step has its own reader scoped to its partition's data slice → no shared state, no synchronization needed.
Scope: scale-up (local threads) or scale-out (remote JVMs).
Pros: clean per-partition restartability (each worker step execution is tracked independently); can distribute across machines; no shared-reader bottleneck.
Cons: you must be able to divide the data into disjoint partitions; more moving parts (partitioner, handler, possibly messaging infra); uneven partitions cause skew.
Best for: large datasets that split naturally (id ranges, files, tenants), when restart correctness or horizontal scale matters.

3. Remote chunking

Mechanism: a single master runs the reader and sends read items (chunks) over a durable channel to remote worker processes that run the processor + writer, reporting results back.
Reader: only on the master (single, sequential read); processing/writing is distributed.
Scope: scale-out for processing/writing.
Pros: works when the read is inherently sequential / can't be partitioned but the CPU-heavy processing or the write is the bottleneck.
Cons: requires reliable messaging middleware; network serialization overhead; the single reader is still a ceiling on read throughput; more operational complexity.
Best for: heavy per-item processing/writing where reading is cheap/sequential and you want to fan work out to a worker fleet.

Decision guide

Need	Pick
Fast win, single box, I/O-bound, order-insensitive	Multithreaded step
Data splits cleanly; need restart correctness or scale-out	Partitioning
Read is sequential but processing/writing is the bottleneck	Remote chunking

Key architectural insight

The multithreaded step's defining weakness — a shared, stateful reader — is exactly what partitioning fixes by giving each worker its own reader over a disjoint slice. That's why, at scale, partitioning is usually the more robust choice; the multithreaded step earns its place only as the cheapest option when its correctness caveats are acceptable. Remote chunking is orthogonal: it addresses processing/write bottlenecks rather than read parallelism. You can even combine techniques (e.g. partitioning across JVMs, multithreading within each), though that compounds complexity and is rarely needed.

## The three built-in scaling strategies Spring Batch offers several ways to go faster; the three relevant here differ in **where the reader lives**, **what's shared**, and **whether they scale up (threads) or out (machines)**. ### 1. Multithreaded step - **Mechanism**: `step.taskExecutor(pool)`; the `TaskExecutorRepeatTemplate` runs chunks concurrently on a thread pool. - **Reader**: **one shared instance** across all threads → must wrap stateful readers in `SynchronizedItemStreamReader`, which serializes reads. - **Scope**: **single JVM**, scale-up only. - **Pros**: trivial to enable; good for I/O-bound process/write. - **Cons**: shared mutable state; no ordering; weak restart; read-bound steps don't benefit; can overwhelm downstream (DB pool). - **Best for**: quick throughput win on order-insensitive, idempotent, I/O-bound steps. ### 2. Partitioning - **Mechanism**: a `Partitioner` divides the input into a set of `ExecutionContext`s (partitions) — e.g. by primary-key range, file, or date. A **master step** uses a `PartitionHandler` to run a **worker step** per partition. `TaskExecutorPartitionHandler` runs partitions on local threads; a `MessageChannelPartitionHandler` runs them on **remote workers** over messaging (e.g. Spring Integration/AMQP). - **Reader**: **each worker step has its own reader** scoped to its partition's data slice → **no shared state**, no synchronization needed. - **Scope**: scale-up (local threads) **or scale-out** (remote JVMs). - **Pros**: clean **per-partition restartability** (each worker step execution is tracked independently); can distribute across machines; no shared-reader bottleneck. - **Cons**: you must be able to **divide the data** into disjoint partitions; more moving parts (partitioner, handler, possibly messaging infra); uneven partitions cause skew. - **Best for**: large datasets that split naturally (id ranges, files, tenants), when restart correctness or horizontal scale matters. ### 3. Remote chunking - **Mechanism**: a **single master** runs the reader and sends read *items* (chunks) over a durable channel to **remote worker** processes that run the processor + writer, reporting results back. - **Reader**: **only on the master** (single, sequential read); processing/writing is distributed. - **Scope**: scale-out for **processing/writing**. - **Pros**: works when the **read is inherently sequential / can't be partitioned** but the CPU-heavy processing or the write is the bottleneck. - **Cons**: requires reliable messaging middleware; network serialization overhead; the single reader is still a ceiling on read throughput; more operational complexity. - **Best for**: heavy per-item processing/writing where reading is cheap/sequential and you want to fan work out to a worker fleet. ## Decision guide | Need | Pick | |---|---| | Fast win, single box, I/O-bound, order-insensitive | Multithreaded step | | Data splits cleanly; need restart correctness or scale-out | Partitioning | | Read is sequential but processing/writing is the bottleneck | Remote chunking | ## Key architectural insight The multithreaded step's defining weakness — a **shared, stateful reader** — is exactly what partitioning fixes by giving each worker its **own** reader over a disjoint slice. That's why, at scale, partitioning is usually the more robust choice; the multithreaded step earns its place only as the **cheapest** option when its correctness caveats are acceptable. Remote chunking is orthogonal: it addresses processing/write bottlenecks rather than read parallelism. You can even combine techniques (e.g. partitioning across JVMs, multithreading within each), though that compounds complexity and is rarely needed.

What single architectural difference makes partitioning restart-safe where a multithreaded step isn't?
Each partition is an independent worker-step execution with its own reader over a disjoint data slice, tracked separately in the repository — so progress is contiguous per partition and restart resumes only the failed partitions, with no shared-state holes.
When is remote chunking the right choice over partitioning?
When the read is inherently sequential or can't be divided into disjoint slices, but the per-item processing or writing is the real bottleneck. The master reads once and fans chunks out to remote workers for processing/writing.
Can you combine partitioning and multithreaded steps?
Yes — you could partition across JVMs and multithread within each worker — but it compounds complexity and the shared-reader caveats reappear inside each worker; usually one strategy suffices.

saying these in an interview costs you the question

Saying partitioning shares one reader (each partition gets its own)
Claiming remote chunking parallelizes reading (the master reader is single/sequential)
Thinking a multithreaded step scales across JVMs
Treating the three as interchangeable rather than matched to the bottleneck

Compare a multithreaded step with partitioning and remote chunking. When would you choose each?

answer

basics

solid answer

code

go deeper

The three built-in scaling strategies

1. Multithreaded step

2. Partitioning

3. Remote chunking

Decision guide

Key architectural insight

context

The three built-in scaling strategies

1. Multithreaded step

2. Partitioning

3. Remote chunking

Decision guide

Key architectural insight

Compare a multithreaded step with partitioning and remote chunking. When would you choose each?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~3 min

The three built-in scaling strategies

1. Multithreaded step

2. Partitioning

3. Remote chunking

Decision guide

Key architectural insight

▾common follow-ups3 questions

▾red flags4 flags

§ context

▾related questions6 questions

▾related topics4 topics

The three built-in scaling strategies

1. Multithreaded step

2. Partitioning

3. Remote chunking

Decision guide

Key architectural insight

answer

go deeper

context