skip to content

What are the roles of ChunkMessageChannelItemWriter and ChunkProcessorChunkHandler in a remote chunking setup?

level: middleimportance: must knowfreq 35%

answer

  1. Writer = master end, sends ChunkRequest, counts ChunkResponse
  2. Handler = worker end, service activator, runs process+write
  3. Handler wraps SimpleChunkProcessor
  4. Builder factories wire both automatically
  5. ChunkRequest out / ChunkResponse back

basics

~20 s

ChunkMessageChannelItemWriter is the master's ItemWriter: instead of writing, it sends chunks to the request channel and collects worker replies. ChunkProcessorChunkHandler is the worker's handler: it receives a chunk, runs the processor+writer, and sends back a response.

solid answer

~40 s

These are the two ends of the wire. On the **master**, `ChunkMessageChannelItemWriter` is plugged in as the step's `ItemWriter`. Rather than persisting items, it wraps each chunk in a `ChunkRequest` and sends it to the outbound message channel, then reads `ChunkResponse` messages from the reply channel and keeps count of expected vs received responses so the step knows when all work is acknowledged and whether any chunk failed. On each **worker**, `ChunkProcessorChunkHandler` is registered as a Spring Integration **service activator** on the request channel. It holds a `SimpleChunkProcessor` (your `ItemProcessor` + `ItemWriter`); on receiving a `ChunkRequest` it runs process then write on those items and returns a `ChunkResponse` carrying success/failure and the sequence info. You rarely instantiate these directly — `RemoteChunkingManagerStepBuilderFactory` wires the writer, and `RemoteChunkingWorkerBuilder` wires the handler.

code

java · 23 lines
java
// WORKER application config
@Configuration
@EnableBatchIntegration
public class WorkerConfig {

    @Autowired
    private RemoteChunkingWorkerBuilder<Order, Order> workerBuilder;

    // Builds an IntegrationFlow that installs a ChunkProcessorChunkHandler
    // as a service activator: consume ChunkRequest -> process+write -> ChunkResponse.
    @Bean
    public IntegrationFlow workerFlow(MessageChannel requests,       // inbound from broker
                                      MessageChannel replies,        // outbound to broker
                                      ItemProcessor<Order, Order> processor,
                                      ItemWriter<Order> writer) {
        return this.workerBuilder
                .itemProcessor(processor)   // wrapped in a SimpleChunkProcessor
                .itemWriter(writer)
                .inputChannel(requests)     // ChunkRequest arrives here
                .outputChannel(replies)     // ChunkResponse sent here
                .build();
    }
}

go deeper

for a junior

Should know one component is on the master and one on the worker.

for a middle

Should name each component's job: writer sends+aggregates, handler processes+writes and replies.

for a senior

Should explain response counting/correlation and how failures propagate via ChunkResponse.

for a principal

Should discuss serialization, correlation guarantees, and why the builders exist over manual wiring.

Remote chunking's plumbing has two Spring Batch Integration components that sit at opposite ends of the messaging channel. Understanding them demystifies the whole pattern. **`ChunkMessageChannelItemWriter<T>` (master side):** - It *is* the master step's `ItemWriter`. A remote-chunking master is just an ordinary chunk-oriented step whose writer, instead of hitting a database/file, ships the chunk elsewhere. - On `write(Chunk<T>)` it builds a `ChunkRequest` (containing the items, the `StepContribution`, a sequence/job id) and sends it to the configured **output/request `MessageChannel`**. - It also drains the **input/reply `PollableChannel`**, reading `ChunkResponse` messages the workers send back. It maintains counters (`localState`) of how many chunks were sent versus acknowledged. - At step end / on each cycle it reconciles responses: it inspects each `ChunkResponse` for success, applies the worker's `StepContribution` to the master's `StepExecution` (so write counts are accurate), and if a response signals failure it surfaces that as a step failure. Because responses are asynchronous, the master may still be collecting acks after the reader is exhausted — the writer blocks/polls until all outstanding responses arrive. - Configured automatically when you build the step via `RemoteChunkingManagerStepBuilderFactory#get(...).outputChannel(...).inputChannel(...)`. **`ChunkProcessorChunkHandler<T>` (worker side):** - Implements `ChunkHandler`. It is exposed as a Spring Integration **service activator** listening on the request channel and replying on the reply channel. - It wraps a `ChunkProcessor` — in practice a `SimpleChunkProcessor<I,O>` built from your `ItemProcessor` and `ItemWriter`. - On each inbound `ChunkRequest` it calls `handleChunk(...)`: runs the processor over the items, then the writer over the results, inside the worker's own transaction, and constructs a `ChunkResponse` (success flag, the resulting `StepContribution`, the chunk sequence number, and any exception). That response goes back to the master. - It deliberately does **not** re-run the reader — the items already arrived in the message. - Configured automatically by `RemoteChunkingWorkerBuilder#itemProcessor(...).itemWriter(...).inputChannel(...).outputChannel(...).build()`, which returns an `IntegrationFlow`. **How they cooperate:** 1. Master reader fills a chunk → `ChunkMessageChannelItemWriter` sends `ChunkRequest`. 2. Broker delivers it to one worker. 3. Worker's `ChunkProcessorChunkHandler` processes+writes → sends `ChunkResponse`. 4. Master's writer consumes the response, updates counts, tracks outstanding chunks. **Gotchas:** - The master must be able to correlate responses to requests; the components use sequence counters, so don't hand-roll channels that reorder/duplicate silently without durable semantics. - Both `ChunkRequest`/`ChunkResponse` and the items must be **serializable** by the transport. - Errors on the worker are reported back in the `ChunkResponse`; the master doesn't 'see' the worker's stack trace directly — it reconstructs failure from the response.

  • Do you normally instantiate ChunkMessageChannelItemWriter and ChunkProcessorChunkHandler by hand?
    No. RemoteChunkingManagerStepBuilderFactory wires the ItemWriter on the master, and RemoteChunkingWorkerBuilder wires the handler flow on the worker. You supply reader/processor/writer and the channels; the builders create these components.
  • How does the master learn whether a worker's chunk succeeded or failed?
    Via the ChunkResponse the worker sends back. ChunkMessageChannelItemWriter reads those responses from the reply channel, applies each StepContribution, and treats a failure flag/exception in a response as a step failure.

saying these in an interview costs you the question

  • Saying the worker re-runs the ItemReader (it doesn't — items arrive in the message)
  • Thinking the master writes items itself (its writer just forwards chunks)
  • Confusing ChunkProcessorChunkHandler with a partition handler (that's remote partitioning)
  • Believing responses are optional (the master must collect them to know completion)

context

resolving links…