How can you make chunk size dynamic instead of fixed, and what role does CompletionPolicy play?

seniorshould knowfreq 40%

answer

chunk(n) == SimpleCompletionPolicy(n)
CompletionPolicy.isComplete after each read
TimeoutTerminationPolicy = time-based
CompositeCompletionPolicy = OR of policies
custom = size by bytes/cost, extend CountingCompletionPolicy

basics

~20 s

Pass a CompletionPolicy to chunk() instead of a fixed number. The policy decides after each read whether the chunk is complete — for example by item count, elapsed time, or custom logic — instead of always committing at a fixed N.

solid answer

~40 s

chunk(int) is shorthand for chunk(new SimpleCompletionPolicy(n)) — the policy that completes a chunk after n reads. The chunk(CompletionPolicy) overload lets you swap in different completion logic. The framework calls the policy after each read to ask 'is this chunk done?'. SimpleCompletionPolicy counts items; TimeoutTerminationPolicy completes a chunk after a wall-clock duration; CompositeCompletionPolicy combines several (e.g. commit at 1000 items OR after 2 seconds, whichever first). You can implement CompletionPolicy yourself to size chunks by, say, accumulated payload bytes. This is useful when item processing cost is highly variable and a fixed count gives you either huge or tiny transactions. The policy controls chunk boundaries; each completed chunk still commits in exactly one transaction.

chunk(int) is shorthand for chunk(new SimpleCompletionPolicy(n)) — the policy that completes a chunk after n reads. The chunk(CompletionPolicy) overload lets you swap in different completion logic. The framework calls the policy after each read to ask 'is this chunk done?'. SimpleCompletionPolicy counts items; TimeoutTerminationPolicy completes a chunk after a wall-clock duration; CompositeCompletionPolicy combines several (e.g. commit at 1000 items OR after 2 seconds, whichever first). You can implement CompletionPolicy yourself to size chunks by, say, accumulated payload bytes. This is useful when item processing cost is highly variable and a fixed count gives you either huge or tiny transactions. The policy controls chunk *boundaries*; each completed chunk still commits in exactly one transaction.

code

java · 17 lines

java

@Bean
public Step step(JobRepository jobRepository,
                 PlatformTransactionManager txManager,
                 ItemReader<Doc> reader, ItemWriter<Doc> writer) {
    // Commit every 1000 items OR every 2 seconds, whichever comes first
    CompositeCompletionPolicy policy = new CompositeCompletionPolicy();
    policy.setPolicies(new CompletionPolicy[] {
        new SimpleCompletionPolicy(1000),
        new TimeoutTerminationPolicy(2000) // milliseconds
    });

    return new StepBuilder("step", jobRepository)
            .<Doc, Doc>chunk(policy, txManager) // dynamic chunk boundaries
            .reader(reader)
            .writer(writer)
            .build();
}

go deeper

for a junior

Likely only knows the fixed chunk(n) form.

for a middle

Knows chunk(n) is count-based and that other options exist.

for a senior

Explains CompletionPolicy strategy, names Simple/Timeout/Composite, and knows the OR semantics and one-tx-per-chunk invariant.

for a principal

Designs custom policies for variable-cost items and reasons about memory/latency tradeoffs and restart implications.

Default: fixed size. chunk(n, transactionManager) is sugar for supplying a SimpleCompletionPolicy initialized with n. That policy simply completes the chunk after n items have been read. This is right for most jobs where each item is roughly uniform in cost.

The abstraction: CompletionPolicy. It is the strategy interface (from Spring Batch's repeat support, org.springframework.batch.repeat.CompletionPolicy) that answers, after each item, 'should the chunk stop now?' The chunk loop consults it via a RepeatContext: start(context) at chunk begin, update(context) after each read, and isComplete(context, result) / isComplete(context) to decide termination. When it returns complete, the accumulated items are processed and written, and the transaction commits.

Built-in policies.

SimpleCompletionPolicy(int) — the count-based default. chunk(50) == chunk(new SimpleCompletionPolicy(50)).
TimeoutTerminationPolicy(long millis) — completes the chunk once a wall-clock duration since chunk start is exceeded. Good when you want bounded latency regardless of throughput.
CompositeCompletionPolicy — holds an array of policies and completes when any of them signals complete (OR semantics). The classic use is 'commit every 1000 items OR every 2 seconds' so a slow trickle of items still commits periodically and a fast burst still caps transaction size.
CountingCompletionPolicy — an abstract base for count-driven custom policies (you supply how much each item 'counts').

Custom sizing. Implement CompletionPolicy (often by extending CountingCompletionPolicy) to size chunks by something other than raw item count — e.g. total serialized bytes, estimated DB cost, or a field on the item. Example: 'complete the chunk once the cumulative payload exceeds 5 MB' to keep each bulk write bounded in memory when item sizes vary wildly.

Wiring it up:

java

CompositeCompletionPolicy policy = new CompositeCompletionPolicy();
policy.setPolicies(new CompletionPolicy[]{
    new SimpleCompletionPolicy(1000),
    new TimeoutTerminationPolicy(2000) // ms
});
return new StepBuilder("step", jobRepository)
    .<In, Out>chunk(policy, transactionManager)
    .reader(reader).processor(processor).writer(writer).build();

Gotchas.

A CompletionPolicy sizes chunks; it does not change the one-transaction-per-chunk rule. Whatever boundary it chooses, that chunk is still one transaction.
Time-based policies still respect end-of-data. A read() returning null always ends the chunk regardless of policy.
CompositeCompletionPolicy is OR, not AND — it completes on the first policy that fires. People often assume 'all must agree'; it's the opposite.
State is per-RepeatContext. Policies must be safe under the step's execution model; don't hold cross-chunk mutable state in a way that breaks restart.
Overkill risk. For uniform items a fixed chunk(n) is simpler and easier to reason about for memory/restart. Reach for policies only when item cost/latency is genuinely variable.

When to use. Variable-cost items (documents of wildly different sizes), latency SLAs on a slow stream (commit at least every few seconds), or memory-bounded bulk writes. Otherwise stick with the fixed commit-interval.

**Default: fixed size.** `chunk(n, transactionManager)` is sugar for supplying a `SimpleCompletionPolicy` initialized with `n`. That policy simply completes the chunk after `n` items have been read. This is right for most jobs where each item is roughly uniform in cost. **The abstraction: `CompletionPolicy`.** It is the strategy interface (from Spring Batch's repeat support, `org.springframework.batch.repeat.CompletionPolicy`) that answers, after each item, 'should the chunk stop now?' The chunk loop consults it via a `RepeatContext`: `start(context)` at chunk begin, `update(context)` after each read, and `isComplete(context, result)` / `isComplete(context)` to decide termination. When it returns complete, the accumulated items are processed and written, and the transaction commits. **Built-in policies.** - **`SimpleCompletionPolicy(int)`** — the count-based default. `chunk(50)` == `chunk(new SimpleCompletionPolicy(50))`. - **`TimeoutTerminationPolicy(long millis)`** — completes the chunk once a wall-clock duration since chunk start is exceeded. Good when you want bounded latency regardless of throughput. - **`CompositeCompletionPolicy`** — holds an array of policies and completes when **any** of them signals complete (OR semantics). The classic use is 'commit every 1000 items OR every 2 seconds' so a slow trickle of items still commits periodically and a fast burst still caps transaction size. - **`CountingCompletionPolicy`** — an abstract base for count-driven custom policies (you supply how much each item 'counts'). **Custom sizing.** Implement `CompletionPolicy` (often by extending `CountingCompletionPolicy`) to size chunks by something other than raw item count — e.g. total serialized bytes, estimated DB cost, or a field on the item. Example: 'complete the chunk once the cumulative payload exceeds 5 MB' to keep each bulk write bounded in memory when item sizes vary wildly. **Wiring it up:** ```java CompositeCompletionPolicy policy = new CompositeCompletionPolicy(); policy.setPolicies(new CompletionPolicy[]{ new SimpleCompletionPolicy(1000), new TimeoutTerminationPolicy(2000) // ms }); return new StepBuilder("step", jobRepository) .<In, Out>chunk(policy, transactionManager) .reader(reader).processor(processor).writer(writer).build(); ``` **Gotchas.** - **A CompletionPolicy sizes chunks; it does not change the one-transaction-per-chunk rule.** Whatever boundary it chooses, that chunk is still one transaction. - **Time-based policies still respect end-of-data.** A `read()` returning null always ends the chunk regardless of policy. - **CompositeCompletionPolicy is OR, not AND** — it completes on the *first* policy that fires. People often assume 'all must agree'; it's the opposite. - **State is per-RepeatContext.** Policies must be safe under the step's execution model; don't hold cross-chunk mutable state in a way that breaks restart. - **Overkill risk.** For uniform items a fixed `chunk(n)` is simpler and easier to reason about for memory/restart. Reach for policies only when item cost/latency is genuinely variable. **When to use.** Variable-cost items (documents of wildly different sizes), latency SLAs on a slow stream (commit at least every few seconds), or memory-bounded bulk writes. Otherwise stick with the fixed commit-interval.

Does CompositeCompletionPolicy complete the chunk when all its policies agree, or when any one does?
When any one signals complete (OR semantics). So 'SimpleCompletionPolicy(1000) + TimeoutTerminationPolicy(2000ms)' commits at whichever threshold is hit first.
How would you size chunks by accumulated payload bytes rather than item count?
Implement CompletionPolicy (often extending CountingCompletionPolicy) so each item contributes its byte size; mark the chunk complete once cumulative bytes exceed a limit. Pass that policy to chunk().

saying these in an interview costs you the question

Thinking chunk(n) and CompletionPolicy are mutually exclusive rather than n being a policy shortcut
Believing CompositeCompletionPolicy uses AND semantics
Assuming a time-based policy changes the one-transaction-per-chunk rule
Reaching for custom policies when items are uniform and a fixed size suffices

How can you make chunk size dynamic instead of fixed, and what role does CompletionPolicy play?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~2 min

▾common follow-ups2 questions

▾red flags4 flags

§ context

▾related questions3 questions

▾related topics3 topics

answer

go deeper

context