As a principal engineer, how would you decide when to use noRollback and how to size chunks, given the rollback-and-rescan cost model of fault-tolerant Spring Batch steps?

## The cost model to internalize A fault-tolerant chunk step has two regimes: - **Happy path (bulk)**: read N, process N, one bulk write, one commit. Throughput scales with N. - **Failure path (rollback + scan)**: on any chunk exception it rolls back all N, then replays the buffered items **one at a time** (N single-item transactions) to isolate and skip/retry the bad one. So the marginal cost of a failure is roughly *proportional to chunk size*. Large chunks amortize commit overhead beautifully when data is clean, but each poison record triggers an N-way rescan. This is the tension every knob below manages. ## Chunk sizing decision - **Clean, trusted data / rare failures** → larger chunks (hundreds+) for throughput; the occasional rescan is negligible. - **Dirty inbound data / high skip rate** → smaller chunks so each rollback discards and re-scans fewer items, and commit checkpoints are more frequent (better restartability). - **Expensive writes (network round-trips)** → bias larger, but only if idempotent, since rescans re-write. - Always pair with realistic load testing that includes the failure path, not just the happy path. ## When noRollback earns its place Use `.noRollback(Type)` when the exception: 1. is thrown in the **processor/read/listener phase** (not the writer), 2. has **no persisted transactional state** to undo, and 3. represents a routine, expected outcome (business/validation rejection) you'll **skip**. In that case rolling back and re-scanning the whole chunk is pure waste; noRollback lets the good items commit in bulk while the rejected item is skipped. Do **not** register writer exceptions — Batch overrides you and rolls back, because the writer may have partially mutated transactional state. Registering it anyway is a correctness trap that gives false confidence. ## Idempotency is a precondition, not a feature Because rollback triggers replay, the **writer must be idempotent** (upserts, dedup keys, idempotency tokens) and the **processor must be pure**. Non-idempotent external effects belong in an **outbox** consumed by a separate idempotent step, or behind an idempotency key the downstream honors. Skip/retry without idempotency is a latent duplicate-data bug. ## Bounding blast radius - `skipLimit` / `retryLimit` cap how much the step tolerates before failing. Set them to distinguish *expected sparse bad records* from *systemic corruption* — you want the job to fail loudly if 40% of records are bad, not silently skip them into a black hole. - Emit metrics on skip counts and rescan frequency; a rising skip rate is an upstream data-quality signal. - Consider `SkipListener` to route skipped items to a dead-letter/quarantine table for later inspection instead of losing them. ## Interplay with retry Retry re-attempts an item (each attempt can roll back) up to `retryLimit`; a still-failing, skippable exception can then be skipped. Retryable transient faults (deadlocks, timeouts) generally *should* roll back (start clean), so noRollback rarely applies to retryable types. Reserve noRollback for terminal, non-transactional, skippable business rejections. ## The principal-level synthesis Treat chunk size, skip/retry limits, noRollback, and writer idempotency as one coupled design: chunk size sets throughput and failure cost, idempotency makes rescans safe, noRollback trims wasted rollbacks for expected rejections, and limits + listeners bound and observe the blast radius. Get those aligned and a batch job is both fast on the happy path and safe under partial failure.