You must apply a price change to roughly ten million rows in a live system that uses Hibernate with a second-level cache and concurrent users. How do you decide between one JPQL bulk UPDATE, chunked bulk statements, and loading entities in batches?
answer
- three axes: expressible, footprint, who holds the rows
- chunk by indexed key, commit per chunk
- idempotent + resumable + throttled
- L2 region drops per chunk = read spike
- versioned for user-held rows; clear the session
basics
~20 sDecide on three axes: is the change expressible in SQL, how long may one transaction hold locks, and who else holds those rows. Usually chunked bulk statements committed per chunk - set-based speed without one huge transaction - plus deliberate cache invalidation and version bumping.
solid answer
~60 sI ask three questions. Is the change expressible in SQL? If the new price is a formula over existing columns, a bulk statement wins by orders of magnitude. If it needs per-row business logic, callbacks or external lookups, I must iterate entities and accept the cost. What transaction footprint is acceptable? One statement over ten million rows is one long write transaction: sustained locks, large redo or WAL, replication lag, a painful rollback. On a live system I chunk by key range - a few thousand to tens of thousands of rows per transaction, resumable, with a throttle - trading atomicity for operability. If the change must be all-or-nothing, that pushes towards a maintenance window instead. Who else holds these rows? Hibernate invalidates the affected second-level cache regions for HQL statements, which is a coarse drop and will cause a read spike; per chunk, that region keeps getting dropped, so I may disable or warm deliberately. For rows users have open I use the versioned form of the statement so their stale copies fail loudly rather than overwrite me. Entity iteration remains the fallback, with batching, periodic flush and clear, and stateless sessions.
code
java · 13 lineslong lastId = 0;
int chunk = 5_000;
while (true) {
int rows = runInOwnTransaction(em -> em.createQuery(
"update versioned Product p set p.price = p.price * 1.1 "
+ "where p.id > :last and p.id <= :upper")
.setParameter("last", lastId)
.setParameter("upper", lastId + chunk)
.executeUpdate());
lastId += chunk;
if (lastId >= maxId) break;
pauseBriefly();
}go deeper
Recognise that bulk statements are far faster than loading entities, and that huge transactions are risky.
Describe chunking with an indexed key, committing per chunk, and clearing the persistence context between chunks.
Add operability: idempotence, resumability, throttling, metrics, cache invalidation cost and version bumping for user-held rows.
Lead with the trade space - expressiveness, transaction footprint, staleness contract - name what is given up (atomicity), and define the rollout, kill switch and stakeholder agreement.
## Framing the decision There is no default answer; there are three axes and the workload picks a point. Expressiveness. A bulk statement can do anything SQL can do to columns. If the price change is price * 1.1, or a join against a rules table, SQL wins: one statement, no data over the wire, no objects on the heap. The moment the rule needs Java - a pricing service call, conditional logic across aggregates, encryption in application code, audit callbacks - the object layer is the only place that logic exists, and iteration is forced. Transaction footprint. Ten million rows in one statement means one transaction holding row locks for its whole duration, undo or WAL proportional to the change, a rollback as expensive as the work, and, on a replicated system, lag while followers apply it. It also blocks nothing until it does: writers touching the same rows queue behind it. Chunking - process by primary key range or by a natural partition, commit each chunk, sleep briefly between chunks - converts one long transaction into many short ones. You lose atomicity, so the job must be idempotent and resumable: a progress marker, a predicate that excludes already-processed rows, and safe re-runs. Concurrency and staleness. Two caches matter. The persistence context of any session running the job must be cleared after each chunk, or the job accumulates stale managed entities. The second-level cache is invalidated per HQL bulk statement for the affected regions - correct, but with a chunked job that means repeated region drops and a read spike against the database for every chunk. Options: accept it in a quiet window; disable the region for the duration; or drive the job through a channel that lets you invalidate once at the end. For rows that interactive users may be holding, the versioned form of the update bumps the version so their next write fails visibly instead of silently restoring old prices. ## The three candidate designs One bulk statement. Simplest, fastest in raw throughput, atomic. Right for maintenance windows, smaller tables, and systems where a multi-minute write transaction is acceptable. Wrong for a busy OLTP system at this row count. Chunked bulk statements. The usual production answer: near set-based speed, bounded lock time, throttleable, resumable, observable (rows per second, chunks remaining). Costs: not atomic, so the system must tolerate a partially applied change - which for a price update usually means agreeing with the business that the rollout is gradual. Chunk by an indexed key so each statement's WHERE clause is a range scan, not a full scan repeated per chunk. Entity iteration in batches. Needed when Java logic is unavoidable. Use a scrollable or keyset-paginated read, apply the change, flush and clear every N entities, and prefer a stateless session where the object-layer features are not needed, since it keeps no persistence context and no cache. Expect one or two orders of magnitude less throughput and plan the runtime accordingly. Batched JDBC writes help, but the ceiling is far below set-based SQL. ## What I would actually propose For a formula-based price change on ten million rows in a live system: chunked bulk UPDATE, keyed on the primary key, chunk size tuned so each transaction lasts well under a second, the versioned form if the catalogue is user-editable, a small pause between chunks, and metrics on rows processed plus database wait events. Ship it as a resumable job with a kill switch. Before it runs, agree with the owners of the second-level cache on invalidation, and warn any reporting consumers that the change is not instantaneous. And the standing caveat for the whole family: whichever bulk shape you choose, the persistence context is never reconciled. Any long-lived session in the job clears after each chunk, and application code that had entities loaded before the run must re-read them. ## Signals that you chose wrong Lock waits and timeouts on user transactions mean the chunks are too big or unthrottled. A database-wide read spike after each chunk means cache invalidation is dominating - reconsider region strategy. A job that cannot be restarted after a failure means the chunking is not idempotent. And a bulk statement that silently disagrees with what users see means version bumping or cache invalidation was skipped.
- What do you give up by chunking, and how do you make that acceptable?Atomicity: for a while some rows carry the new price and some the old. You make it acceptable by agreeing the rollout is gradual with the business, by making each chunk idempotent so re-runs are safe, and by recording progress so a failed job resumes rather than restarts. If all-or-nothing is a hard requirement, the answer is a maintenance window, not chunking.
- Why can a chunked bulk job be harder on the second-level cache than a single statement?Each HQL bulk statement invalidates the affected cache regions, so a hundred chunks means a hundred coarse region drops and a hundred read spikes as traffic re-populates the cache. A single statement invalidates once. Mitigations are running in a quiet period, disabling the region for the job, or shaping chunk size so invalidation cost stays proportionate.
- When would you accept the much slower entity-iteration approach?When the change genuinely requires the object layer: lifecycle callbacks that maintain audit columns, cascading, domain validation, or logic that calls out to Java code. Then I iterate with keyset pagination, flush and clear every few hundred entities, use a stateless session where possible, and size the batch window around the measured throughput.
saying these in an interview costs you the question
- Proposing a single ten-million-row statement on a live OLTP system without discussing lock time or replication lag
- Chunking with OFFSET pagination, so each chunk rescans a growing prefix
- Forgetting the job's own session accumulates stale managed entities across chunks
- Ignoring that concurrent holders of the rows need version bumping or will overwrite the change
- Treating throughput as the only axis and never asking whether the change must be atomic