An import loop persists 200,000 entities through one EntityManager without ever flushing or clearing it. It starts fast, gets progressively slower, and eventually runs out of memory. Explain what is happening and how you would restructure it.

## What `persist()` actually does `persist()` does not necessarily write a row. It makes the instance **managed**: registered in the persistence context, scheduled for insertion at the next flush. The persistence context — the first-level cache — is a map from entity key to instance, plus, for dirty checking, a snapshot of each entity's state. So a loop of 200,000 `persist()` calls builds a 200,000-entry map of objects that Hibernate must keep alive until the context is cleared or closed. ## Failure 1 — memory Heap use is roughly *(entity + snapshot + map entry) × n*, and none of it is eligible for garbage collection while the context lives, because the context strongly references everything. This is the direct cause of the OutOfMemoryError. Enlarging the heap only moves the threshold. If the entities also cascade to child collections, each child is managed too, and the multiplier grows. ## Failure 2 — quadratic time Flushing means: walk every managed entity, compare its current state to its snapshot field by field, and schedule the resulting SQL. That is O(number of managed entities) per flush, regardless of how many changed. If flushes happen periodically — because a query is issued, because the batch size is reached, or at commit — the work is 1 + 2 + 3 + … over a growing context, i.e. quadratic overall. The observable symptom is exactly the one described: fast at the start, crawling by the end. ## Failure 3 — one round trip per row Even with memory under control, `n` separate `INSERT` statements means `n` network round trips. At 1 ms each, 200,000 rows is over three minutes of pure latency. Two things block batching more often than people expect: - **`GenerationType.IDENTITY`.** Hibernate must know the generated key immediately, so it executes each insert on `persist()` rather than deferring it to flush — which makes batching impossible. Sequence-based generators with a pooled optimiser allow both deferral and batching. - **Interleaved statement types.** A batch breaks whenever the statement changes, so mixing inserts into two tables row by row produces batches of one. Ordering inserts by entity type keeps batches full. ## The restructured loop ```java int batch = 500; for (int i = 0; i < rows.size(); i++) { em.persist(toEntity(rows.get(i))); if (i > 0 && i % batch == 0) { em.flush(); // send the pending inserts em.clear(); // detach them so the heap can be reclaimed } } em.flush(); em.clear(); ``` Both calls are required and in that order: `flush()` without `clear()` leaves memory growing; `clear()` without `flush()` discards the pending work. Supporting settings and habits: - Enable JDBC batching and align its size with the loop's batch size, so a flush emits a few multi-row statements instead of 500 individual ones. - Use a sequence generator, ideally pooled, so ids can be assigned without a round trip per row. - Keep the second-level cache out of the write path for imports; populating it with rows nobody will read is pure overhead and eviction churn. - Beware `cascade = ALL` graphs: persisting a parent can drag hundreds of children into the context per iteration. - Consider a **stateless** write path when the job needs no dirty checking, no cascade and no first-level cache — it writes directly and keeps nothing, so memory is flat by construction. - For pure data movement inside the database, an `insert ... select` bulk statement beats every ORM approach, because no row ever leaves the database. ## Transaction shape One enormous transaction holds locks and undo/redo space for the entire run and forces a full rollback on the last row's failure. For imports, chunked transactions (commit every *k* batches) with a recorded resume point are usually the right trade; if the job must be atomic, that is a deliberate decision to hold one long transaction, not an accident. ## How to verify the fix Run the job with statement counting and a heap graph. You want: a flat memory profile across the run, statement count roughly *rows / batch size*, and a wall-clock time that scales linearly with row count. If time is still super-linear, something is still holding entities — usually a cascaded collection or a forgotten `clear()`.