ElementTree.parse OOMs on a 6 GB XML feed; how do you stream it with iterparse?

## Why the in-memory parse dies `ET.parse(path)` is a batch operation: it reads the whole document and returns only once every element object exists. Each element carries a tag string, an attribute dict, text and tail strings and a child list, so the object graph for a record-heavy feed routinely costs several times the on-disk bytes. Take a fraud-scoring service whose nightly reconciliation reads a 6 GB transaction feed: the process is not slow, it is dead — and the failure mode is nasty, because a 27-minute suite that ingests a scaled-down fixture in CI passes happily and only production sees the resident-set curve climb until the kernel intervenes. The fix is not a bigger box. It is to stop holding the document. ## The iterparse contract `ET.iterparse(source, events=None)` takes a filename or a binary file object and returns an iterator of `(event, element)` pairs, produced incrementally as the parser consumes input. The event names are `'start'`, `'end'`, `'start-ns'`, `'end-ns'`, `'comment'` and `'pi'`; the default is `('end',)`. The semantics of the two main events are what the whole technique rests on. `'start'` fires when the open tag has been read: the element exists and its attributes are set, but its children and text are not yet parsed, so it is not safe to process. `'end'` fires when the close tag has been read: that element and everything beneath it are complete. Record processing therefore belongs on `'end'`, keyed by tag. Crucially, `iterparse` is still *building a tree* as it goes. Each finished element is appended to its parent. If you do nothing, you have simply re-implemented `parse` with extra steps and the same peak memory. ## The two-step release ```python context = ET.iterparse(path, events=("start", "end")) _, root = next(context) # the root element, as soon as it opens for event, elem in context: if event == "end" and elem.tag == "txn": handle(elem) elem.clear() # drop children, text, tail and attributes root.clear() # unhook the emptied record from the tree ``` `Element.clear()` resets one element: children removed, `text`, `tail` and `attrib` emptied. The element object itself survives, and — this is the part candidates miss — it is still in its parent's child list. So `elem.clear()` alone leaves a growing list of empty shells hanging off the root, and peak memory still rises with record count. Clearing the root drops those references so the shells can be collected. In a quick measurement over a synthetic feed, adding `root.clear()` roughly halved peak traced allocation at five thousand records, and the gap widens linearly beyond that, because one path is O(records) and the other is O(1). `root.clear()` is safe here precisely because you have already finished with every previous record. If the root carries attributes you need, read them once at the `start` event before the loop. For nested structures where the record's parent is not the root, keep a reference to that parent and clear it instead, or remove the finished child explicitly. ## What you give up Streaming buys constant memory and pays for it in expressiveness. You get exactly one forward pass, so any aggregate — sums, per-account grouping, deduplication — must be accumulated as you go, and a computation that needs to look back at earlier records needs its own bounded state, not the tree. You cannot run a `findall` over the document afterwards, because the document no longer exists. Anything that genuinely needs random access wants a different design: a first pass that extracts a compact index, or a load into storage that supports queries. Error handling changes shape too. A malformed document raises `ET.ParseError` partway through the loop, after side effects have already happened for the records that parsed. If the consumer is not idempotent, wrap it in a transaction, stage the output, or record a resume position. ## XMLPullParser, and when to prefer it `iterparse` insists on owning the source: you give it a path or a file object and it reads. When the bytes arrive some other way — a socket, a decompressing wrapper, chunks from an HTTP response — use `ET.XMLPullParser(events=('end',))`, feed it with `feed(chunk)` as data arrives, and drain completed pairs with `read_events()` between chunks. It is the same event model with the pump inverted, and it is what lets a streaming consumer stay non-blocking. ## Framing the answer in an interview The strongest version of this answer names the memory model first (objects, not bytes), then the event semantics (`end` means complete), then the two-step release with a reason for the second step, and closes on the tradeoff: one pass, incremental state, no post-hoc search. Mentioning that the CI fixture is too small to catch this is what makes it sound like a lesson learned rather than a recipe memorised.