A chat-transcript archiver using xml.etree.ElementTree on user-supplied exports intermittently blows its 92nd-percentile latency budget — how do you confirm entity expansion is the cause and harden the parse?

## Start by separating "slow" from "amplified" The two look identical on a latency dashboard and are trivial to tell apart once you record the right pair of numbers. For every export, log: - the input size in bytes, - the wall time of the parse, - and peak allocation during it — `tracemalloc` around the call is enough while you are investigating, or a resident-memory sample if you cannot afford the overhead. Normal parses scale roughly linearly with input size. An entity-expansion document is a **vertical outlier**: a payload measured in hundreds of bytes producing seconds of CPU and a memory spike, because the cost lives in expanded text that never appeared in the input at all. If your outliers sit on the size trend line, you have a slow parse — a large export, a cold disk, a contended worker — and entity expansion is not your problem. ## Then get proof rather than a plausible story On CPython 3.14 with a modern linked libexpat, the amplification guard fires and the failure arrives as `xml.etree.ElementTree.ParseError` whose message reads "limit on input amplification factor (from DTD and entities) breached". That string is the confirmation you want, and it means the attack was refused rather than absorbed. Two caveats decide how much comfort to take from it. 1. First, the guard is in the C library, so read `xml.parsers.expat.EXPAT_VERSION` on the exact image you deploy: 2.4.1 introduced the amplification limit and 2.6.0 fixed the quadratic large-token case, and a build linking an older system library has neither — there the same document produces no error, just consumption, which is precisely the intermittent-timeout shape you are chasing. 2. Second, if your code wraps the parse in a broad `except` and returns a default, that message never reaches a log; the first fix is often just to log the exception with the document's size, source and a truncated prefix, and to keep the offending payload for analysis. ## Also check what the archiver does with entity references it does not expand External general entities have not been processed by default since Python 3.7.1, and `xml.etree.ElementTree` raises on the undefined reference — but if any code path reads the same exports through `xml.sax` with default features, that path completes silently and stores empty fields. Two ingestion routes disagreeing about the same document is worth finding while you are in there. ## Now harden, in the order the controls actually bite 1. **Cap the bytes before the parser sees them.** An upload limit is not an XML control, but it removes the largest class of accidental blowups and costs nothing. Apply it to the decompressed size if exports arrive compressed — a size limit on compressed bytes bounds nothing about what gets parsed. 2. **Refuse the DOCTYPE.** An export produced by your own client has no legitimate reason to declare entities. Rejecting any document with a DOCTYPE is a policy you can state in one line, test, and explain in a review. The standard library has no switch for it: `xml.etree.ElementTree.XMLParser` exposes only `entity` and `target`, so you would drive `xml.parsers.expat` yourself, install an `EntityDeclHandler` that raises, and call `SetParamEntityParsing` with `xml.parsers.expat.XML_PARAM_ENTITY_PARSING_NEVER` — then rebuild tree construction on top of that. 3. **Which is why the real fix is a hardened third-party XML library.** Its parse entry points are drop-in replacements that refuse DTDs, entity declarations and external references outright and raise a dedicated exception, giving every ingestion path one stated policy instead of a per-module mix of loud errors, silent skips and version-dependent C guards. 4. **Bound the blast radius.** A parse already running inside libexpat cannot be interrupted: there is no cancellation point, so a timer in a thread will not stop it and the worker keeps allocating. If you need a hard ceiling on the tail, parse untrusted exports in a subprocess you can kill on a deadline, with a memory limit applied to that process. That also converts the failure mode from "the archiver stops serving" into "one export failed", which is what a percentile budget actually cares about. ## Close on the operational loop - Alert on parse failures by class, not just on latency; keep a counter for amplification refusals separately from ordinary parse errors, because a rising count is an attack signal rather than a quality signal. - And record the libexpat version alongside the Python version in your build metadata, so the next person does not have to rediscover that the protection they are relying on came from a C library they never chose.