Why does a collections.defaultdict grow when you only read a missing key?

## Insertion is the point, not a side effect The reason `groups[k].append(v)` works at all is that the object `__missing__` returns is the object it has just stored. If it handed back a detached empty list, the `append` would mutate a throwaway and the mapping would stay empty forever. So insert-on-miss is load-bearing behaviour, not an oversight — but it means the subscript operator on a `defaultdict` is a *write*, and code review habits built around plain `dict` do not transfer. ## Exactly which operations insert Only `__getitem__` consults `__missing__`. That gives a short, memorable split: - **Inserts:** `d[key]`, and anything built on subscripting — `d[key].append(...)`, `if d[key]:`, `len(d[key])`, `for x in d[key]:`, an f-string interpolating `d[key]`. - **Does not insert:** `key in d`, `d.get(key)`, `d.get(key, fallback)`, `d.keys()` / `d.values()` / `d.items()`, `d.pop(key, fallback)`, and `d.setdefault(key, value)` — `setdefault` inserts the value *you* passed and ignores `default_factory` entirely. That last one surprises people: on a `defaultdict`, `setdefault` behaves exactly as it does on a plain `dict`. ## Where it bites in production Take a clinical-lab result loader ingesting a 6,800-row batch into `results = defaultdict(list)`, keyed by collection date parsed out of each row. The upstream file uses a locale-dependent date format, so a slice of the batch parses into a differently-formatted key string than the rest. Nothing raises. Downstream, a summary routine written as `if results[expected_date]:` probes a handful of dates that were never populated, and each probe *creates* them. Three failures follow, in escalating order of confusion: 1. **Counts drift.** `len(results)` is larger after the report ran than before it. A metric derived from `len(results)` reports more collection dates than the batch contained. 2. **Empty groups leak into output.** `defaultdict` is a `dict` subclass, so serializers accept it happily and write out the manufactured `[]` values as if they were real, empty result sets. A consumer cannot distinguish "no results that day" from "we never looked at that day". 3. **Iteration explodes.** The moment one of those probing lookups happens inside `for date in results:`, the next step of the loop raises `RuntimeError: dictionary changed size during iteration`. This is the good outcome — it is the only one that is loud — and it is routinely misdiagnosed as a threading problem when it is a single-threaded self-inflicted insert. ## How to keep the behaviour and lose the hazard **Read with the non-inserting API.** Anywhere the intent is inspection rather than accumulation, write `d.get(key, ())` or guard with `if key in d`. Reserve the subscript for the accumulation site itself. This is the fix that generalizes; the others are safety nets. **Freeze the factory when accumulation ends.** `results.default_factory = None` after the load loop turns every later miss back into a `KeyError`. Reporting code that probes an unknown date now fails loudly instead of inventing an entry, and the data already accumulated is untouched. **Convert at the boundary.** `dict(results)` returns an ordinary `dict` with the same items and no factory. Handing that out — to a serializer, to another module, to a caller — means no downstream code can trip the behaviour, and no reviewer of that code has to know about it. **Iterate over a snapshot when a lookup is unavoidable.** `for date in list(results):` iterates a copy, so an insert during the loop no longer invalidates the iteration. Prefer removing the insert, but this is the correct patch when the lookup genuinely belongs there. **Validate keys at the edge.** The lab loader's real defect is not `defaultdict`, it is a key format that was never normalized. Normalizing dates at ingest — parsing to a `datetime.date` and keying on that object rather than on whatever string the file carried — removes the mismatch and, with it, the probing that exposed the insert. ## Interview framing The question is asked because it separates people who have *used* `defaultdict` from people who have *debugged* one. The strong answer names the mechanism (`__missing__` stores before returning), draws the insert/no-insert line correctly, and volunteers the freeze-or-convert habit without being prompted. ## Why the design is still right It is tempting to call insert-on-read a wart, but the alternative is worse. A `__missing__` that returned a detached default would make `d[k].append(v)` silently do nothing — a far quieter bug than a mapping that grew. The type is honest about what it is: a mapping whose subscript is a write. What goes wrong in practice is not the semantics but the assumption, carried over from plain `dict`, that reads are free. Treat a `defaultdict` as an accumulator with a narrow lifetime, use the non-inserting API everywhere else, and the behaviour stays where it earns its keep.