A metrics scraper prunes stale series from its dict registry inside a broad except that swallows errors, and each pass drops only one series — how do you diagnose and fix it?

## Reading the symptom Three facts arrive together: the registry grows without bound, each housekeeping pass removes exactly one entry no matter how many are stale, and the logs are clean. That combination is almost diagnostic on its own. "Exactly one per pass" means the loop aborts immediately after its first successful mutation. "Clean logs" means something is catching the abort. Put them together and you are looking at a loop of this shape: ```python def prune(registry): pruned = 0 try: for name, series in registry.items(): if series.is_stale(): del registry[name] pruned += 1 except Exception: pass return pruned ``` The first `del` succeeds. The next call to the view iterator's `__next__` compares the registry's current size against the size recorded when iteration began, finds a mismatch, and raises `RuntimeError: dictionary changed size during iteration`. The bare handler discards it, `prune` returns 1, and the caller — which probably logs "pruned 1 series" — looks healthy. Two defects are stacked here, and both need naming. The mutation-during-iteration bug is the mechanical one. The catch-all handler is the one that turned a loud, immediate, obvious exception into a slow memory leak, and it is the more serious of the two. ## Confirming it before changing anything Do not fix a guess. In order of cost: 1. **Make the handler talk.** Replace `pass` with `logging.exception("prune failed")`. One deploy, and the traceback names the file, the line and the exception type. 2. **Reproduce in a test.** A registry with two stale entries and one live one, one call to `prune`, assert the registry holds only the live one. It fails immediately and keeps failing until the fix lands. This test is the deliverable, not the log line. 3. **Instrument the caller.** Compare the number of entries the pass believed were stale against `len(registry)` before and after. A gap between "found stale" and "actually removed" localises the abort without touching the handler. 4. **Confirm the growth is the registry** rather than something else, with `tracemalloc` snapshots taken a few minutes apart and diffed, or simply by logging `len(registry)` on every pass. ## The fix Two phases: decide, then mutate. ```python def prune(registry): stale = [name for name, series in registry.items() if series.is_stale()] for name in stale: registry.pop(name, None) return len(stale) ``` The comprehension finishes iterating before a single deletion happens, so the guard never fires. `dict.pop` with a default tolerates a key another path already removed. `for name in list(registry):` is the smaller edit and equally correct; the two-phase form is preferable here because it gives you the count to log and a natural place to record *which* series were dropped. If the registry is large and mostly stale, rebuilding is cheaper still — but only if you replace the contents rather than the object, because a scrape thread holding a reference to the old dict would otherwise keep publishing from it. ## The concurrency variant A metrics scraper usually has more than one writer: a scrape loop inserting new series and a housekeeping timer pruning old ones. Then the same `RuntimeError` appears in a pruner whose body does not mutate anything at all, intermittently, only under load — because the *other* thread changed the size mid-traversal. The guard reports a size mismatch; it does not care who caused it. The fix is not a retry. Either hold the lock that protects the registry across the whole traversal, or take a snapshot under the lock (`items = list(registry.items())`) and do the slow decision work outside it. Which one depends on how long the predicate takes: a cheap staleness check can run under the lock, an expensive one must not. ## Closing the class, not the instance The one-line fix ships in an afternoon. The reason this survived to production is worth a longer conversation, and on a four-person team it is a cheap one to have: * **Ban the silent catch-all.** `except Exception: pass` in a maintenance path is a defect on sight. If a broad catch is genuinely needed — a background loop that must not die — it must log with the traceback and increment a counter, and that counter must be visible. * **Alert on the invariant, not the log line.** Registry size, or entries pruned per pass, is the signal that would have caught this in hours instead of weeks. * **Make the review rule concrete.** "Mutating the collection you are iterating" is easy to spot once someone has been bitten; agree that it is a blocking review comment and let the linter carry what it can. * **Test the sweeper with more than one item.** A cleanup test that stages a single stale entry passes happily against this bug. Two is the number that matters. ## What separates a senior answer A junior fixes the loop. A senior fixes the loop, deletes the handler that hid it, adds the two-item test that would have caught it, and asks what else in the service is wrapped in a catch-all that has been quietly swallowing exceptions all along.