A nightly subscription-billing run dies with RecursionError — how do you tell runaway recursion from legitimately deep data?

## Why the distinction is the whole diagnosis A `RecursionError` has exactly two causes, and they have opposite fixes. **Runaway recursion** — a cycle in the data or a base case that never matches — will exhaust any limit you give it, so raising the ceiling converts a fast failure into a slow, expensive one. **Legitimately deep data** — a chain that really is thousands of links long — is a capacity question, and the answer may be to allow more depth or to stop using the call stack to hold it. Deciding which one you have is the first, and often the only, real step. ## Read the traceback first CPython's traceback machinery collapses repeated frames, so the shape of the output is diagnostic: * **One or two distinct frames, then `[Previous line repeated 996 more times]`.** That is a self-sustaining cycle. Nearly always: a missing or unreachable base case, a structure that contains itself, mutual recursion between two functions with no terminating branch, or an attribute-lookup loop. * **A long, varied chain of frames.** Real depth. The call chain shows progress through genuinely different work at each level. Watch for the recursion that is not written as recursion at all. A `__getattr__` that reads `self.<something missing>` re-enters itself on every lookup; a `__repr__` that formats a self-referential object recurses through formatting — often while the logger is trying to render the very object involved in the original failure, which is how a small bug turns into a confusing one. ## Reproduce on one record, not the whole run A nightly batch is the worst place to debug. Capture the identity of the failing unit at the boundary: ```python import sys def process(account, walk): try: return walk(account) except RecursionError: print("recursion blew up on", account.id, "limit", sys.getrecursionlimit()) raise ``` Then reproduce that one record in isolation with the limit **lowered** — a value of a few hundred makes a runaway fail almost instantly and, importantly, keeps the traceback small enough to read. Thread an explicit depth parameter through the walk and record the maximum reached; comparing that number with the size of the record separates "1000 levels for 1000 links" from "1000 levels over a five-element structure". A cycle check settles it definitively: carry a set of already-visited identifiers through the walk and assert that the current node is not in it. If the assertion fires, you have a cycle in the data and the traversal was never going to terminate. ## The clock-skew shape A characteristic real cause in billing work: plan amendments that record which earlier amendment they supersede, ordered by an effective timestamp written by whichever host handled the change. When two hosts' clocks disagree, two amendments can each end up recorded as superseding the other. Nothing in the row-level data looks wrong; each record is individually valid. But the "walk back to the original plan" traversal now has a two-node cycle, and it recurses until the interpreter stops it. The tell is exactly the traceback shape above — two frames, repeated — plus a memory curve that climbs to a **2.4 GB working set** before the error, because every live frame retains its locals and the amendment objects they reference. Structural corruption from a timing defect looks like a code bug in the traceback, which is why the cycle probe matters more than re-reading the traversal. ## What memory tells you RSS climbing steeply just before the failure is normal for deep recursion and says little on its own — thousands of frames each holding locals add up quickly, even when the underlying data is small. What is informative is the *ratio*: a large working set against a small input is a cycle re-holding the same few objects through thousands of frames; a large working set against a genuinely large input is depth. Note that in a batch the memory may never be released promptly even after the exception, since the frames are only freed as the stack unwinds. ## Do not fix it by raising the limit Raising `sys.setrecursionlimit` before diagnosing is the standard mistake. For a cycle it buys nothing but a longer, more memory-hungry failure, and it can turn a catchable exception into a process death if the recursion passes through C code. It also hides the defect: the run may then succeed on most nights and fail on the ones where the skewed records appear, which is a far harder bug to schedule work against. ## Dispositions once you know * **Cycle:** fix it where it is created — a deterministic ordering key rather than a wall-clock comparison, plus a database-level guard against a two-way supersedes relation. Make the traversal defensive as well: carry the visited set in production, and fail with the offending identifiers rather than with a stack error. * **Genuine depth:** decide deliberately. Cap the depth at a documented bound and reject beyond it, give the work a thread with a larger stack, or move the pending work off the call stack. Whatever you choose, record the expected maximum depth as a tested property of the ingestion path so the next growth in the data is caught by a test rather than at 3 a.m. * **Either way:** log the entity identifier at the point of failure. A `RecursionError` without the record that caused it is nearly undebuggable in a batch that processes millions of them.