Why does compile() reject a tree edited with ast.NodeTransformer until you fix its locations?

## The failure Edit a tree, hand it to `compile()`, and you get something that surprises people the first time: ```pycon >>> import ast >>> tree = ast.parse("x = 500") >>> tree.body[0].value = ast.Constant(value=1000) >>> compile(tree, "<codemod>", "exec") TypeError: required field "lineno" missing from expr ``` Note the exception type. This is not a `SyntaxError` - the tree is syntactically fine. It is a `TypeError` about a **missing required field**, because `compile()` is being handed an AST object that does not satisfy the shape the compiler requires. ## Why the field is required Every expression and statement node produced by `ast.parse()` carries four position attributes: `lineno`, `col_offset`, `end_lineno` and `end_col_offset`. The parser fills them in from the source text. The compiler consumes them: it writes a line-number table into the code object so that at runtime, a traceback frame, a profiler sample, a coverage record or a debugger breakpoint can be mapped back to a place in a file. A node you construct in Python - `ast.Constant(value=1000)`, `ast.Call(func=..., args=[], keywords=[])` - has no such attributes, because you never gave it any. The compiler cannot invent them, so it refuses. ## The two repairs **`ast.fix_missing_locations(tree)`** walks the tree from the top and, for any node missing position attributes, copies them from its parent. It is idempotent, cheap, and the standard last step of a rewrite. It is also blunt: an entire spliced-in block inherits one parent's line number, so five injected statements all claim to be on the same line. **`ast.copy_location(new_node, old_node)`** copies the four attributes from one node to another. Use it when your transformer replaces node A with node B and you want B to occupy exactly A's span - the finer tool, and the one that keeps diagnostics honest. A third helper, `ast.increment_lineno(node, n)`, shifts a subtree's line numbers, which matters when you inject code that genuinely adds lines to a file you also rewrite on disk. The usual discipline: `ast.copy_location()` per substitution while you rewrite, then one `ast.fix_missing_locations()` over the whole tree as a backstop before compiling. ## Why a wrong position is worse than a missing one Consider instrumenting a clinical-lab result loader whose resident memory climbs without bound across a run, by injecting an allocation-tracking call at the top of every function with an `ast.NodeTransformer` and compiling the rewritten tree at import time. The point of the exercise is to learn *which* loader function accumulates. Every answer you get - the traceback of the eventual failure, the line attribution in a sampling profiler, the breakpoint you set - is read out of the position fields you just wrote. If `ast.fix_missing_locations()` smeared thirty injected statements onto the line of their enclosing function, your report says "line 41" for everything and identifies nothing. Worse, the filename argument to `compile()` and the line numbers together decide what a traceback frame prints: pass `"<codemod>"` and no source is shown at all; pass the real path with shifted line numbers and the traceback confidently shows the *wrong* source line, which is how an instrumentation pass costs a team the better part of a release train chasing a function that was never the problem. So the rule is: injected nodes inherit the position of the *real* node they instrument, never an arbitrary one, and you compile with the real filename only if the line numbers still correspond to that file. ## The order of operations 1. Rewrite with `ast.NodeTransformer`, using `ast.copy_location()` on each substitution. 2. Call `ast.fix_missing_locations(tree)` once on the result. 3. `compile(tree, filename, "exec")` to get a code object. 4. Run it, or cache it. Step 2 before step 3 is not optional, and no amount of correct structure substitutes for it. A useful test-time check: `ast.unparse()` will happily render a tree whose positions are missing, so "it unparses fine" proves nothing about whether it compiles. Only `compile()` enforces the position fields. ## Where the cost lands Parsing and rewriting a module is not free, and doing it on every import of every file is felt at start-up. If a rewrite pass is permanent rather than a temporary investigation, compile once and cache the resulting code object keyed by the source's content hash, so the tree work happens only when the file changes. That also forces a useful question: a pass expensive enough to need caching is usually a pass that should have been a build step producing reviewable output, not an invisible transformation applied at load time.