Why does a Python code-checking tool use ast.parse() instead of regular expressions over the source?

## What `ast.parse()` gives you `ast.parse(source)` hands a string of Python source to the interpreter's own parser and stops at the *abstract syntax tree* - the structured representation CPython builds before it compiles bytecode. You get back an `ast.Module` object. **Nothing in the source runs**; parsing is not execution. The tree is made of ordinary Python objects with typed fields. A module holds a `body` list of statements. An assignment holds `targets` and a `value`. A call node (`ast.Call`) holds `func`, `args` and `keywords`. Every literal - number, string, `True`, `None` - is an `ast.Constant` carrying its `value`. You can look at any of it with `ast.dump(node, indent=2)`, which prints the node type and its fields and is the fastest way to learn what shape a construct actually has. Because it is the interpreter's parser, the grammar is not an approximation. If `ast.parse()` accepts the text, the running interpreter would too; if it does not, you get the same `SyntaxError` an import would raise. ## Why text matching loses A regex sees characters. The tree sees constructs. Concretely, a pattern like `\bopen\(` matches: - the word inside a string literal or a comment, where no call happens at all; - a call spelled across a line break or wrapped in parentheses, which it may miss entirely; - an attribute call on some unrelated object, which it cannot tell apart from the builtin. The tree makes each of those a different question with a definite answer: is the node an `ast.Call`, is its `func` an `ast.Name` or an `ast.Attribute`, what is the `id`, which enclosing `ast.FunctionDef` contains it. Equally important, the tree is *normalised*: whitespace, redundant parentheses and quote style are gone, so `x=1` and `x = 1` parse to the same structure. A checker wants that; a formatter, as it happens, does not. ## Getting around the tree Three traversal styles cover almost everything: - `ast.walk(tree)` yields every node in the tree in no guaranteed depth-first order, with no parent context. It is the one-liner for "find all nodes of type X". - `ast.iter_child_nodes(node)` yields one level of children, for hand-written recursion. - `ast.NodeVisitor` dispatches to a `visit_<NodeType>` method per node type, which is what a real linter uses because rules become methods. ```python import ast tree = ast.parse("total = rows + 1") names = [n.id for n in ast.walk(tree) if isinstance(n, ast.Name)] ``` ## Positions, and what the tree cannot tell you Most nodes carry `lineno`, `col_offset`, `end_lineno` and `end_col_offset`, which is how a tool reports `file.py:12:5` and how an editor underlines exactly the offending expression. The limits matter as much as the power. The AST is **per file and purely syntactic**. It does not resolve imports, does not know types, and does not know whether the name `open` refers to the builtin or to something rebound three modules away - name resolution and typing are layers you build on top. It also discards comments and exact formatting, so a suppression comment such as a `# noqa`-style marker has to be recovered from the raw text or the token stream, never from the tree. Finally the grammar moves with the language, and a visitor written against an older one silently under-reports rather than failing. `ast.Match` nodes exist only from Python 3.10; PEP 695 type-parameter syntax and `ast.TypeAlias` arrived in 3.12; Python 3.14 added nodes for template strings (PEP 750). `ast.parse()` takes a `feature_version` argument to *restrict* itself to an older grammar, but it can never parse syntax newer than the interpreter running it - so a tool that must analyse code for several Python versions either runs under the newest one it supports or keeps a parser of its own. ## The practical shape of a checker Parse the file, walk or visit the tree, collect `(rule, lineno, col_offset, message)` tuples, print them. That is genuinely the whole architecture of a simple linter, and it is why `ast` is in the standard library: treating code as data is a supported, first-class thing to do in Python. ## Two details worth carrying `ast.parse()` takes a `mode` argument. The default `"exec"` parses a whole module and gives an `ast.Module`; `mode="eval"` parses a single expression and gives an `ast.Expression`, which is how you check that a configuration value is one expression rather than a block of statements. And it raises `SyntaxError` on anything the running interpreter cannot parse - including source written for a newer Python - so a tool that walks a repository must catch it per file and report the file rather than dying on the first one it does not understand.