Is re.compile worth it given that the re module already caches compiled patterns?

## What actually happens on a module-level call `re.search(r"\d+", s)` does not compile `\d+` afresh each time. Internally the module keeps a cache mapping `(type of pattern, pattern text, flags)` to the compiled `re.Pattern` object. The first call compiles and stores; every later call with the same key finds the ready-made object and dispatches to its method. You can observe the caching directly, because `re.compile` goes through the same cache: ```python import re p1 = re.compile(r"\d+") p2 = re.compile(r"\d+") p1 is p2 # True - the same cached object re.purge() p3 = re.compile(r"\d+") p1 is p3 # False - purge() cleared the cache ``` So the common claim "always `re.compile` your patterns, otherwise Python recompiles them every call" is wrong as stated. What remains after the caching is a *lookup*: hashing the pattern string and the flags, a dict probe, and an extra function call layer. Measurable in a tight loop, invisible almost everywhere else. ## Where re.compile does pay **1. The cache is bounded and it evicts.** CPython's pattern cache holds 512 entries; when it fills, the least-recently-used entry is dropped. A program that generates patterns dynamically — one per rule, one per field, one per file — can push past that and start recompiling on every call, and nothing tells you. Consider a video-metadata extractor that builds a pattern per caption-format rule and runs a 340-case regression pack: hundreds of distinct patterns cycling through a bounded cache is exactly the shape that thrashes. A compiled object held in a variable is immune: it exists because you hold a reference, not because a cache decided to keep it. **2. It removes the lookup from the hot path.** In a loop over millions of lines the saving is real, if modest. Measure it rather than asserting it — `timeit` will tell you in seconds whether it matters for your pattern and input. **3. The `pos` and `endpos` arguments exist only on the methods.** `pattern.search(s, pos, endpos)` restricts the scan to a window **without slicing the string**, so no copy is made. The module-level functions have no such parameters. Tokenisers and incremental scanners need this. **4. It is better engineering.** A compiled pattern named at module scope — `_RESOLUTION = re.compile(r"\d+p")` — is compiled once at import, is visible to a reader looking for the module's regexes, can be tested in isolation, and has an obvious place for the comment explaining it. A pattern literal inlined in a function body is invisible until you grep. **5. It fails fast.** A malformed pattern raises `re.error` at compile time. Compiled at import, a typo is an immediate startup failure; hidden inside a rarely-taken branch, it is a production incident at 3 a.m. ## Flags and the cache key The cache key includes the flags, so the same pattern text with different flags is a different entry — as it must be. It also includes the *type*, which is why a `str` pattern and the equivalent `bytes` pattern are separate compiled objects and never collide. And the `DEBUG` flag deliberately bypasses caching, so its output is printed on every compile. ## The stale-value trap Because a compiled pattern is a cached object keyed by text, the natural instinct to build a "smart" wrapper that caches compiled patterns *itself*, keyed by something else — a rule name, a config version — is where genuine staleness bites. If configuration changes the pattern text but your key does not change with it, the wrapper keeps returning a compiled pattern for the *old* rule, and the extractor silently keeps matching yesterday's format. The module cache does not have this problem precisely because its key **is** the pattern text: a changed pattern is a different key. If you write your own layer, key it on the pattern text too, or clear it when configuration reloads. ## What to say in an interview "Compiling is cached, so `re.compile` is not primarily a performance fix. I use it because it gives me a named constant compiled at import, it cannot be evicted from a bounded cache, and it is the only way to pass `pos` and `endpos`. If someone claims a large speed-up, I ask for the benchmark — and I check whether their program has more distinct patterns than the cache holds, because that is where the real regression usually is." `re.purge()` exists to clear the cache; it is a benchmarking and memory-reclamation tool, not something a normal program calls. If you are timing regex code, purge between runs or your first measurement will be the only honest one.