Why is passing data-derived text as re.sub's replacement string unsafe, and what is the fix?

### Two grammars, one call `re.sub(pattern, repl, string)` parses two different mini-languages. The `pattern` is a regular expression, where `.`, `*`, `(`, `[` and friends are metacharacters. The string `repl` is a **replacement template**, where almost nothing is special *except* backslash sequences: * `\1` … `\99` — insert the text captured by that group. * `\g<1>`, `\g<name>` — the explicit forms; `\g<0>` is the whole match. * `\\` — one literal backslash. * `\n`, `\t` and the other standard string escapes. * `\` followed by any other ASCII letter — an error, not a literal. Everything else is copied through. The failure mode follows directly: the moment part of `repl` comes from data rather than from your source, you have handed that data a grammar. ### What actually goes wrong ```pycon >>> import re >>> re.sub(r"x", "C:\\genome\\1new", "x") Traceback (most recent call last): ... re.error: missing < at position 4 ``` The value looks like a plain Windows path. The template reader sees `\g` and demands `<`, so a six-hour nightly annotation run dies at hour five on the one record whose field held a path. Change the letter and you get a different failure for the same reason: `\1` when the pattern defines one group inserts that group's text instead of a literal `1`; `\1` with no groups raises "invalid group reference"; `\d` raises "bad escape". Worst of all is the silent case — a `\n` in the data becomes an actual newline in the output, so the file has one extra line per bad record and nothing raises at all. Corruption that does not raise is the expensive kind: you find it when a downstream consumer misaligns columns, weeks later. Note that the exception class here is `re.error` — also exposed under the name `re.PatternError` since Python 3.13, which is what a 3.14 traceback prints — and its message points at the *replacement*, not the pattern — which is why the first debugging instinct, staring at the regex, wastes the most time. ### The fix Pass a callable. Its return value is inserted verbatim, with no template scan: ```python import re value = r"C:\genome\1new" re.sub(r"x", lambda m: value, "x") # 'C:\\genome\\1new' — literal ``` This is not a trick; it is the documented difference between the two forms of `repl`, and it is the reason "when do you need a callable?" has an answer beyond "when you need arithmetic". If you are forced to keep the string form — a config file that ships templates, say — then escape the value for *that* grammar by doubling backslashes (`value.replace("\\", "\\\\")`), and be aware you have just made the value non-round-trippable if it also contained legitimate group references. ### `re.escape` is the other grammar The reflex answer in interviews is "use `re.escape`", and it is wrong here. `re.escape` prepares text for use in a **pattern**: it backslash-escapes regex metacharacters so a literal string matches itself. It knows nothing about `\g<name>`, and running it over a replacement adds backslashes that the template will then try to interpret. The correct pairing is: data going into the pattern gets `re.escape`; data going into a string replacement gets escaped for the template, or, better, does not go into a string replacement at all. ```python re.sub(re.escape(user_pattern), lambda m: user_value, text) ``` ### The related ambiguity Even with hand-written templates, the numeric form has a parsing hazard: `\10` means group 10, not group 1 followed by `0`. If the pattern has one group, that raises; if it has ten, you silently get the wrong group. `\g<1>0` is unambiguous, and is the reason to prefer the `\g<...>` spelling in any template that abuts digits. Named groups plus `\g<name>` remove the class of problem entirely. ### Operating advice Three habits make this a non-issue. First, treat the replacement side as untrusted whenever any of it is data — default to a callable. Second, unit-test the substitution with a hostile value (a backslash, a `\g`, a `\1`, a newline) rather than with clean sample rows; the clean rows will never catch it. Third, ask whether a regex is needed at all: if you are swapping a fixed substring for another fixed substring, `str.replace` has no grammar on either side and cannot be attacked by a backslash. `Match.expand` exists for the case where you genuinely want template semantics applied on purpose — for user-configurable rewrite rules, for example — and it makes that intent explicit rather than accidental. ### In an interview State that `repl` is a template, name two concrete failures (a raised `re.error` on `\g`, a silent group insertion on `\1`), give the callable as the fix, and explicitly reject `re.escape` as belonging to the pattern side. That last correction is what makes the answer read as experience rather than recall.