skip to content

In pandas, why can sku.str.contains('BLK+GRN') miss the value 'CAP-OS-BLK+GRN' while sku.str.replace('+', '/') edits it correctly?

level: middleimportance: should knowfreq 45%

answer

  1. two methods, two regex defaults
  2. a plus sign is a quantifier
  3. the regex keyword on each method
  4. escape or switch to literal

basics

~20 s

Series.str.contains treats its pattern as a regular expression by default (regex=True), so '+' means one or more K. Series.str.replace defaults to regex=False and replaces the literal '+'. Pass regex=False to contains for a literal match.

solid answer

~30 s

`str.contains` has `regex=True` by default, which makes `"BLK+GRN"` mean `BL`, one or more `K`, then `GRN`; it never matches the plus sign, and the row comes back `False` with no error. `str.replace` has had `regex=False` by default since pandas 2.0, so `"+"` is the character itself. Fix the filter with `regex=False`, or escape the metacharacter (`re.escape(...)` or `r"BLK\+GRN"`). Related traps: `"."` in a `contains` pattern matches any character, and a capture group in it makes `contains` emit a `UserWarning` pointing to `str.extract`, so use `(?:...)`. `startswith` and `endswith` never use regex; they take a string or a tuple of strings.

code

python · 8 lines
python
import pandas as pd

sku = pd.Series(["CAP-OS-BLK+GRN", "CAP-OS-BLKGRN", "TSH-L-RED"])

print(sku.str.contains("BLK+GRN").tolist())               # regex: K+ is one or more K
print(sku.str.contains("BLK+GRN", regex=False).tolist())  # literal plus sign
print(sku.str.replace("+", "/").tolist())                 # replace is literal by default
print(sku.str.contains(r"-(?:L|XL)-").tolist())           # non-capturing group, no warning

go deeper

for a junior

Recall that str.contains reads its pattern as a regex by default while str.replace reads it as plain text.

for a middle

Explain which string methods use regex, literal text or a length rule, and how regex=False, escaping and startswith avoid the traps.

for a senior

Diagnose a filter that silently drops rows because of an unescaped metacharacter, and build patterns from data with re.escape.

for a principal

Consider a team rule for string filters, literal by default with regex only by intent, so data-driven patterns cannot change meaning.

## The two defaults pandas string methods do not agree on whether a pattern is a **regular expression** (a small language in which `.`, `+`, `*`, `?`, `(`, `[` and `|` are operators) or **literal text**. The defaults, from the method signatures: | method | how `pat` is read by default | switch | |---|---|---| | `str.contains` | regular expression | `regex=False` for literal | | `str.match`, `str.fullmatch` | always a regular expression | none | | `str.extract`, `str.extractall`, `str.count` | always a regular expression | none | | `str.replace` | **literal** (since pandas 2.0) | `regex=True` for regex | | `str.split` | literal if one character, regex if longer | `regex=True` or `False` | | `str.startswith`, `str.endswith` | always literal; a string or a tuple | none | ## Why the filter fails silently In a catalog where bundle SKUs join two colours with a plus sign (`"CAP-OS-BLK+GRN"`), the filter `sku.str.contains("BLK+GRN")` compiles the pattern as a regex. `K+` means "one or more K", so the pattern matches `BLKGRN` or `BLKKGRN` and **never** a literal `+`. The pattern is valid, so nothing raises: the bundle row is just `False`, and the filtered report quietly loses it. Meanwhile `sku.str.replace("+", "/")` reads `"+"` literally and produces `"CAP-OS-BLK/GRN"` as intended, which makes the two lines look inconsistent until you know the defaults. Other characters cause the same quiet errors in `contains`: - `.` matches any character, so `contains("1.5")` also matches `"125"`; - `(` and `[` start groups and classes, and an unbalanced one makes the call fail with a regex error; - `|` means "or", so `contains("S|M")` matches any value containing `S` or `M`. ## Fixes 1. Pass **`regex=False`** when the pattern is plain text: `sku.str.contains("BLK+GRN", regex=False)`. 2. Or keep the regex and **escape** the operator: `r"BLK\+GRN"`, or build the pattern with `re.escape(text)` when the text comes from users or data. 3. For prefixes and suffixes, use **`startswith`/`endswith`**, which never interpret regex and accept a tuple such as `("TSH-", "HOD-")`. 4. In `replace`, opt in with `regex=True` when you do want a pattern; a callable `repl` or a compiled pattern requires it, and combining either with `regex=False` raises `ValueError`. ## Patterns built from data The trap is worst when the pattern is not typed by hand but taken from data, such as a list of SKUs to exclude or a search box in a notebook tool: - a value containing `+`, `.`, `(` or `?` changes meaning silently, or makes the call fail with a regex error for an unbalanced bracket; - `"|".join(codes)` builds an "any of these" regex only if every code is escaped first: `"|".join(map(re.escape, codes))`; - when you only need exact membership rather than a substring, `isin(codes)` avoids regex altogether and states the intent better. ## The capture-group warning `contains` only answers yes or no. When its regex has a **capture group**, as in `r"-(L|XL)-"`, pandas emits a `UserWarning`: the pattern has match groups, and `str.extract` is the method that returns them. The booleans are unaffected. Write the group as non-capturing, `r"-(?:L|XL)-"`, to keep the filter quiet, or call `extract` if you wanted the size. ## Literal replacement in practice Because `replace` is literal by default, everyday cleaning reads naturally: - `sku.str.replace("+", "/")` swaps every plus sign; `n=1` limits it to the first occurrence per value (the default `n=-1` replaces all). - `sku.str.replace(".", "-")` turns `TSH.L.RED` into `TSH-L-RED`; the dot is a plain character here, not "any character". - A regex replacement must be asked for: `sku.str.replace(r"-+", "-", regex=True)` collapses repeated hyphens, and group references such as `\1` in `repl` work only in that mode. ## Case and flags - `case=False` makes `contains` and `replace` case-insensitive; `flags=re.IGNORECASE` does the same through the `re` module flags. - A **compiled** pattern (`re.compile(...)`) carries its own flags. `replace` raises `ValueError` if you also pass `case` or `flags`, so put them in `re.compile`. - `match` anchors at the start of the string, `fullmatch` requires the whole string, and `contains` matches anywhere. What `contains` returns for **missing** entries depends on the column's dtype and its `na` argument; filtering with such a mask is covered with boolean masks.

  • How do you make the contains filter case-insensitive?
    Pass `case=False`, or `flags=re.IGNORECASE`. If the pattern is a compiled regex, put the flag in `re.compile` instead, because the compiled object carries its own flags and pandas will not combine them with separate `case` or `flags` arguments.
  • How do str.match and str.fullmatch differ from str.contains?
    All three use regular expressions. `contains` looks for a match anywhere, like `re.search`; `match` needs the match to begin at the first character, like `re.match`; `fullmatch` needs the pattern to cover the whole string, like `re.fullmatch`. For validating SKU format, `fullmatch` is the strict choice.

  • str.contains matches literal text unless you pass regex=True.
  • str.replace reads its pattern as a regex by default in current pandas.
  • str.startswith accepts a regular expression such as '^TSH'.
  • A '+' in a contains pattern raises an error, so silent misses cannot happen.
  • A capture group in a contains pattern makes it return the matched text.