skip to content

In pandas 3.0, a filter on df['route'].str.contains('jfk', case=False) meets rows with no route; what does the mask hold there, and how does that differ from 2.x?

level: seniorimportance: should knowfreq 45%

answer

  1. default fill depends on dtype
  2. new default text dtype
  3. object dtype keeps the old fill
  4. spell out the na argument

basics

~20 s

In pandas 3.0 text infers the str dtype, so str.contains returns a bool mask with False at missing rows: the filter drops them and its negation keeps them. In 2.x's object dtype the mask held a missing value and filtering raised ValueError.

solid answer

~40 s

`str.contains` fills missing entries with its `na` argument, whose default depends on the dtype. In pandas 3.0 a text column infers the `str` dtype, so the default is `False` and the mask is plain `bool`: `df[mask]` silently drops rows with no route, and `df[~mask]` silently keeps them. On pandas 2.x, or an object column in 3.0, the mask holds a missing value (`NaN` or `None`), so `df[mask]` raises `ValueError: Cannot mask with non-boolean array containing NA / NaN values`. On the nullable `"string"` dtype the mask is `boolean` with `<NA>`, which indexing treats as `False` both ways. I pass `na=False` or `na=True` explicitly, keep `case=False`, and add `regex=False` when the pattern is literal text.

code

python · 17 lines
python
import pandas as pd

routes = ["JFK-LAX", None, "ORD-SFO", "jfk-mia"]

for dtype in [None, object, "string"]:
    flights = pd.DataFrame({"route": pd.Series(routes, dtype=dtype)})
    mask = flights["route"].str.contains("jfk", case=False)
    print(flights["route"].dtype, mask.dtype, mask.tolist())
    try:
        print("  match:", flights[mask].index.tolist())
    except ValueError as err:
        print("  match:", err)

flights = pd.DataFrame({"route": routes})
explicit = flights["route"].str.contains("jfk", case=False, na=False)
print(flights[~explicit].index.tolist())  # [1, 2]: the missing route is kept
print(flights[~flights["route"].str.contains("jfk", case=False, na=True)].index.tolist())  # [2]

go deeper

for a junior

Recall that str.contains builds a mask from text, that case=False ignores case, and that missing values need the na argument to land where you want them.

for a middle

Explain how the default fill depends on the dtype: False on str, a missing value on object, pd.NA on the nullable string dtype, and why a mask holding missing values cannot filter rows.

for a senior

Show the production judgment: notice that pandas 3.0 turns an old crash into a silent choice about missing rows, check dtypes across environments, and make na= explicit, including for the negated filter.

for a principal

Treat the 3.0 string dtype as a migration risk for filters: code that relied on an error to surface missing text now runs, so tests over missing values matter more than before.

## The scenario A flight table has a `route` column such as `"JFK-LAX"`, some rows have no route recorded, and an analyst filters with: ```python flights[flights["route"].str.contains("jfk", case=False)] ``` `Series.str.contains(pat, case=True, flags=0, na=<default>, regex=True)` returns a mask that is `True` where the pattern occurs in the text. What it puts at the **missing** rows decides whether the filter works, and that depends on the column's dtype. ## What the mask holds, dtype by dtype The `na` argument is the fill value for missing entries. Its docstring in pandas 3.0 says the default depends on the dtype: `False` for the `"str"` dtype, `numpy.nan` for object dtype and `pandas.NA` for the nullable `StringDtype`. | Column dtype | How it arises | Mask at a missing row | Mask dtype | `df[mask]` | `df[~mask]` | |---|---|---|---|---|---| | `str` | **default for text in pandas 3.0** | `False` | `bool` | drops the row | **keeps** the row | | object | default for text in pandas 2.x; `dtype=object` in 3.0 | `NaN` or `None` | object | **raises `ValueError`** | `~` raises `TypeError` | | `"string"` (nullable) | `astype("string")` | `<NA>` | `boolean` | drops the row | drops the row | So the same line of code: - **in pandas 3.0 on a default text column** runs silently, and missing routes are treated as "does not contain jfk"; - **on pandas 2.x**, or on an object column in 3.0, raises `ValueError: Cannot mask with non-boolean array containing NA / NaN values`; - **on the nullable `"string"` dtype** drops the missing rows from both the filter and its negation, because indexing treats `pd.NA` in a boolean mask as `False` and `~pd.NA` stays `pd.NA`. The `str` dtype result is the subtle one in review. A mask and its negation partition the rows, so `df[~mask]` ("routes that do not touch JFK") quietly includes every row with **no route at all**. ## Making the filter explicit and portable 1. **Pass `na=` yourself.** `na=False` drops missing rows from the match, `na=True` keeps them. The intent is then visible in the code and identical on `str`, object and `"string"` columns. In pandas 3.0 `na` must be a bool or a missing-value scalar; other values raise `ValueError`. 2. **Decide about the negation separately.** If "not JFK" should exclude unknown routes too, write `~route.str.contains("jfk", case=False, na=True)` or combine with `route.notna()`. 3. **Use `case=False`** rather than lower-casing the column first; it is a keyword of `str.contains` and works with or without regular expressions. 4. **Set `regex=False` for literal text.** `regex=True` is the default, so `"."`, `"+"` or `"("` in the pattern mean regex syntax: `str.contains("A+B")` matches `"AB"` but not `"A+B"`. 5. **Check the dtype** with `df.dtypes` when a filter behaves differently between environments; a column read with `dtype=object`, or built from mixed values, does not get the `str` dtype. ## Diagnosing the error on older code - The traceback names `Cannot mask with non-boolean array containing NA / NaN values`: the mask is object dtype with missing entries. - Calling `dropna()` on the mask is not a fix. The shorter mask no longer covers every label of the frame, and `df[...]` raises `IndexingError: Unalignable boolean Series provided as indexer`. - The fix is `na=False` (or `na=True`) in the `str.contains` call, or `mask.fillna(False)` when the mask is built elsewhere. ## A test that pins the behaviour Because the outcome hinges on dtype and version, a small test is worth more than a code comment: 1. Build a frame with one matching route, one non-matching route and one missing route. 2. Assert the row labels of `df[mask]` and of `df[~mask]` separately, so the destination of the missing row is fixed in writing. 3. Run the same assertions with the column cast to object and to `"string"` if the pipeline can receive either. 4. Keep `na=` explicit in the code under test, so the test documents a choice rather than a default. ## Why pandas 3.0 behaves differently In pandas 3.0, text columns infer the new `str` dtype instead of object. That dtype can hold only strings or missing values, uses `NaN` as its missing value, and its boolean string methods return NumPy `bool` arrays with `False` at missing entries. That makes the filter safe to run, but it also turns a former crash into a silent decision about missing data, which is why spelling out `na=` is the habit worth keeping.

  • Why does df[~mask] keep the rows with no route on the default str dtype?
    Because the mask holds `False` at those rows, and `~` turns `False` into `True`. The `str` dtype records "missing" as "does not match", so the negated filter treats unknown routes as "not JFK". Pass `na=True` before negating, or add `& route.notna()`, if unknown routes should be excluded.
  • The pattern is 'JFK+LAX'; why does it miss the route 'JFK+LAX'?
    `regex=True` is the default, so `+` means "one or more of the previous character" and the pattern looks for `JF`, then one or more `K`, then `LAX`, with no plus sign. Pass `regex=False` to match literal text, or escape the pattern with `re.escape`.
  • How does the result change if the column was cast to the nullable "string" dtype?
    The mask becomes the nullable `boolean` dtype with `<NA>` at missing rows. Indexing treats `<NA>` as `False`, and `~<NA>` stays `<NA>`, so missing rows drop from both `df[mask]` and `df[~mask]`. That logic is the nullable dtypes' three-valued rule, covered with NA-aware dtypes.

  • Assuming str.contains returns NaN at missing rows on every pandas 3.0 text column
  • Assuming a mask and its negation always split the rows the same way on every dtype
  • Fixing the non-boolean mask error by calling dropna on the mask
  • Treating the pattern as literal text although regex=True is the default
  • Lower-casing the whole column first because str.contains has no case option

Assumes pandas 3.0: text columns infer the str dtype, whose str.contains fills missing rows with False; in 2.x text was object dtype and the mask held a missing value, which raises when used to filter.