skip to content

Boolean Masks

Rows are filtered with boolean Series combined by &, | and ~, plus isin and between, while where and mask keep the shape. Interviewers check parentheses and the and-versus-& error.

on this pageshow

questions

5

In pandas, how do you keep only the DataFrame rows that satisfy two conditions at once, and why does each comparison need its own parentheses?

level: juniorimportance: must knowfreq 80%

answer

  1. element-wise operators, not keywords
  2. operator precedence in Python
  3. bitwise operators bind before comparisons
  4. one boolean Series per condition

basics

~20 s

Build one boolean Series per condition and combine them with & (and), | (or) and ~ (not), wrapping each comparison in parentheses: Python binds & tighter than > or ==, so an unparenthesised filter parses wrongly and raises.

solid answer

~40 s

Each comparison such as `df['dep_delay'] > 15` returns a boolean `Series` aligned to the frame's index. I combine them with `&`, `|` and `~`, never `and`/`or`/`not`, and pass the result to `df[...]` or `df.loc[mask, cols]`. The parentheses are not style: Python gives `&` higher precedence than `>`, so `df['dep_delay'] > 15 & df['arr_delay'] > 30` becomes `df['dep_delay'] > (15 & df['arr_delay']) > 30`, a chained comparison that raises the ambiguous-truth `ValueError` (or a `TypeError` when `&` meets a text column). Writing `(a > 15) & (b > 30)`, or using `.gt()`, fixes it. The filtered frame keeps the original row labels, so I call `reset_index(drop=True)` only if I need 0..n-1.

code

python · 23 lines
python
import pandas as pd

flights = pd.DataFrame(
    {
        "carrier": ["AA", "DL", "UA", "AA", "WN"],
        "dep_delay": [5, 42, 18, 75, 0],
        "arr_delay": [2, 50, 10, 80, -3],
    },
    index=[101, 102, 103, 104, 105],
)

late = flights["dep_delay"] > 15
long_arrival = flights["arr_delay"] > 30

print(flights[late & long_arrival])           # rows 102 and 104, labels kept
print(flights[late | (flights["carrier"] == "WN")].index.tolist())
print(flights[~late].index.tolist())          # [101, 105]
print(late.sum(), late.mean())                # 3 0.6

try:
    flights[flights["dep_delay"] > 15 & flights["arr_delay"] > 30]
except ValueError as err:
    print(err)  # The truth value of a Series is ambiguous. ...

go deeper

for a junior

Recall the three operators, &, | and ~, and write every comparison inside its own parentheses. Know that df[mask] keeps the matching rows with all their columns.

for a middle

Explain the precedence rule: & is evaluated before >, so the unparenthesised filter turns into a chained comparison. Mention .gt() and friends, loc[mask, cols], and that the original labels survive.

for a senior

Show you debug filters by naming and inspecting masks, checking their sum before trusting a result, and noticing when ~ is applied to integers or a mask comes from a differently indexed frame.

for a principal

Frame boolean filters as team readability: named masks and method comparisons in shared code make reviews catch logic slips that dense one-line filters hide.

## What a boolean mask is in pandas A **boolean mask** is a `Series` of `True`/`False` values that shares the frame's **index** (its row labels). You build one by comparing a column to a value: `df['dep_delay'] > 15` returns a `bool`-dtype `Series` with one entry per row. Passing that Series to `df[...]` or `df.loc[...]` keeps the rows whose entry is `True` and drops the rest. - **One mask per condition.** Each comparison (`>`, `<`, `==`, `!=`, `>=`, `<=`) or method such as `isin` produces its own mask. - **Masks combine element by element.** Row 102 of the combined mask is computed only from row 102 of each input mask. - **Masks align on labels.** A mask built from the same frame shares its index, so the rows line up automatically. ## The three operators: `&`, `|` and `~` pandas overloads Python's **bitwise** operators to mean element-wise logic on boolean Series: | Operator | Meaning | Example | |---|---|---| | `&` | and: both conditions hold | `(df['dep_delay'] > 15) & (df['arr_delay'] > 30)` | | `\|` | or: at least one holds | `(df['carrier'] == 'AA') \| (df['carrier'] == 'DL')` | | `~` | not: flips every value | `~(df['dep_delay'] > 15)` | The keywords `and`, `or` and `not` do not work: Python asks the whole Series for one truth value, and pandas refuses with `ValueError: The truth value of a Series is ambiguous`. ## Why every comparison needs its own parentheses Python's operator precedence puts `&`, `|` and `^` **above** the comparison operators. So pandas never gets a chance to see what you meant: 1. You write `df['dep_delay'] > 15 & df['arr_delay'] > 30`. 2. Python first evaluates `15 & df['arr_delay']`, a bitwise and between an integer and a column. 3. What remains is the chained comparison `df['dep_delay'] > (15 & df['arr_delay']) > 30`, which Python expands into two comparisons joined by `and`. 4. That hidden `and` asks a Series for a single truth value, so pandas raises the ambiguous-truth `ValueError`. The failure mode depends on the column types. With two `int64` columns you get the `ValueError` above. When the column next to `&` holds floats or text (a `float64` or `str` column), the bitwise step itself raises a `TypeError` first. Either way the fix is the same: wrap each comparison, `(a > 15) & (b > 30)`. The pandas user guide states this rule directly: the operators **must** be grouped with parentheses. The comparison **methods** sidestep precedence because a method call binds tightest of all: `df['dep_delay'].gt(15) & df['arr_delay'].gt(30)` needs no extra parentheses. `eq`, `ne`, `lt`, `le`, `gt` and `ge` all exist on `Series`. ## What `df[mask]` returns - **The same columns, fewer rows.** `df[mask]` with a boolean Series returns a `DataFrame` holding only the `True` rows, with every column and every dtype unchanged. - **The original labels survive.** If rows 102 and 104 match, the result's index is `[102, 104]`, not `[0, 1]`. Call `reset_index(drop=True)` if you need a fresh 0..n-1 index. - **Rows and columns in one step.** `df.loc[mask, ['carrier', 'dep_delay']]` filters rows and picks columns together. `iloc` accepts only a plain boolean array, so pass `mask.to_numpy()` there; a boolean Series makes `iloc` raise `ValueError`. - **Counting without filtering.** Because `True` counts as 1, `mask.sum()` is the number of matching rows and `mask.mean()` is the matching share. ## Habits that keep filters correct 1. **Name the masks.** `late = df['dep_delay'] > 15` and `long_arr = df['arr_delay'] > 30` make `df[late & long_arr]` readable and testable on its own. 2. **Negate with `~`, and only on booleans.** `~` on a `bool` Series flips each value; on an `int64` column of 0s and 1s it performs a bitwise not and yields `-1` and `-2`, which is not a mask at all. 3. **Check the mask's length and index.** With `df[...]`, a mask whose index differs from the frame's is reindexed to this frame's index with a `UserWarning`, and if some labels cannot be matched pandas raises `IndexingError: Unalignable boolean Series provided as indexer`. Building masks from the frame you filter avoids both. 4. **Keep the filter and any later write separate.** Selecting rows here is read-only; changing values for the masked rows is a different operation (covered with assigning values). ## Common mistakes - Writing `and`/`or` between masks because they read naturally. - Dropping the parentheses and blaming the data when a `ValueError` or `TypeError` appears. - Expecting the filtered frame to be renumbered from zero. - Using `~` on an integer flag column instead of converting it to `bool` first.

  • Why does the filtered frame keep labels such as 102 and 104 instead of starting again at 0?
    Boolean selection keeps the original index of every surviving row, which is what lets you join the result back to the source or compare it with other selections. If you need a clean 0..n-1 index, call `reset_index(drop=True)`; without `drop=True` the old labels become a column named `index`.
  • How do you count the rows a mask would select without building the filtered frame?
    Sum the mask: `True` counts as 1 and `False` as 0, so `late.sum()` is the number of matching rows and `late.mean()` is the matching share. `late.any()` and `late.all()` answer whether at least one or every row matches.
  • Can the same mask pick columns too, and does iloc accept it?
    `df.loc[mask, ['carrier', 'dep_delay']]` filters rows and selects columns in one call. `iloc` accepts only a plain boolean array, so a boolean Series there raises `ValueError`; pass `mask.to_numpy()` instead.

  • Writing df[df.a > 1 and df.b < 2] because and reads naturally
  • Treating the parentheses around each comparison as an optional style choice
  • Expecting the filtered frame to be renumbered from zero automatically
  • Negating a mask with not instead of the ~ operator
  • Applying ~ to a 0/1 integer column and expecting a boolean flip
open as a page

In pandas 3.0, what triggers 'ValueError: The truth value of a Series is ambiguous', and how do you fix each trigger?

level: middleimportance: must knowfreq 65%

basics

~20 s

Anything that asks a whole Series for one bool raises it: and, or, not, if, a chained comparison like 10 < s < 50, or comparisons joined by & without parentheses. Use &, |, ~ element-wise, or reduce with .any(), .all(), .empty or .item().

open as a page

In pandas, how do Series.isin and Series.between build row masks, and which of their edge cases catch analysts out?

level: middleimportance: should knowfreq 60%

basics

~20 s

isin(values) is True where an element equals one of a list-like's values; between(left, right) is True where left <= value <= right, endpoints included by default. A bare string in isin raises TypeError, and type mismatches silently match nothing.

open as a page

In pandas, how do DataFrame.where and DataFrame.mask differ from filtering with df[mask], and what happens to an int64 column?

level: middleimportance: should knowfreq 45%

basics

~20 s

df[mask] drops non-matching rows; where and mask keep the full shape and index, replacing values instead. where keeps values where the condition is True, mask replaces them; without other, int64 becomes float64 to hold NaN.

open as a page

In pandas 3.0, a filter on df['route'].str.contains('jfk', case=False) meets rows with no route; what does the mask hold there, and how does that differ from 2.x?

level: seniorimportance: should knowfreq 45%

basics

~20 s

In pandas 3.0 text infers the str dtype, so str.contains returns a bool mask with False at missing rows: the filter drops them and its negation keeps them. In 2.x's object dtype the mask held a missing value and filtering raised ValueError.

open as a page