In pandas, how do you keep only the DataFrame rows that satisfy two conditions at once, and why does each comparison need its own parentheses?
answer
- element-wise operators, not keywords
- operator precedence in Python
- bitwise operators bind before comparisons
- one boolean Series per condition
basics
~20 sBuild one boolean Series per condition and combine them with & (and), | (or) and ~ (not), wrapping each comparison in parentheses: Python binds & tighter than > or ==, so an unparenthesised filter parses wrongly and raises.
solid answer
~40 sEach comparison such as `df['dep_delay'] > 15` returns a boolean `Series` aligned to the frame's index. I combine them with `&`, `|` and `~`, never `and`/`or`/`not`, and pass the result to `df[...]` or `df.loc[mask, cols]`. The parentheses are not style: Python gives `&` higher precedence than `>`, so `df['dep_delay'] > 15 & df['arr_delay'] > 30` becomes `df['dep_delay'] > (15 & df['arr_delay']) > 30`, a chained comparison that raises the ambiguous-truth `ValueError` (or a `TypeError` when `&` meets a text column). Writing `(a > 15) & (b > 30)`, or using `.gt()`, fixes it. The filtered frame keeps the original row labels, so I call `reset_index(drop=True)` only if I need 0..n-1.
code
python · 23 linesimport pandas as pd
flights = pd.DataFrame(
{
"carrier": ["AA", "DL", "UA", "AA", "WN"],
"dep_delay": [5, 42, 18, 75, 0],
"arr_delay": [2, 50, 10, 80, -3],
},
index=[101, 102, 103, 104, 105],
)
late = flights["dep_delay"] > 15
long_arrival = flights["arr_delay"] > 30
print(flights[late & long_arrival]) # rows 102 and 104, labels kept
print(flights[late | (flights["carrier"] == "WN")].index.tolist())
print(flights[~late].index.tolist()) # [101, 105]
print(late.sum(), late.mean()) # 3 0.6
try:
flights[flights["dep_delay"] > 15 & flights["arr_delay"] > 30]
except ValueError as err:
print(err) # The truth value of a Series is ambiguous. ...go deeper
Recall the three operators, &, | and ~, and write every comparison inside its own parentheses. Know that df[mask] keeps the matching rows with all their columns.
Explain the precedence rule: & is evaluated before >, so the unparenthesised filter turns into a chained comparison. Mention .gt() and friends, loc[mask, cols], and that the original labels survive.
Show you debug filters by naming and inspecting masks, checking their sum before trusting a result, and noticing when ~ is applied to integers or a mask comes from a differently indexed frame.
Frame boolean filters as team readability: named masks and method comparisons in shared code make reviews catch logic slips that dense one-line filters hide.
## What a boolean mask is in pandas A **boolean mask** is a `Series` of `True`/`False` values that shares the frame's **index** (its row labels). You build one by comparing a column to a value: `df['dep_delay'] > 15` returns a `bool`-dtype `Series` with one entry per row. Passing that Series to `df[...]` or `df.loc[...]` keeps the rows whose entry is `True` and drops the rest. - **One mask per condition.** Each comparison (`>`, `<`, `==`, `!=`, `>=`, `<=`) or method such as `isin` produces its own mask. - **Masks combine element by element.** Row 102 of the combined mask is computed only from row 102 of each input mask. - **Masks align on labels.** A mask built from the same frame shares its index, so the rows line up automatically. ## The three operators: `&`, `|` and `~` pandas overloads Python's **bitwise** operators to mean element-wise logic on boolean Series: | Operator | Meaning | Example | |---|---|---| | `&` | and: both conditions hold | `(df['dep_delay'] > 15) & (df['arr_delay'] > 30)` | | `\|` | or: at least one holds | `(df['carrier'] == 'AA') \| (df['carrier'] == 'DL')` | | `~` | not: flips every value | `~(df['dep_delay'] > 15)` | The keywords `and`, `or` and `not` do not work: Python asks the whole Series for one truth value, and pandas refuses with `ValueError: The truth value of a Series is ambiguous`. ## Why every comparison needs its own parentheses Python's operator precedence puts `&`, `|` and `^` **above** the comparison operators. So pandas never gets a chance to see what you meant: 1. You write `df['dep_delay'] > 15 & df['arr_delay'] > 30`. 2. Python first evaluates `15 & df['arr_delay']`, a bitwise and between an integer and a column. 3. What remains is the chained comparison `df['dep_delay'] > (15 & df['arr_delay']) > 30`, which Python expands into two comparisons joined by `and`. 4. That hidden `and` asks a Series for a single truth value, so pandas raises the ambiguous-truth `ValueError`. The failure mode depends on the column types. With two `int64` columns you get the `ValueError` above. When the column next to `&` holds floats or text (a `float64` or `str` column), the bitwise step itself raises a `TypeError` first. Either way the fix is the same: wrap each comparison, `(a > 15) & (b > 30)`. The pandas user guide states this rule directly: the operators **must** be grouped with parentheses. The comparison **methods** sidestep precedence because a method call binds tightest of all: `df['dep_delay'].gt(15) & df['arr_delay'].gt(30)` needs no extra parentheses. `eq`, `ne`, `lt`, `le`, `gt` and `ge` all exist on `Series`. ## What `df[mask]` returns - **The same columns, fewer rows.** `df[mask]` with a boolean Series returns a `DataFrame` holding only the `True` rows, with every column and every dtype unchanged. - **The original labels survive.** If rows 102 and 104 match, the result's index is `[102, 104]`, not `[0, 1]`. Call `reset_index(drop=True)` if you need a fresh 0..n-1 index. - **Rows and columns in one step.** `df.loc[mask, ['carrier', 'dep_delay']]` filters rows and picks columns together. `iloc` accepts only a plain boolean array, so pass `mask.to_numpy()` there; a boolean Series makes `iloc` raise `ValueError`. - **Counting without filtering.** Because `True` counts as 1, `mask.sum()` is the number of matching rows and `mask.mean()` is the matching share. ## Habits that keep filters correct 1. **Name the masks.** `late = df['dep_delay'] > 15` and `long_arr = df['arr_delay'] > 30` make `df[late & long_arr]` readable and testable on its own. 2. **Negate with `~`, and only on booleans.** `~` on a `bool` Series flips each value; on an `int64` column of 0s and 1s it performs a bitwise not and yields `-1` and `-2`, which is not a mask at all. 3. **Check the mask's length and index.** With `df[...]`, a mask whose index differs from the frame's is reindexed to this frame's index with a `UserWarning`, and if some labels cannot be matched pandas raises `IndexingError: Unalignable boolean Series provided as indexer`. Building masks from the frame you filter avoids both. 4. **Keep the filter and any later write separate.** Selecting rows here is read-only; changing values for the masked rows is a different operation (covered with assigning values). ## Common mistakes - Writing `and`/`or` between masks because they read naturally. - Dropping the parentheses and blaming the data when a `ValueError` or `TypeError` appears. - Expecting the filtered frame to be renumbered from zero. - Using `~` on an integer flag column instead of converting it to `bool` first.
- Why does the filtered frame keep labels such as 102 and 104 instead of starting again at 0?Boolean selection keeps the original index of every surviving row, which is what lets you join the result back to the source or compare it with other selections. If you need a clean 0..n-1 index, call `reset_index(drop=True)`; without `drop=True` the old labels become a column named `index`.
- How do you count the rows a mask would select without building the filtered frame?Sum the mask: `True` counts as 1 and `False` as 0, so `late.sum()` is the number of matching rows and `late.mean()` is the matching share. `late.any()` and `late.all()` answer whether at least one or every row matches.
- Can the same mask pick columns too, and does iloc accept it?`df.loc[mask, ['carrier', 'dep_delay']]` filters rows and selects columns in one call. `iloc` accepts only a plain boolean array, so a boolean Series there raises `ValueError`; pass `mask.to_numpy()` instead.
- Writing df[df.a > 1 and df.b < 2] because and reads naturally
- Treating the parentheses around each comparison as an optional style choice
- Expecting the filtered frame to be renumbered from zero automatically
- Negating a mask with not instead of the ~ operator
- Applying ~ to a 0/1 integer column and expecting a boolean flip