skip to content

Selectors and Setters

Selecting by label with loc, by position with iloc, by condition with boolean masks, and writing values back in one step. Interviewers expect the right accessor without guessing.

on this pageshow

explore

questions

17

In pandas, how do you keep only the DataFrame rows that satisfy two conditions at once, and why does each comparison need its own parentheses?

level: juniorimportance: must knowfreq 80%

answer

  1. element-wise operators, not keywords
  2. operator precedence in Python
  3. bitwise operators bind before comparisons
  4. one boolean Series per condition

basics

~20 s

Build one boolean Series per condition and combine them with & (and), | (or) and ~ (not), wrapping each comparison in parentheses: Python binds & tighter than > or ==, so an unparenthesised filter parses wrongly and raises.

solid answer

~40 s

Each comparison such as `df['dep_delay'] > 15` returns a boolean `Series` aligned to the frame's index. I combine them with `&`, `|` and `~`, never `and`/`or`/`not`, and pass the result to `df[...]` or `df.loc[mask, cols]`. The parentheses are not style: Python gives `&` higher precedence than `>`, so `df['dep_delay'] > 15 & df['arr_delay'] > 30` becomes `df['dep_delay'] > (15 & df['arr_delay']) > 30`, a chained comparison that raises the ambiguous-truth `ValueError` (or a `TypeError` when `&` meets a text column). Writing `(a > 15) & (b > 30)`, or using `.gt()`, fixes it. The filtered frame keeps the original row labels, so I call `reset_index(drop=True)` only if I need 0..n-1.

code

python · 23 lines
python
import pandas as pd

flights = pd.DataFrame(
    {
        "carrier": ["AA", "DL", "UA", "AA", "WN"],
        "dep_delay": [5, 42, 18, 75, 0],
        "arr_delay": [2, 50, 10, 80, -3],
    },
    index=[101, 102, 103, 104, 105],
)

late = flights["dep_delay"] > 15
long_arrival = flights["arr_delay"] > 30

print(flights[late & long_arrival])           # rows 102 and 104, labels kept
print(flights[late | (flights["carrier"] == "WN")].index.tolist())
print(flights[~late].index.tolist())          # [101, 105]
print(late.sum(), late.mean())                # 3 0.6

try:
    flights[flights["dep_delay"] > 15 & flights["arr_delay"] > 30]
except ValueError as err:
    print(err)  # The truth value of a Series is ambiguous. ...

go deeper

for a junior

Recall the three operators, &, | and ~, and write every comparison inside its own parentheses. Know that df[mask] keeps the matching rows with all their columns.

for a middle

Explain the precedence rule: & is evaluated before >, so the unparenthesised filter turns into a chained comparison. Mention .gt() and friends, loc[mask, cols], and that the original labels survive.

for a senior

Show you debug filters by naming and inspecting masks, checking their sum before trusting a result, and noticing when ~ is applied to integers or a mask comes from a differently indexed frame.

for a principal

Frame boolean filters as team readability: named masks and method comparisons in shared code make reviews catch logic slips that dense one-line filters hide.

## What a boolean mask is in pandas A **boolean mask** is a `Series` of `True`/`False` values that shares the frame's **index** (its row labels). You build one by comparing a column to a value: `df['dep_delay'] > 15` returns a `bool`-dtype `Series` with one entry per row. Passing that Series to `df[...]` or `df.loc[...]` keeps the rows whose entry is `True` and drops the rest. - **One mask per condition.** Each comparison (`>`, `<`, `==`, `!=`, `>=`, `<=`) or method such as `isin` produces its own mask. - **Masks combine element by element.** Row 102 of the combined mask is computed only from row 102 of each input mask. - **Masks align on labels.** A mask built from the same frame shares its index, so the rows line up automatically. ## The three operators: `&`, `|` and `~` pandas overloads Python's **bitwise** operators to mean element-wise logic on boolean Series: | Operator | Meaning | Example | |---|---|---| | `&` | and: both conditions hold | `(df['dep_delay'] > 15) & (df['arr_delay'] > 30)` | | `\|` | or: at least one holds | `(df['carrier'] == 'AA') \| (df['carrier'] == 'DL')` | | `~` | not: flips every value | `~(df['dep_delay'] > 15)` | The keywords `and`, `or` and `not` do not work: Python asks the whole Series for one truth value, and pandas refuses with `ValueError: The truth value of a Series is ambiguous`. ## Why every comparison needs its own parentheses Python's operator precedence puts `&`, `|` and `^` **above** the comparison operators. So pandas never gets a chance to see what you meant: 1. You write `df['dep_delay'] > 15 & df['arr_delay'] > 30`. 2. Python first evaluates `15 & df['arr_delay']`, a bitwise and between an integer and a column. 3. What remains is the chained comparison `df['dep_delay'] > (15 & df['arr_delay']) > 30`, which Python expands into two comparisons joined by `and`. 4. That hidden `and` asks a Series for a single truth value, so pandas raises the ambiguous-truth `ValueError`. The failure mode depends on the column types. With two `int64` columns you get the `ValueError` above. When the column next to `&` holds floats or text (a `float64` or `str` column), the bitwise step itself raises a `TypeError` first. Either way the fix is the same: wrap each comparison, `(a > 15) & (b > 30)`. The pandas user guide states this rule directly: the operators **must** be grouped with parentheses. The comparison **methods** sidestep precedence because a method call binds tightest of all: `df['dep_delay'].gt(15) & df['arr_delay'].gt(30)` needs no extra parentheses. `eq`, `ne`, `lt`, `le`, `gt` and `ge` all exist on `Series`. ## What `df[mask]` returns - **The same columns, fewer rows.** `df[mask]` with a boolean Series returns a `DataFrame` holding only the `True` rows, with every column and every dtype unchanged. - **The original labels survive.** If rows 102 and 104 match, the result's index is `[102, 104]`, not `[0, 1]`. Call `reset_index(drop=True)` if you need a fresh 0..n-1 index. - **Rows and columns in one step.** `df.loc[mask, ['carrier', 'dep_delay']]` filters rows and picks columns together. `iloc` accepts only a plain boolean array, so pass `mask.to_numpy()` there; a boolean Series makes `iloc` raise `ValueError`. - **Counting without filtering.** Because `True` counts as 1, `mask.sum()` is the number of matching rows and `mask.mean()` is the matching share. ## Habits that keep filters correct 1. **Name the masks.** `late = df['dep_delay'] > 15` and `long_arr = df['arr_delay'] > 30` make `df[late & long_arr]` readable and testable on its own. 2. **Negate with `~`, and only on booleans.** `~` on a `bool` Series flips each value; on an `int64` column of 0s and 1s it performs a bitwise not and yields `-1` and `-2`, which is not a mask at all. 3. **Check the mask's length and index.** With `df[...]`, a mask whose index differs from the frame's is reindexed to this frame's index with a `UserWarning`, and if some labels cannot be matched pandas raises `IndexingError: Unalignable boolean Series provided as indexer`. Building masks from the frame you filter avoids both. 4. **Keep the filter and any later write separate.** Selecting rows here is read-only; changing values for the masked rows is a different operation (covered with assigning values). ## Common mistakes - Writing `and`/`or` between masks because they read naturally. - Dropping the parentheses and blaming the data when a `ValueError` or `TypeError` appears. - Expecting the filtered frame to be renumbered from zero. - Using `~` on an integer flag column instead of converting it to `bool` first.

  • Why does the filtered frame keep labels such as 102 and 104 instead of starting again at 0?
    Boolean selection keeps the original index of every surviving row, which is what lets you join the result back to the source or compare it with other selections. If you need a clean 0..n-1 index, call `reset_index(drop=True)`; without `drop=True` the old labels become a column named `index`.
  • How do you count the rows a mask would select without building the filtered frame?
    Sum the mask: `True` counts as 1 and `False` as 0, so `late.sum()` is the number of matching rows and `late.mean()` is the matching share. `late.any()` and `late.all()` answer whether at least one or every row matches.
  • Can the same mask pick columns too, and does iloc accept it?
    `df.loc[mask, ['carrier', 'dep_delay']]` filters rows and selects columns in one call. `iloc` accepts only a plain boolean array, so a boolean Series there raises `ValueError`; pass `mask.to_numpy()` instead.

  • Writing df[df.a > 1 and df.b < 2] because and reads naturally
  • Treating the parentheses around each comparison as an optional style choice
  • Expecting the filtered frame to be renumbered from zero automatically
  • Negating a mask with not instead of the ~ operator
  • Applying ~ to a 0/1 integer column and expecting a boolean flip
open as a page

In pandas, what do emp['salary'], emp[['salary']] and emp[0:2] return, and why does emp[0] raise KeyError?

level: juniorimportance: must knowfreq 70%

basics

~20 s

On a DataFrame, [] with one name returns that column as a Series, a list returns a DataFrame, and an integer slice returns rows by position. emp[0] raises KeyError because a scalar key looks for a column named 0.

open as a page

In pandas, how do df.loc and df.iloc differ on a frame indexed by non-sequential employee IDs, and what does each slice include?

level: juniorimportance: must knowfreq 85%

basics

~20 s

df.loc selects by index label and its slices include both endpoints; df.iloc selects by 0-based position and its slices exclude the stop. On an ID index, loc[1007] is employee 1007 and iloc[1] is the second row.

open as a page

In pandas, how do you set a column's value only on the rows that match a condition, such as flagging suspicious payments?

level: juniorimportance: must knowfreq 75%

basics

~20 s

Use one loc call with the boolean mask as the row selector and the column label as the column selector: df.loc[df['amount'] > 5000, 'is_fraud'] = True. Initialise a new column first so its dtype stays clean.

open as a page

In pandas 3.0, what triggers 'ValueError: The truth value of a Series is ambiguous', and how do you fix each trigger?

level: middleimportance: must knowfreq 65%

basics

~20 s

Anything that asks a whole Series for one bool raises it: and, or, not, if, a chained comparison like 10 < s < 50, or comparisons joined by & without parentheses. Use &, |, ~ element-wise, or reduce with .any(), .all(), .empty or .item().

open as a page

In pandas 3.0, why does `df.loc[mask, 'fee'] = 2.5` raise TypeError on an int64 fee column, and how do you fix it?

level: seniorimportance: must knowfreq 55%

basics

~10 s

In pandas 3.0 an in-place write whose value the column's dtype cannot hold raises TypeError instead of upcasting the column. Cast first, for example df['fee'] = df['fee'].astype('float64'), then repeat the loc write.

open as a page

In pandas, how do Series.isin and Series.between build row masks, and which of their edge cases catch analysts out?

level: middleimportance: should knowfreq 60%

basics

~20 s

isin(values) is True where an element equals one of a list-like's values; between(left, right) is True where left <= value <= right, endpoints included by default. A bare string in isin raises TypeError, and type mismatches silently match nothing.

open as a page

In pandas, how do DataFrame.where and DataFrame.mask differ from filtering with df[mask], and what happens to an int64 column?

level: middleimportance: should knowfreq 45%

basics

~20 s

df[mask] drops non-matching rows; where and mask keep the full shape and index, replacing values instead. where keeps values where the condition is True, mask replaces them; without other, int64 becomes float64 to hold NaN.

open as a page

In pandas, when should you read one cell with DataFrame.at or DataFrame.iat instead of loc and iloc, and what do they reject?

level: middleimportance: should knowfreq 40%

basics

~20 s

Use at (labels) or iat (positions) when you want exactly one scalar: they skip the key-type dispatch loc and iloc perform. They reject list-like keys, iat rejects non-integers, and at raises KeyError for a missing label.

open as a page

In pandas, why does emp.loc[[1007, 9999]] raise KeyError when only 9999 is missing, and how do you keep the IDs that exist?

level: middleimportance: should knowfreq 50%

basics

~20 s

Since pandas 1.0, loc with a list raises KeyError if any label is absent rather than returning NaN rows. Keep the present IDs with emp.loc[emp.index.intersection(ids)], or use emp.reindex(ids) when you want NaN rows for the missing ones.

open as a page

In pandas, what happens when `df.loc` assigns to a row or column label that does not exist yet, and why can't `iloc` do that?

level: middleimportance: should knowfreq 45%

basics

~20 s

Writing to a missing label through loc enlarges the frame: a new row or column is added, unfilled cells become NaN and int64 columns can widen to float64. iloc and iat are positional, so a position past the end raises IndexError.

open as a page

In pandas, how does `DataFrame.update` write corrected values from another frame, and what does it leave untouched?

level: middleimportance: should knowfreq 30%

basics

~20 s

DataFrame.update aligns the other frame on index and column labels and overwrites the caller's matching cells with the other frame's non-NA values, in place, returning None. NaN in the other frame never overwrites, and unmatched rows or columns are ignored.

open as a page

In pandas 3.0, a filter on df['route'].str.contains('jfk', case=False) meets rows with no route; what does the mask hold there, and how does that differ from 2.x?

level: seniorimportance: should knowfreq 45%

basics

~20 s

In pandas 3.0 text infers the str dtype, so str.contains returns a bool mask with False at missing rows: the filter drops them and its negation keeps them. In 2.x's object dtype the mask held a missing value and filtering raised ValueError.

open as a page

In pandas, a sales frame indexed by (region, quarter) raises UnsortedIndexError on sales.loc[pd.IndexSlice[:, 'Q1':'Q2'], :]; why, and how do you select quarters across regions?

level: seniorimportance: should knowfreq 30%

basics

~20 s

A range slice inside a MultiIndex level needs the index lexsorted at least to that level; set_index on unsorted rows is not, so pandas raises UnsortedIndexError. Call sort_index() first, or use xs('Q2', level='quarter') or a list of quarters.

open as a page

In pandas 3.0, pay = emp.set_index('name')['salary'] makes pay[0] raise KeyError where pandas 2.x returned the first salary; what changed, and what still behaves positionally?

level: seniorimportance: should knowfreq 50%

basics

~20 s

In pandas 3.0, Series [] and []= treat an integer key as a label, never a position, so pay[0] looks for a name 0 and raises KeyError. Use pay.iloc[0]. Integer slices such as pay[0:2] stay positional.

open as a page

A pandas script writes a sorted, capped fee Series back with `df.iloc`, and fees land on the wrong payments, while `df.loc` puts them right; why?

level: seniorimportance: should knowfreq 40%

basics

~20 s

A .loc write aligns a right-hand Series on its index labels first, so each fee returns to its own payment. iloc ignores labels and writes values in their current order, so a re-sorted Series lands on the wrong rows.

open as a page

In pandas 3.0, what does `fees.clip(upper=50.0, inplace=True)` return, and which in-place methods changed their return value?

level: middleimportance: nice to knowfreq 25%

basics

~10 s

In pandas 3.0 it returns fees itself, already clipped. replace, fillna, ffill, bfill, interpolate, where, mask and clip now return self with inplace=True; drop, rename, sort_values, set_index and dropna still return None.

open as a page