skip to content

In pandas, how do Series.isin and Series.between build row masks, and which of their edge cases catch analysts out?

level: middleimportance: should knowfreq 60%

answer

  1. membership versus range
  2. list-like argument required
  3. exact equality, no type coercion
  4. inclusive defaults to both ends

basics

~20 s

isin(values) is True where an element equals one of a list-like's values; between(left, right) is True where left <= value <= right, endpoints included by default. A bare string in isin raises TypeError, and type mismatches silently match nothing.

solid answer

~40 s

`s.isin(values)` returns a boolean Series that is `True` where the element exactly equals one of `values`, which must be list-like: a list, set, array, `Index` or `Series`. A single string raises `TypeError`, and an `int64` column tested against `['1', '2']` silently returns all `False`. `NaN` in the list matches `NaN` in the column, and "not in" is `~s.isin(...)`. `s.between(left, right, inclusive="both")` is `(left <= s) & (s <= right)`; `inclusive` also takes `"neither"`, `"left"` and `"right"`. Missing values give `False`, and reversed bounds return all `False` without complaint. On a frame, `df.isin({col: values})` gives a boolean frame I reduce with `.all(axis=1)` or `.any(axis=1)`.

code

python · 26 lines
python
import numpy as np
import pandas as pd

flights = pd.DataFrame(
    {
        "carrier": ["AA", "DL", "UA", "B6"],
        "origin": ["JFK", "ATL", "ORD", "JFK"],
        "dep_delay": [5, 42, 18, 75],
    },
    index=[101, 102, 103, 104],
)

print(flights[flights["carrier"].isin({"AA", "UA"})].index.tolist())   # [101, 103]
print(flights[~flights["carrier"].isin(["AA", "DL"])].index.tolist())  # [103, 104]
print(flights["dep_delay"].between(15, 60).tolist())                 # [False, True, True, False]
print(flights["dep_delay"].between(15, 42, inclusive="left").tolist())  # [False, False, True, False]
print(pd.Series([1, 2, 3]).isin(["1", "2"]).tolist())               # all False
print(pd.Series([1.0, np.nan]).isin([np.nan]).tolist())              # [False, True]

rules = {"carrier": ["AA", "UA"], "origin": ["JFK", "ORD"]}
print(flights[flights[["carrier", "origin"]].isin(rules).all(axis=1)])

try:
    flights["carrier"].isin("AA")
except TypeError as err:
    print(err)

go deeper

for a junior

Recall that isin takes a list of allowed values and between takes two bounds, and that ~ in front of isin means not in.

for a middle

Explain the edges: list-like only, equality with no string-to-number conversion, NaN matching NaN, the four inclusive options, and reducing DataFrame.isin with all or any over axis=1.

for a senior

Show the habit of distrusting an all-False result: check dtypes and whitespace in lookup values, and pick half-open bands so boundary values are counted once.

for a principal

Point out that allow-lists living in code drift from reference data; a shared, typed lookup avoids silent misses when codes change type or format.

## Two shortcuts that return ordinary masks `Series.isin` and `Series.between` are pandas methods that build a boolean `Series` aligned to the original index. You use their result exactly like a comparison: pass it to `df[...]` or `df.loc[...]`, combine it with `&`, `|` and `~`, or reduce it with `.sum()` and `.any()`. | Method | Question it answers | Equivalent long form | |---|---|---| | `s.isin(values)` | Is the value one of these? | `(s == v1) \| (s == v2) \| ...` | | `s.between(left, right)` | Is the value inside this range? | `(left <= s) & (s <= right)` | Both read better than the long form, and `isin` stays a single vectorised call however long the list grows. ## `isin`: membership in a list-like - **Pass a list-like.** A `list`, `set`, `tuple`, NumPy array, `Index` or `Series` all work. A single string does not: `df['carrier'].isin('AA')` raises `TypeError: only list-like objects are allowed to be passed to isin(), you passed a str`. Wrap it: `isin(['AA'])`. - **Exact equality, no coercion.** Each element must equal a listed value. An `int64` column tested against `['1', '2']` returns all `False` with no error, because the integer 1 never equals the string `'1'`. The same silent miss happens with stray whitespace or different case in text. - **Missing values can match.** If the list contains `np.nan`, a `NaN` in the column matches it: `pd.Series([1.0, np.nan]).isin([np.nan])` gives `[False, True]`. - **"Not in" is `~`.** `df[~df['carrier'].isin(['AA', 'DL'])]` keeps the other carriers. `Series.isin` has no `invert=` argument (that one belongs to `numpy.isin`), and Python's `not in` raises the ambiguous-truth error. - **A Series argument is used for its values.** `Series.isin(other_series)` ignores `other_series`' index and tests membership in its values. ### `DataFrame.isin` for per-column rules `DataFrame.isin` returns a boolean **DataFrame** of the same shape. Passing a `dict` maps column names to allowed values; columns not named in the dict come back all `False`. Collapse the frame to a row mask with `.all(axis=1)` (every rule holds) or `.any(axis=1)` (at least one holds): ```python rules = {"carrier": ["AA", "UA"], "origin": ["JFK", "ORD"]} row_mask = flights[["carrier", "origin"]].isin(rules).all(axis=1) ``` Unlike `Series.isin`, a `DataFrame` passed to `DataFrame.isin` is matched on **both** index and column labels, so a lookup frame with a different index can produce `False` everywhere. ## `between`: a closed range by default `Series.between(left, right, inclusive="both")` returns `True` where `left <= value <= right`. The `inclusive` argument accepts four strings: | `inclusive=` | Test | |---|---| | `"both"` (default) | `left <= s <= right` | | `"neither"` | `left < s < right` | | `"left"` | `left <= s < right` | | `"right"` | `left < s <= right` | Points worth knowing: 1. **Endpoints are included by default.** `pd.Series([0, 15, 60, 90]).between(15, 60)` is `[False, True, True, False]`. 2. **Missing values are `False`**, because any comparison with `NaN` is `False`. 3. **Reversed bounds do not raise.** `between(60, 15)` simply returns all `False`, which can hide a swapped argument. 4. **It works on anything comparable**: numbers, datetimes (string bounds such as `'2026-01-01'` are parsed for a datetime column) and strings (lexicographic order). 5. **Half-open ranges** for bucketing consecutive bands use `inclusive="left"`, so a value on a boundary lands in exactly one band. ## `isin` on the index The same method exists on `Index` objects, which is useful when you hold a list of row labels and do not know which are present. With `s` indexed by `[2, 4, 9]`: - `s[s.index.isin([2, 4, 6])]` keeps only the labels that exist, here 2 and 4, and nothing else. - `s.reindex([2, 4, 6])` returns all three requested labels and fills the missing label 6 with `NaN`. So `index.isin` is the filter, while `reindex` is a reshaping step that can add rows you did not have. ## Putting them together on a delay table A typical filter on a flight table keeps a few carriers and a delay band: ```python mask = flights["carrier"].isin({"AA", "UA"}) & flights["dep_delay"].between(15, 60) flights[mask] ``` Each method returns a mask, so the parentheses rule for comparisons does not bite here: a method call already binds tighter than `&`. ## Mistakes that survive code review - `isin('AA')` with a bare string, expecting equality or substring matching. - Assuming `between` excludes its endpoints, then double-counting values that sit on a band boundary. - Testing an integer column against string codes and trusting an all-`False` result. - Chaining a dozen `==` comparisons with `|` where one `isin` would do.

  • Why can isin on an integer column return all False without raising?
    `isin` tests equality and does not turn strings into numbers, so the integer 1 is simply not equal to the string `'1'`. Nothing is wrong syntactically, so nothing raises. Check `s.dtype` and convert either the column or the lookup list, for example with `pd.to_numeric`, before trusting an empty match.
  • How do you split delays into bands so a value on a boundary lands in exactly one band?
    Use half-open ranges with `inclusive="left"`: `between(0, 15, inclusive="left")`, then `between(15, 60, inclusive="left")`. With the default `"both"`, a delay of exactly 15 is `True` in both bands and gets counted twice.
  • What changes when you pass a Series to Series.isin versus a DataFrame to DataFrame.isin?
    `Series.isin(other)` uses only `other`'s values; its index is ignored. `DataFrame.isin(other_frame)` matches on both index and column labels, so a lookup frame with different row labels can mark everything `False`. For a per-column allow-list, pass a dict instead.

  • Passing a single string to isin and expecting an equality or substring test
  • Assuming between excludes both endpoints by default
  • Expecting isin to raise when the list's values have the wrong type
  • Believing isin can never match missing values in the column
  • Reaching for an invert= argument on Series.isin to express not in