A pandas boolean consent column holds pd.NA for unknown patients; why do those rows appear in neither df[df['consent']] nor df[~df['consent']]?
answer
- unknown stays unknown under negation
- a mask cannot keep a maybe
- three-valued and, or
- True or unknown is still True
- give the unknowns their own bucket
basics
~20 sIn the nullable boolean dtype, ~ maps pd.NA to pd.NA, and indexing with a boolean-dtype mask treats pd.NA as False. Unknown-consent rows therefore drop out of both selections; route them explicitly with isna() or by filling the mask.
solid answer
~40 sA `boolean` column follows **Kleene three-valued logic**: `~pd.NA` is `pd.NA`, `pd.NA | True` is `True`, `pd.NA & False` is `False`, and every other combination with `pd.NA` stays `pd.NA`. Comparisons propagate it too, so `visits > 4` on an `Int64` column is `<NA>` where visits is missing. When such a mask selects rows, pandas treats `pd.NA` as False, so the unknown rows vanish from `df[consent]` and from `df[~consent]` without an error. If you build the mask with `map({'yes': True, 'no': False})` instead, you get an `object` column holding `NaN`, and indexing raises `ValueError: Cannot mask with non-boolean array containing NA / NaN values`. The fix is a deliberate decision: select `consent.isna()` as its own bucket, or fill the mask with the value the rule requires.
code
python · 20 linesimport pandas as pd
df = pd.DataFrame({
"patient": ["p01", "p02", "p03", "p04"],
"consent": ["yes", "no", "unknown", "yes"],
"visits": pd.array([5, 2, 6, None], dtype="Int64"),
})
mapped = df["consent"].map({"yes": True, "no": False})
print(mapped.dtype) # object: True, False, NaN, True
# df[mapped] raises ValueError: Cannot mask with non-boolean
# array containing NA / NaN values
consent = mapped.astype("boolean") # True, False, <NA>, True
print(df[consent]["patient"].tolist()) # ['p01', 'p04']
print(df[~consent]["patient"].tolist()) # ['p02']; p03 is in neither
print(df[consent.isna()]["patient"].tolist()) # ['p03']
busy = df["visits"] > 4 # True, False, True, <NA>
print(consent | busy) # True, False, True, True
print(consent & busy) # True, False, <NA>, <NA>go deeper
Recall that pd.NA in a boolean filter means unknown, that df[mask] drops those rows silently, and that isna() is how you select them.
Explain the Kleene table for & and |, why ~ leaves pd.NA unchanged, and how an object mask with NaN differs from a boolean mask with pd.NA.
Diagnose vanished rows from the split halves not summing to the table, then make the unknown case an explicit, reviewed decision with isna() or a deliberate fill.
Set a team rule for tri-state flags: whether unknown is modelled with the boolean dtype and pd.NA or as a separate category, and how filters must state their treatment of it.
## The setup: consent that can be unknown A clinic-visits table records consent as `yes`, `no` or `unknown`. Mapping that to True and False leaves the third answer with nowhere to go, and pandas offers two very different places to put it: - `df['consent'].map({'yes': True, 'no': False})` sends `unknown` to `NaN`. Because a NumPy `bool` array cannot hold `NaN`, the result is an **`object`** column of `True`, `False` and `NaN`. - Adding `.astype('boolean')` turns that into the nullable **`boolean`** dtype: a values array plus a mask, with the unknown cell read back as **`pd.NA`**. The two behave differently the moment they become a filter. ## Kleene logic in the boolean dtype `pd.NA` means "a value exists but we do not know it". pandas' `boolean` dtype implements **Kleene three-valued logic** for `&`, `|` and `^`: the result is `pd.NA` only when the unknown value could change the answer. | Expression | Result | Why | |---|---|---| | `True \| pd.NA` | `True` | true whichever value the NA hides | | `False \| pd.NA` | `pd.NA` | depends on the hidden value | | `False & pd.NA` | `False` | false whichever value the NA hides | | `True & pd.NA` | `pd.NA` | depends on the hidden value | | `pd.NA ^ True` | `pd.NA` | exclusive-or always depends on it | | `~pd.NA` | `pd.NA` | the negation of an unknown is unknown | Comparisons propagate as well: `pd.NA == pd.NA` is `pd.NA`, and on an `Int64` column `df['visits'] > 4` returns `<NA>` wherever visits is missing. That is the sharpest difference from NumPy floats, where `NaN > 4` is plain `False`. ## What a mask does with pd.NA A filter must keep or drop each row; it cannot keep a row "maybe". pandas resolves this at indexing time: 1. With a **`boolean`-dtype mask**, `df[mask]`, `s[mask]` and `df.loc[mask]` treat `pd.NA` as **False**. The row is dropped silently. 2. With an **`object` mask** holding `NaN`, pandas refuses: `ValueError: Cannot mask with non-boolean array containing NA / NaN values`. 3. In a **Python `if`**, `bool(pd.NA)` raises `TypeError: boolean value of NA is ambiguous`, which surfaces inside `apply` callbacks and loops. Point 1 explains the puzzle. `df[consent]` keeps only the True rows. `~consent` turns True into False and False into True but leaves `pd.NA` as `pd.NA`, which is again treated as False. The unknown patients are in neither half, and the two halves no longer add up to the table. Compare a NumPy-backed condition: for a `float64` column with `NaN`, `x > 4` is `False` in the gap, so `~(x > 4)` is `True` and the missing rows land in the complement instead. Neither result is "correct"; both are a decision pandas made for you. ## Combining conditions Kleene logic decides what survives a combined filter. With `consent` as above and `busy = df['visits'] > 4`: - `consent | busy` is True for an unknown-consent patient with 6 visits, because `pd.NA | True` is True, so that row is kept; - `consent & busy` is `pd.NA` for the same patient, so that row is dropped; - a consenting patient whose visit count is missing gives `True & pd.NA`, which is `pd.NA`, and is dropped from the `&` filter too. An object column holding `NaN` does not follow these rules. pandas' logical operators on it treat the `NaN` position as False in the output, even for `| True`, so results quietly differ from the `boolean` dtype. ## Fixing it on purpose The cure is to stop letting the indexer decide what unknown means: - **Give unknowns their own bucket.** `df[consent.isna()]` selects exactly the unknown rows, so the three selections cover the table. - **Fill the mask with the business answer.** `consent.fillna(False)` treats unknown as no consent, the conservative choice for clinical data; `fillna(True)` keeps them. Filling in general is the lesson on filling and dropping. - **Never test `== pd.NA`.** It returns `<NA>` everywhere and selects nothing. - **Check the dtype before filtering.** An `object` mask is a sign the `astype('boolean')` step is missing. Operator precedence (parentheses around each comparison) and the ambiguous truth value of a whole Series belong to the lesson on boolean masks. ## Interview angle This is a diagnosis question: rows have disappeared and nothing raised. A strong candidate names the propagation through `~`, the NA-as-False rule at indexing, the Kleene table, and an explicit fix.
- How would you keep the unknown-consent rows in df[mask] instead of dropping them?Fill the mask before indexing, for example `df[consent.fillna(True)]`, so the indexer sees a definite True. That turns "unknown" into a business decision written in the code. For consent data the conservative opposite, `fillna(False)`, is usually right, with the unknown rows reported separately through `consent.isna()`.
- Why does if row['consent']: inside an apply callback crash on some rows?For a `boolean` column the missing cell comes back as `pd.NA`, and `bool(pd.NA)` raises `TypeError: boolean value of NA is ambiguous`, because the value could be True or False. Test with `pd.isna(value)` first, or better, replace the row-wise `if` with a vectorised mask that has been filled deliberately.
- Does the same NA-as-False rule apply when assigning through df.loc[mask, col] = value?Yes. Both reading and setting with `.loc` convert a `boolean`-dtype mask to a plain NumPy boolean array with `pd.NA` mapped to False, so unknown rows are neither selected nor updated. The assignment succeeds quietly and leaves those rows unchanged.
A bouncer with a list marked yes, no and unknown: told to admit the yes names, he turns unknowns away; told to admit the no names, he turns the unknowns away again, because he only acts on a definite answer.
- ~mask turns every NA into True, so the complement catches the unknowns.
- pd.NA | True is NA, because anything combined with NA is NA.
- df[mask] raises an error whenever the boolean mask contains pd.NA.
- df[df['consent'] == pd.NA] selects the unknown-consent rows.
- NaN in an object mask is treated as False, just like pd.NA.