skip to content

In pandas 3.0, why does `customers[customers['balance'] < 0]['status'] = 'review'` leave `customers` unchanged, and what is ChainedAssignmentError?

level: seniorimportance: must knowfreq 60%

answer

  1. two indexing steps, one statement
  2. the temporary is thrown away
  3. a warning, not an exception
  4. reference counts, named subsets

basics

~20 s

The first indexing step returns a temporary that behaves as a copy; the value lands in it and is discarded. pandas 3.0 flags this with ChainedAssignmentError, a warning, not an exception; assign in one loc step instead.

solid answer

~40 s

The line runs as two operations: `customers[mask]` builds a temporary DataFrame, then `['status'] = 'review'` writes into that temporary, which is thrown away. In pandas 3.0, under Copy-on-Write, an indexing result always behaves as a copy, so chained assignment never updates the original. pandas detects the pattern through the temporary's reference count and emits `ChainedAssignmentError`, which is a `Warning` subclass, so the code keeps running. The same warning covers inplace methods on a column, like `customers['status'].fillna('active', inplace=True)`. The fix is one statement targeting the frame, such as `customers.loc[mask, 'status'] = 'review'`. Detection misses writes into a named subset, which change only that subset.

code

python · 18 lines
python
import warnings

import pandas as pd
from pandas.errors import ChainedAssignmentError

customers = pd.DataFrame(
    {"balance": [120.0, -40.0, 15.0], "status": ["active", "active", "active"]}
)

with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    customers[customers["balance"] < 0]["status"] = "review"   # two steps

print(customers["status"].tolist())   # ['active', 'active', 'active']
print(any(w.category is ChainedAssignmentError for w in caught))   # True

customers.loc[customers["balance"] < 0, "status"] = "review"   # one step
print(customers["status"].tolist())   # ['active', 'review', 'active']

go deeper

for a junior

Recall that two indexing steps in one assignment write into a throwaway object in pandas 3.0, and that one loc assignment is the fix.

for a middle

Explain getitem then setitem on a temporary, and that ChainedAssignmentError is a Warning subclass, so the program keeps running with unchanged data.

for a senior

Show you know the detection's blind spots, named subsets and PyPy, and how you would turn the warning into a test failure during an upgrade.

for a principal

Treat the warning as a migration gate: decide whether CI fails on it and how to find the silent named-subset writes it cannot see.

## What the statement really does Chained assignment means updating a frame through **two indexing steps in one statement**: ```python customers[customers["balance"] < 0]["status"] = "review" ``` Python runs this as two operations. First a **getitem**, `customers[customers["balance"] < 0]`, builds a temporary DataFrame holding the matching rows. Then a **setitem**, `temp["status"] = "review"`, writes into that temporary. The temporary is never bound to a name, so it is discarded at the end of the line, and with it the only object that received the new value. In pandas 3.0 every indexing result behaves as a copy under Copy-on-Write, so there is no case in which the temporary writes through to `customers`. The release notes say chained assignment "will stop working", and the user guide says it will "consistently never work". ## What ChainedAssignmentError is Because the statement silently does nothing, pandas tries to detect it and emits **`pandas.errors.ChainedAssignmentError`**. Despite the name, it is a **`Warning` subclass**, not an exception: - the statement completes and the program keeps running, so it is easy to scroll past in a notebook; - its message says the value is being set on a copy "through chained assignment", that this "never works to update the original", and suggests a single-step `.loc[row_indexer, col_indexer] = value` instead; - a second message covers **inplace methods** called on a column, such as `customers["status"].fillna("active", inplace=True)`, and suggests `customers.fillna({"status": "active"}, inplace=True)` or `customers["status"] = customers["status"].fillna("active")`. The single-step `loc` assignment is the fix; its forms are covered in the lesson on assigning values. ## How it differed in pandas 2.x | Statement | pandas 2.x | pandas 3.0 | |---|---|---| | `df[mask]["b"] = v` | did not update `df`; the classic `SettingWithCopyWarning` case | does not update `df`; `ChainedAssignmentError` | | `df["b"][mask] = v` | often updated `df` | does not update `df`; `ChainedAssignmentError` | | `df["b"].fillna(0, inplace=True)` | often updated `df` | does not update `df`; `ChainedAssignmentError` | | `df.loc[mask, "b"] = v` | updates `df` | updates `df` | The mask-first form was already broken in 2.x, because a boolean selection builds new arrays. The column-first forms are the dangerous ones in a migration: code that worked in 2.x stops updating anything in 3.0. `SettingWithCopyWarning` itself is removed in pandas 3.0, because there is no longer an ambiguous case to warn about. ## The limits of detection pandas spots chained assignment by looking at the **reference count** of the object being written into: a temporary created inside the same statement has fewer references than a named object. That design has blind spots worth knowing: 1. **Named intermediates are not chained assignment.** `negatives = customers[mask]` followed by `negatives["status"] = "review"` is a legitimate write to `negatives`. No warning fires, and `customers` is unchanged. If the author expected `customers` to change, the bug is silent. 2. **The check is switched off on PyPy**, where reference counts do not work the same way. 3. **It is a warning**, so the default warnings filter can show it once per location, and logging setups can swallow it. A test suite can turn it into a failure with a warnings filter whose action is `"error"` for `ChainedAssignmentError`, which makes the column-first 2.x forms impossible to merge. ## Diagnosing a dead assignment When a cleaning step "runs but changes nothing" after an upgrade: - look for two `[]`, `loc` or `iloc` steps on the left of one `=`; - look for `inplace=True` on a method called on `df[col]`; - look for a named subset that is modified and then never assigned back. Rewrite each as one statement whose target is the frame you want to change. ## A worked migration case A 2.x cleaning notebook flags customers in debt with `customers["status"][customers["balance"] < 0] = "review"`. On 2.x the flag appeared, so downstream reports filtered on it. On pandas 3.0 the same cell prints a `ChainedAssignmentError` warning and the status column stays `"active"` everywhere; the report downstream quietly shows no customers to review. Nothing crashed, which is exactly why reading the warning matters. The one-line fix is `customers.loc[customers["balance"] < 0, "status"] = "review"`, and the same cell then behaves identically on 2.x and 3.0, which makes it safe to commit before the upgrade lands.

  • Why does `negatives = customers[mask]` followed by `negatives['status'] = 'review'` raise no warning?
    Because it is not chained assignment: `negatives` is a named object and writing into it is legitimate, so it changes. pandas only flags writes into a temporary created in the same statement, which it detects from the reference count. `customers` stays unchanged, so if the author meant to update it, the bug is silent and only a test or a data diff will catch it.
  • How do you rewrite `customers['status'].replace('active', 'open', inplace=True)` for pandas 3.0?
    Make the frame the target. Either call the method on the frame with a per-column mapping, `customers.replace({'status': {'active': 'open'}}, inplace=True)`, or drop `inplace` and assign the result back: `customers['status'] = customers['status'].replace('active', 'open')`. The original line only changes a temporary column and emits ChainedAssignmentError.
  • How can a test suite make sure no chained assignment slips through?
    Turn the warning into a failure, for example with a warnings filter whose action is 'error' for `pandas.errors.ChainedAssignmentError` in the test configuration. Then any two-step write on an unnamed temporary fails the test. It will not catch named-subset writes or code that only runs on PyPy, where the check is disabled.

  • ChainedAssignmentError is an exception, so the bad line stops the program
  • df['b'][mask] = v still works in 3.0 because the column comes first
  • SettingWithCopyWarning still marks chained writes in pandas 3.0
  • If no warning appears, the subset write must have reached the original frame
  • Calling .copy() on the temporary makes the chained write update the frame

Assumes pandas 3.0; in 2.x the column-first form often updated the frame and SettingWithCopyWarning flagged risky cases, in 3.0 chained assignment never updates and warns with ChainedAssignmentError.