skip to content

Method Chaining

A chain of assign, pipe, query and rename builds a result without temporaries, and 3.0 adds pd.col expressions. Interviewers ask how to read, debug and trust a long chain.

on this pageshow

questions

6

In a pandas method chain, why should DataFrame.assign receive a lambda rather than a column taken from the original frame?

level: middleimportance: must knowfreq 58%

answer

  1. which frame does the expression see
  2. callable vs already-computed Series
  3. new columns missing on the original
  4. alignment by index label

basics

~20 s

DataFrame.assign calls a lambda with the intermediate frame at that step, while trips['fare'] reads the original frame: columns created earlier in the chain are missing (KeyError), changed values are stale, and relabelled rows silently misalign.

solid answer

~40 s

`assign` treats a callable and a non-callable differently: a `lambda d: ...` is called with the frame as it is at that point in the chain, while a Series such as `trips["fare"]` was computed from the original variable before the chain ran. So a column added two steps earlier raises `KeyError`, a column whose values an earlier step changed is read in its old state, and after `reset_index` or another relabel the stale Series is aligned by index label onto the wrong rows with no error at all. A lambda avoids all three, and later keywords in the same `assign` can read columns created by earlier ones.

code

python · 21 lines
python
import pandas as pd

trips = pd.DataFrame({
    "city": ["Lyon", "Oslo", "Lyon", "Pune"],
    "fare": [12.0, 30.0, 8.0, 5.0],
    "tip": [2.0, 0.0, 1.0, 0.5],
    "km": [3.0, 12.0, 2.0, 4.0],
})

stale = (
    trips.query("tip < 2")
    .reset_index(drop=True)
    .assign(total=trips["fare"] + trips["tip"])
)
fresh = (
    trips.query("tip < 2")
    .reset_index(drop=True)
    .assign(total=lambda d: d["fare"] + d["tip"])
)
print(stale)
print(fresh)

go deeper

for a junior

Remember that assign returns a new frame and that a lambda in assign receives the frame at that step of the chain.

for a middle

Explain the callable versus non-callable rule and show the three failures: KeyError for a new column, stale values, and label misalignment after reset_index.

for a senior

Show you have caught the silent case in review: a stale outer reference that only goes wrong once someone adds a reset_index or a value change upstream.

for a principal

Treat it as a review rule for the team: no outer frame names inside assign in a chain, so correctness never depends on index labels happening to line up.

## What `DataFrame.assign` does `DataFrame.assign(**kwargs)` returns a **new DataFrame** holding every existing column plus the ones named by its keyword arguments; a keyword that matches an existing column overwrites it in the result. The calling frame is never modified, which is what makes `assign` usable inside a **method chain**: a single parenthesised expression in which each method is called on the result of the previous one, so no temporary variables are needed. Each keyword value can be one of two things: - a **callable** (usually a `lambda`) that pandas calls with the frame being built at that step, using whatever it returns as the column; - a **non-callable** such as a Series, an array or a scalar, which pandas assigns as it is. That distinction decides whether a chain is correct. ## The stale-frame bug Consider a ride-sharing trips table with `city`, `fare`, `tip` and `km`. Inside a chain, the frame at step three is not the frame the chain started from. Writing `trips["fare"]` inside a later `assign` reaches **back to the original variable**, not to the intermediate result. Three things can go wrong: 1. **A column created earlier in the chain does not exist on the original.** `trips.assign(total=trips["fare"] + trips["tip"]).assign(per_km=trips["total"] / trips["km"])` raises `KeyError: 'total'`, because `trips` never had that column. 2. **Values changed earlier in the chain are ignored.** If an earlier step doubled `fare`, a later `trips["fare"]` still reads the old values, and the result mixes two versions of the data without any warning. 3. **Rows filtered or relabelled earlier misalign.** A Series passed to `assign` is aligned to the intermediate frame's **index labels**. After `query(...)` alone, the surviving labels still match and the numbers happen to be right. After `.reset_index(drop=True)` or a sort that is followed by a relabel, label 0 now means a different trip, and the stale Series puts another trip's total on each row. The third case is the dangerous one: no exception, plausible numbers, wrong answer. ## The fix: let pandas hand you the current frame A callable receives the frame **as it is at that step**: ```python summary = ( trips .query("tip < 2") .reset_index(drop=True) .assign( total=lambda d: d["fare"] + d["tip"], per_km=lambda d: d["total"] / d["km"], ) ) ``` Two details of `assign` make this work: - callables are **evaluated in order**, and a later keyword in the same call can read a column created by an earlier one (`per_km` reads `total` above); - the callable must not modify its input frame; the docstring says so explicitly and adds that pandas does not check it. The same "callable receives the current object" contract is accepted by `loc[...]`, `[]` and `pipe`, so a filter written as `.loc[lambda d: d["fare"] > 6]` is also chain-safe. | Value passed to `assign` | Evaluated against | Safe after earlier steps change rows or columns? | |---|---|---| | `lambda d: d["fare"] * 2` | the intermediate frame | yes | | `trips["fare"] * 2` | the original frame, then aligned by label | no | | scalar such as `"EUR"` | broadcast to every row | yes | | plain list or NumPy array | assigned by position, length must match | only if the length is still right | ## The same mistake in other clothes The outer-reference bug is not limited to `trips["fare"]`. Watch for: - `trips.fare` attribute access, which is the same Series from the same original frame; - `trips["fare"].to_numpy()` or `.values`, which is worse: an array has no labels, so it is placed **by position** and raises `ValueError` only if the length no longer matches; - a Series taken from a **different frame**, such as a lookup table, whose index happens to share some labels with the trips table; - a helper function that reads a module-level frame instead of the frame it was passed. In each case the fix is the same: read columns from the frame the step receives. ## Reading such a chain in review - Every `assign` value that reads a column should be a callable; an outer variable name inside a chain is a red flag. - Names in the lambda parameter (`d`, `df_`) should differ from the outer frame's name, so a reviewer can see at a glance which frame is read. - A chain should be wrapped in parentheses with one method per line, so each step can be commented out while debugging. ## Why interviewers ask it The question separates someone who has copied chained code from someone who has debugged it. The expected answer names the mechanism (callables get the current frame, non-callables are taken as they are), names the two silent failures (stale values and label misalignment) and the loud one (`KeyError` for a new column), and shows the lambda form. Someone who adds that later keywords in one `assign` can see earlier ones shows they have read the method's contract rather than guessed it.

  • Can a later keyword in the same DataFrame.assign call use a column created by an earlier keyword?
    Yes. `assign` evaluates its keyword arguments in order against the frame it is building, so `assign(total=lambda d: d["fare"] + d["tip"], per_km=lambda d: d["total"] / d["km"])` works. That only helps with callables: a non-callable like `trips["total"]` is evaluated by Python before `assign` is even called, so it fails with `KeyError`.
  • When does passing trips['fare'] inside a chain happen to give the right numbers?
    When every earlier step kept the original index labels and did not change that column, for example a plain `query` filter: the Series is aligned by label, so surviving rows match and dropped rows are ignored. That is luck, not design; adding a `reset_index` or a value change upstream later breaks it silently, which is why reviewers ask for the lambda everywhere.

  • Inside a chain, trips['col'] always refers to the current intermediate frame
  • A Series passed to assign is placed by position, never aligned by label
  • assign modifies the calling DataFrame in place and returns None
  • A wrong column reference in a chain always raises an error
  • Later keywords in one assign cannot see columns made by earlier ones
open as a page

Why does calling rename(..., inplace=True) in the middle of a pandas method chain break the chain?

level: juniorimportance: should knowfreq 52%

basics

~10 s

DataFrame.rename with inplace=True modifies the frame and returns None, so the next method is called on None and raises AttributeError; in a chain, let every method return a new frame instead.

open as a page

In pandas DataFrame.query, how do you compare a column with a Python variable and refer to a column named 'trip km'?

level: middleimportance: should knowfreq 50%

basics

~20 s

Inside a DataFrame.query string, prefix a Python variable with @ (fare > @min_fare) and wrap a column name that is not an identifier in backticks (trip km > 3); a bare name always means a column.

open as a page

A long pandas method chain over trip data runs without error but returns wrong columns; how do you inspect each step without splitting it up?

level: seniorimportance: should knowfreq 42%

basics

~10 s

Insert DataFrame.pipe steps that print or assert on the intermediate frame and return it unchanged, bisect the chain line by line, and harden silent steps such as rename, whose errors argument defaults to 'ignore'.

open as a page

In pandas 3.0, assign keywords built from lambdas in a loop all give the last column's values; why, and how does pd.col fix it?

level: seniorimportance: should knowfreq 38%

basics

~20 s

Each lambda looks up the loop variable only when DataFrame.assign calls it, after the loop ends, so all read the last column. In pandas 3.0, pd.col(c) records the column name when the expression is built.

open as a page

As the lead of a pandas team, how would you set the rule for long method chains versus named intermediate frames, and how is it reviewed?

level: principalimportance: should knowfreq 32%

basics

~20 s

Chain the linear steps of one pandas stage, break into named frames where a result has its own meaning or is reused, move complex steps into tested pipe functions, and enforce no inplace and lambdas or pd.col in review.

open as a page