In pandas 3.0, assign keywords built from lambdas in a loop all give the last column's values; why, and how does pd.col fix it?
answer
- when the closure looks up c
- assign calls callables later
- an expression object, not a function
- assign, loc and [] only
basics
~20 sEach lambda looks up the loop variable only when DataFrame.assign calls it, after the loop ends, so all read the last column. In pandas 3.0, pd.col(c) records the column name when the expression is built.
solid answer
~30 sIt is Python late binding: `lambda d: d[c] * rate` stores no value of `c`; it reads `c` when called, and `assign` calls the lambdas only after the comprehension finished, so every one sees the last name. The old fix is a default argument, `lambda d, c=c: ...`. In pandas 3.0, `pd.col(c) * rate` builds an expression object during the loop, so the name and the rate are captured right there and evaluated later against the intermediate frame. `pd.col` works in `assign`, `loc` and `[]`, not in `query`, and a wrong name raises `ValueError` listing the columns.
code
python · 11 linesimport pandas as pd
trips = pd.DataFrame({"fare": [12.0, 30.0], "tip": [2.0, 0.0]})
rate = 1.1
late = {f"{c}_eur": (lambda d: d[c] * rate) for c in ["fare", "tip"]}
bound = {f"{c}_eur": pd.col(c) * rate for c in ["fare", "tip"]}
print(trips.assign(**late))
print(trips.assign(**bound))
print(pd.col("fare") * rate)go deeper
Know that pd.col('fare') in pandas 3.0 can replace lambda d: d['fare'] inside assign.
Explain that assign calls callables later, and that pd.col builds an expression evaluated against the frame at that step.
Diagnose the loop-built lambda bug as Python late binding, fix it with a default argument or pd.col, and state where pd.col is accepted in 3.0.
Weigh adopting pd.col as house style against teams still on 2.x and against the methods that do not accept it yet.
## The bug: lambdas built in a loop A pipeline converts several money columns of a trips table to another currency. Writing one `assign` keyword per column by hand is tedious, so someone builds the keywords in a loop: ```python rate = 1.1 kwargs = {f"{c}_eur": lambda d: d[c] * rate for c in ["fare", "tip"]} trips.assign(**kwargs) ``` Both new columns come out equal to `tip * 1.1`. Nothing is wrong with `assign`; the cause is a rule of **Python closures**: a lambda does not copy the value of `c` when it is created, it looks `c` up **when it is called**. `assign` calls the lambdas only after the comprehension has finished, when `c` is `"tip"` for every one of them. This is **late binding**, and it bites in pandas specifically because `assign` defers evaluation of callables by design. The traditional fixes are Python-level: - bind the current value as a default argument: `lambda d, c=c: d[c] * rate`; - build each callable in a small factory function that takes `c` as a parameter; - give up on the loop and write each keyword explicitly. ## `pd.col` in pandas 3.0 In **pandas 3.0**, `pd.col(name)` returns an **expression object** (`pandas.api.typing.Expression`) that stands for "the column called `name` of whatever frame this is evaluated against". Operators and Series methods on it build a larger expression instead of computing anything: ```python kwargs = {f"{c}_eur": pd.col(c) * rate for c in ["fare", "tip"]} trips.assign(**kwargs) ``` This version is correct because `pd.col(c)` is **called during the loop**: the column name is stored inside the expression at that moment, and `* rate` stores the value of `rate` at that moment too. When `assign` later evaluates each expression against the intermediate frame, there is no free loop variable left to look up. What `pd.col` supports in 3.0, from the release notes and `pandas/core/col.py`: | Capability | Example | |---|---| | arithmetic and comparison operators | `pd.col("fare") + pd.col("tip")`, `pd.col("km") > 3` | | Series methods and accessors | `pd.col("city").str.upper()`, `pd.col("fare").sum()` | | NumPy ufuncs | `np.log(pd.col("fare"))` | | a readable repr | `pd.col("fare") * 1.1` prints as `col('fare') * 1.1` | ## Where it can be used, and where it cannot The 3.0 release notes say `pd.col()` works anywhere that accepts a callable taking the calling DataFrame and returning a Series: **`DataFrame.assign`, `DataFrame.loc`, and `[]` for getting and setting**. So these are valid: 1. `trips.assign(total=pd.col("fare") + pd.col("tip"))` 2. `trips.loc[pd.col("fare") > 6]` 3. `trips[pd.col("km") > 3]` It is **not** accepted by `query`, which takes only a string: `trips.query(pd.col("fare") > 6)` raises `ValueError: expr must be a string to be evaluated`. The release notes say support is expected to grow in later releases, so check before using it elsewhere. ## Errors look different - A misspelt column in `pd.col("fair")` raises **`ValueError`**, not `KeyError`, with a message listing the frame's columns: `Column 'fair' not found in given DataFrame` plus a hint. - An expression has no truth value; using one in `if` or `and` raises `TypeError: boolean value of an expression is ambiguous`. Combine conditions with `&` and `|`. - Expressions cannot be iterated or copied. ## Migrating a chain to `pd.col` - Lambdas still work in pandas 3.0; nothing forces a rewrite. - Code that uses `pd.col` fails on 2.x with `AttributeError`, because the function does not exist there, so pin the pandas version before adopting it. - Inside one `assign`, expressions are evaluated in order against the frame being built, exactly like lambdas, so `assign(total=pd.col("fare") + pd.col("tip"), per_km=pd.col("total") / pd.col("km"))` works. - A `try/except KeyError` around a lambda-based step will not catch a missing-column error from `pd.col`, which raises `ValueError`. ## Choosing between a lambda and `pd.col` - Prefer `pd.col` for **column arithmetic, comparisons and accessor calls**: shorter, readable in a repr, immune to late binding. - Keep a **lambda** when the step needs something an expression does not offer, such as calling a function that takes the whole frame, or when the code must still run on pandas 2.x, where `pd.col` does not exist. - Either way, never read the outer frame's variable inside the chain. An interviewer asking this wants three things: the Python cause (late binding, not a pandas bug), the 3.0 tool that removes it and why it does, and the boundary of that tool (assign, loc and `[]` yes; query no).
- Can you pass pd.col to DataFrame.query in pandas 3.0?No. `query` accepts only a string and raises `ValueError: expr must be a string to be evaluated` for an expression object. In 3.0, `pd.col` is accepted by `assign`, `loc` and `[]`; to filter with one, write `trips.loc[pd.col("fare") > 6]`. The release notes say support is expected to widen in later releases, so a 3.0 codebase should not assume it.
- What does pandas 3.0 raise for pd.col('fair') when the column is called 'fare'?A `ValueError`, not the `KeyError` that `d["fair"]` would raise, with the message `Column 'fair' not found in given DataFrame` and a hint listing the actual column names. Code that catches `KeyError` around a lambda version needs updating when it moves to `pd.col`.
- assign evaluates every lambda immediately when the dict is built
- pd.col is a shorter spelling of a lambda with the same late binding
- pd.col works anywhere a string expression works, including query
- A missing pd.col column raises KeyError like df['name'] does
- pd.col has been available since pandas 2.0
Assumes pandas 3.0, which added pd.col; on 2.x the default-argument lambda is the only fix.