skip to content

A long pandas method chain over trip data runs without error but returns wrong columns; how do you inspect each step without splitting it up?

level: seniorimportance: should knowfreq 42%

answer

  1. a step that returns its input
  2. pipe with a print or assert
  3. rename's default errors setting
  4. set_axis labels by position
  5. comment out lines bottom-up

basics

~10 s

Insert DataFrame.pipe steps that print or assert on the intermediate frame and return it unchanged, bisect the chain line by line, and harden silent steps such as rename, whose errors argument defaults to 'ignore'.

solid answer

~30 s

I put the chain in parentheses with one method per line, then add `.pipe(peek, "after rename")` where `peek` prints `df.shape` and `df.columns` and returns `df` unchanged, so the chain keeps flowing. Commenting out lines from the bottom, or cutting it in half, locates the step. Then I look for the silent defaults: `rename` ignores keys that are not there because `errors` defaults to `"ignore"`, `set_axis` relabels by position, an outer `trips[...]` inside `assign` reads the original frame, and a missing `@` in `query` can match a column. Useful checks stay as permanent `pipe(check_...)` steps.

code

python · 23 lines
python
import pandas as pd

trips = pd.DataFrame({
    "city": ["Lyon", "Oslo", "Pune"],
    "fare": [12.0, 30.0, 5.0],
    "km": [3.0, 12.0, 4.0],
})

def peek(df, label):
    print(f"[{label}] shape={df.shape} cols={list(df.columns)}")
    return df

out = (
    trips
    .pipe(peek, "raw")
    .rename(columns={"kms": "distance_km"})
    .pipe(peek, "after rename")
)

try:
    trips.rename(columns={"kms": "distance_km"}, errors="raise")
except KeyError as exc:
    print("KeyError:", exc)

go deeper

for a junior

Recall that pipe calls your function with the frame and continues with whatever it returns.

for a middle

Write a peek helper that returns the frame, and explain how to bisect a parenthesised chain.

for a senior

Name the silent pandas defaults that make chains go wrong without errors, rename's errors='ignore' first, and harden them.

for a principal

Decide which checks become permanent pipe steps in team pipelines, so contracts fail loudly at the step that breaks them.

## Why chains are hard to debug A long method chain is one Python expression. It has no intermediate variables to print, and a traceback points at the call that failed, not at what the frame looked like when it got there. Worse, the commonest chain bugs **do not raise at all**: the chain finishes and the result is quietly wrong. The job is to make each step observable without rewriting the pipeline into a dozen named frames. ## Silent failures specific to chain steps These pandas behaviours produce no error: - **`rename` ignores unknown keys by default.** Its `errors` argument defaults to `"ignore"`, so `rename(columns={"kms": "distance_km"})` on a frame whose column is `km` does nothing. Passing `errors="raise"` turns the typo into `KeyError: "['kms'] not found in axis"`. - **`set_axis` relabels by position.** `.set_axis(["city", "fare", "tip", "km"], axis=1)` checks only the count (a wrong count raises `ValueError: Length mismatch`). If an earlier step reordered the columns, the labels land on the wrong data and nothing complains. - **A stale outer reference in `assign`.** `trips["fare"]` instead of `lambda d: d["fare"]` reads the original frame and aligns by label. - **A missing `@` in `query`** on a name that is also a column compares the column with itself. ## Making steps visible with `pipe` `DataFrame.pipe(func, *args, **kwargs)` calls `func(frame, *args, **kwargs)` and returns whatever `func` returns. A function that prints or checks something and then **returns the frame unchanged** can be dropped between any two steps: ```python def peek(df, label): print(f"[{label}] shape={df.shape} cols={list(df.columns)}") return df ``` Useful variants: 1. **`peek`** prints shape, columns, dtypes or `head()` after a step. 2. **`check`** asserts an expectation and raises if it fails, such as no duplicate trip IDs, no negative fares, or an expected column set. 3. **A breakpoint step**, a function that calls `breakpoint()` and returns `df`, stops in the debugger with the intermediate frame in hand. 4. **Snapshotting**: a function that stores `df` in a dict or a list for later comparison. The key detail is the `return df`: a helper that returns `None`, such as `df.info()` used directly, would end the chain. `pipe` also accepts a `(callable, data_keyword)` tuple for functions that take the frame under another argument name, so third-party functions fit into the chain the same way. Keep checkpoints cheap and harmless: - print `df.shape`, `list(df.columns)` or `df.dtypes`, not the whole frame, when the table is large; - never modify `df` inside a checkpoint; it must return its input exactly as received; - in production code, log instead of printing, and remove pure print helpers before merging. ## A procedure for a wrong result 1. **Format the chain** inside parentheses, one method per line, so lines can be commented out from the bottom up. 2. **Bisect**: cut the chain at its midpoint (assign that partial result to a variable in a notebook) and check it; repeat on the half that is wrong. 3. **Insert `pipe(peek, ...)`** around the suspect step to compare shape and columns before and after. 4. **Tighten the silent steps**: `errors="raise"` on `rename`, a `check` after `set_axis`, lambdas or `pd.col` (pandas 3.0) instead of outer references, `@` on every variable in `query`. 5. **Keep the useful checks** as permanent `pipe(check_...)` steps; delete the `peek` calls. | Symptom | Likely step | Confirm with | |---|---|---| | a later step raises `KeyError` for a column you renamed | `rename` key typo, silently ignored | `errors="raise"` | | columns have plausible names but swapped values | `set_axis` after a reorder | `peek` before and after | | row counts right, numbers wrong for some rows | outer `trips[...]` inside `assign` | replace with a lambda and compare | | `query` returns zero rows | missing `@` on a name shared with a column | read the string against `df.columns` | ## What an interviewer listens for They want to hear that you do not break a pipeline apart by reflex, that you know `pipe` takes a function returning the frame, and that you can name the silent pandas defaults, `rename`'s `errors="ignore"` above all, rather than only printing everything. Turning a debugging aid into a permanent validation step is the production-minded finish.

  • Why can't you use .pipe(lambda d: d.info()) as a checkpoint in a pandas chain?
    `pipe` returns whatever the function returns, and `DataFrame.info()` prints and returns `None`, so the next link is called on `None` and fails. A checkpoint must return its input: a named helper that calls `d.info()` and then `return d`, or `lambda d: (print(d.shape), d)[1]`, though the named helper reads better in review.
  • When would you turn a debugging pipe step into a permanent one?
    When it checks a contract the rest of the pipeline relies on: an expected column set after renames, no negative fares, unique trip IDs before a join. A `pipe(check_...)` that raises a clear error at that point is cheaper than finding a wrong summary later. Pure print helpers come out before merging.

  • A chain must be split into named variables before it can be debugged
  • rename raises KeyError by default when a key is not a column
  • set_axis matches new labels to columns by name
  • pipe ignores the function's return value and passes the frame on
  • If a chain runs without error, every step did what it was meant to