A pandas loan table assigns risk tiers with `df.apply(tier, axis=1)`; how do you replace that apply with `np.where` or `np.select`?
answer
- one Python call per row
- conditions as boolean Series
- two branches versus many
- first true condition wins
- NaN fails every comparison
basics
~20 sBuild each condition as a boolean Series over whole columns, then use np.where for two outcomes or np.select(conditions, choices, default=...) for several, where the first true condition wins. Handle NaN explicitly, since it fails every comparison.
solid answer
~40 s`df.apply(tier, axis=1)` builds a Series for every row and calls my function once per row, so it runs at Python speed. I rewrite the if/elif chain as boolean Series: `np.where(df["dti"] > 0.4, "manual", "auto")` for two branches, and `np.select(conditions, choices, default="standard")` for several, ordering the conditions the way the elifs were ordered because the first true one wins. Each condition is evaluated over the whole column at once. The trap is missing data: `NaN < 580` is False, so a row without a score falls to the default, so I put a `df["score"].isna()` condition first. `Series.case_when` is the pandas-native version of the same pattern.
code
python · 25 linesimport numpy as np
import pandas as pd
loans = pd.DataFrame({
"score": [540, 610, 680, 790, np.nan],
"dti": [0.55, 0.42, 0.30, 0.10, 0.20],
})
def tier(row):
if row["score"] < 580:
return "high"
if row["score"] < 670:
return "elevated"
return "standard"
loans["tier_apply"] = loans.apply(tier, axis=1) # one call per row
loans["review"] = np.where(loans["dti"] > 0.4, "manual", "auto")
conditions = [loans["score"].isna(), loans["score"] < 580, loans["score"] < 670]
choices = ["unscored", "high", "elevated"]
loans["tier"] = np.select(conditions, choices, default="standard")
print(loans) # row 4: tier_apply says standard, tier says unscoredgo deeper
Recall that np.where picks between two values with a boolean Series and that apply with axis=1 calls your function once per row.
Explain np.select's ordered conditions and explicit default, how conditions combine with & and |, and why the result is assigned by position.
Spot the NaN fall-through to the default tier, guard it with an isna condition first, and verify tier counts after the rewrite.
Treat tier thresholds as business rules: keep them in one reviewed definition shared by scoring and reporting, so a rewrite for speed cannot change who lands in which tier.
## Why `apply(axis=1)` is the thing to replace `DataFrame.apply(func, axis=1)` passes **each row** to `func` as a Series whose index is the column names, and collects the results. For a tiering rule such as "score below 580 is high risk, below 670 elevated, otherwise standard" that means one Python function call, one row Series and one if/elif walk **per row**. The pandas user guide is direct about it: most tasks can and should be done with built-in methods or vectorized operations, and `apply` is for when no vectorized alternative exists. The library-neutral lesson on vectorization covers why a per-row loop is slow in general. The pandas question is narrower: which calls replace the `apply`, and what changes in the result. ## Step 1: turn each branch into a boolean Series Every `if` test becomes an expression over a whole column: - `df["score"] < 580` is a boolean Series, one value per row. - Combine tests with `&`, `|` and `~`, each side in parentheses, never with `and` / `or`. - The tests run once over the column, not once per row. ## Step 2: pick the selector | Rule shape | Call | Result | |---|---|---| | Two outcomes | `np.where(cond, a, b)` | a NumPy array, `a` where true, `b` elsewhere | | Several ordered outcomes | `np.select(condlist, choicelist, default=...)` | a NumPy array, first matching choice per row | | Several outcomes, pandas-native | `s.case_when([(cond, value), ...])` | a Series; rows matching nothing keep the caller's value | | Numeric ranges only | `pd.cut(col, bins, labels=...)` | an ordered categorical | `np.select` evaluates the condition list **in order** and takes the choice of the **first** true condition, exactly like an if/elif chain. Put the narrowest test first: if `score < 670` comes before `score < 580`, the high-risk rows are swallowed by the elevated tier. Pass `default` explicitly. Its own default is `0`, and with string choices a recent NumPy refuses to find a common dtype and raises `TypeError`. The arrays `np.where` and `np.select` return have no index. Assigning one with `df["tier"] = ...` places it **by position**, which is correct here because the conditions were built from the same frame. `Series.case_when` (added in pandas 2.2) keeps the index instead, and its first matching condition also wins. ## Step 3: decide what missing data means Any comparison with NaN is False. A row with no credit score fails `score < 580`, fails `score < 670`, and lands in the **default**, so an unscored applicant would be filed as standard risk. The original `apply` version usually has the same bug, since `NaN < 580` is False in plain Python too. Make the rule explicit: 1. Put `df["score"].isna()` as the **first** condition with its own label, such as `"unscored"`. 2. Or choose a default that means "needs review" rather than the safest tier. 3. Then check the tier counts, for example with `value_counts(dropna=False)`. ## When `apply` or a loop is still the honest answer - The rule calls something that only takes one value at a time, such as an external scoring function. - The logic depends on earlier rows in a way no shift, cumulative or window method expresses. - `iterrows()` is the slowest route: it builds a Series per row, which can upcast an int column to float, and writing to that row does not change the frame. `itertuples()` is the faster loop when one is unavoidable; the lesson on speed and memory measures the difference. ## Verifying the rewrite A rewrite for speed must not change who lands in which tier, so check it before deleting the old code: - Run both versions on a realistic sample and diff them with `Series.compare`, which lists only the rows whose values differ. - Expect the differences to be the rows the old code got wrong, such as missing scores, and confirm each one is intended. - Compare `value_counts(dropna=False)` of the old and new tier columns, so a tier that silently emptied out is visible. ## What interviewers listen for - Naming **why** `apply(axis=1)` is slow: a row Series and a Python call per row. - **Order** of conditions in `np.select` and an explicit `default`. - The **NaN** trap and how it is handled. - Knowing that the returned array is assigned **by position**, so it must come from the same frame.
- Why does the order of `condlist` in `np.select` matter?`np.select` gives each row the choice of the first condition that is true, like an if/elif chain. With overlapping thresholds such as `score < 670` and `score < 580`, the broader test must come after the narrower one, or the narrower tier never receives any rows.
- How does `Series.case_when` differ from `np.select`?`case_when`, added in pandas 2.2, is called on a Series that supplies the fallback values, takes a list of `(condition, replacement)` tuples, and returns a Series with the caller's index. `np.select` takes separate condition and choice lists plus a `default` and returns a bare NumPy array. Both use the first matching condition.
- When is `iterrows()` acceptable for deriving a column?Almost never for deriving a column. It yields each row as a Series, which can change dtypes such as int to float, and assigning into that row does not reach the frame. If a genuine per-row step remains, `itertuples()` is faster and keeps values as they are; collect results in a list and assign once.
- Says apply(axis=1) is vectorized because pandas runs it internally
- Lists np.select conditions broadest first and loses the narrow tier
- Combines conditions with Python and/or instead of & and |
- Ignores that a NaN score falls through to the default tier
- Writes into the row inside iterrows and expects the frame to change