skip to content

Vectorized Speed and Memory

pandas memory grows with object columns and copies, and speed comes from vectorizing. Interviewers ask how to measure memory_usage, shrink dtypes and speed up a slow loop.

on this pageshow

questions

6

In pandas, why is a loop over df.iterrows() slow, and what does switching to df.itertuples() change?

level: juniorimportance: must knowfreq 70%

answer

  1. a new object for every row
  2. one dtype per row Series
  3. namedtuple of plain values
  4. still a Python-level loop

basics

~20 s

iterrows() builds a full pandas Series for every row, which is slow and coerces the row's values to one dtype. itertuples() yields lightweight namedtuples of the values, is much faster and keeps each value's type, but is still a Python loop.

solid answer

~40 s

`DataFrame.iterrows()` yields `(index, Series)` pairs, so every row pays for constructing a pandas Series with its own index and dtype. A Series holds one dtype, so a row of an `int64` and a `float64` column comes back as `float64`, and a row with text becomes `object`. `itertuples()` yields namedtuples of the raw values (the index first, as `Index`), skips the Series construction, keeps each value's type and is generally much faster; columns whose names are not valid identifiers are renamed to positions such as `_2`. Both still run the loop body once per row in the interpreter, so on millions of rows the real fix is a column-wise expression or a compiled engine. In pandas 3.0, Copy-on-Write also guarantees that writing to an `iterrows` row never updates the frame.

code

python · 13 lines
python
import pandas as pd

df = pd.DataFrame({"user_id": [7, 8], "amount": [9.5, 0.25], "plan": ["pro", "free"]})

for label, row in df.iterrows():
    print(label, row["user_id"], row.dtype)

for row in df.itertuples():
    print(row.Index, row.user_id, type(row.user_id).__name__, row.plan)

scores = [row.amount * 2 for row in df.itertuples(index=False)]
df = df.assign(score=scores)
print(df)

go deeper

for a junior

Know that iterrows() yields a Series per row and is slow, that itertuples() yields namedtuples and is faster, and that writing to the row does not change the frame.

for a middle

Explain the dtype coercion inside iterrows(), the Index field and renamed columns in itertuples(), and why both remain Python-level loops.

for a senior

Profile before rewriting, and know when a loop is legitimate (per-row I/O) versus when it should become column arithmetic or a compiled function.

for a principal

Treat row loops over large frames as a review smell with stated exceptions, so the team learns where speed comes from rather than swapping one loop for another.

## What `iterrows` does per row `DataFrame.iterrows()` is a generator of `(index_label, row)` pairs, where each `row` is a **new `Series`** whose index is the frame's column labels. To build it, pandas takes the row out of the frame's values and constructs a full pandas object around it, with an index, a name and a dtype. For a frame of 50 million rows that is 50 million Series constructions before the loop body does any work of its own. That construction has two costs: - **Time.** Building a Series involves dtype inference and object setup, many times the cost of reading a few numbers. - **Types.** A Series holds a **single dtype**, so the row's values are coerced to one common type. The docstring's own example: a frame with an `int64` column and a `float64` column yields rows of dtype `float64`, so the integer comes back as `1.0`. Add a text column and the row becomes `object`, so the numbers come back as boxed Python objects. The second point causes real bugs: code that checks `isinstance(row["count"], int)` or formats an id with `{:d}` fails on the coerced value. ## What `itertuples` does instead `DataFrame.itertuples(index=True, name="Pandas")` yields one **namedtuple** per row: - The first field is the index label, named `Index`, unless `index=False`. - The remaining fields are the column values, taken column by column, so each keeps its own column's type. - Fields are read by attribute (`row.amount`) or position (`row[1]`). - Column names that are not valid Python identifiers, repeat, or start with an underscore are **renamed to positional names** such as `_2`, because namedtuple fields must be identifiers. - `name=None` returns plain tuples, slightly cheaper still. Because no Series is built, the user guide describes `itertuples` as generally much faster than `iterrows`, and in most cases the preferable way to iterate over values. | | `iterrows()` | `itertuples()` | |---|---|---| | yields | `(label, Series)` | namedtuple (or tuple) | | per-row cost | a Series construction | a tuple of values | | dtypes | coerced to one per row | each value keeps its column's type | | field access | `row["col"]`, any label | `row.col`; odd names become positions | | writes back to the frame | no | no | ## Neither one is vectorised Both methods still execute the loop body in the Python interpreter once per row. Switching from `iterrows` to `itertuples` removes the Series overhead, which is often a several-fold gain, but the per-row Python work remains. For an hour-long scoring loop the order of options is: 1. **Profile it.** Time a 100,000-row slice with `%timeit`, or run a profiler such as `cProfile`, to see whether the time goes to iteration or to the work inside the loop. 2. **Express the logic on whole columns**, so the arithmetic runs in compiled code over arrays (covered in the material on applying functions and on NumPy vectorisation). 3. **If the logic truly needs a loop**, write it over NumPy arrays from `to_numpy()` and compile it with Numba, or use a pandas method that accepts `engine="numba"`. 4. **Use `itertuples`** when a Python loop is unavoidable, for example when each row triggers an API call or a file write, where the per-row work dwarfs the iteration cost anyway. ## Never write through the row The documentation warns: you should **never modify** something you are iterating over. A row from `iterrows` is its own Series, and in pandas 3.0, where Copy-on-Write is the only mode, assigning `row["amount"] = 0` changes that Series and never the frame. In 2.x, depending on the dtypes, the row could be a view of the frame, which is why the warning exists. A namedtuple is immutable, so `itertuples` cannot even be tried that way. To change values, compute a new column, or collect results in a list and assign the list as a column after the loop. ## A quick check in an interview Asked to fix a slow `iterrows` loop, a good answer names three things: - the per-row Series construction as the cost - the dtype coercion as a correctness hazard - `itertuples` as the cheaper loop, with column-wise operations as the real fix A strong answer also says how the gain would be measured, on a realistic slice, before and after.

  • What happens to a column named 'order date' in itertuples()?
    namedtuple fields must be valid Python identifiers, so pandas renames that field to a positional name such as `_2`, counting the `Index` field as position 0. Access it by position, rename the column first, or pass `name=None` and work with plain tuples.
  • Why does row['user_id'] come back as 7.0 from iterrows()?
    Each row is one Series with one dtype. When the frame's other columns are float, pandas coerces the row to `float64`, so the integer becomes a float. `itertuples()` reads values column by column and keeps `user_id` as an integer.

  • iterrows() preserves each column's dtype in the row it yields.
  • Assigning to row['col'] inside iterrows() updates the DataFrame.
  • itertuples() is vectorised, so it is as fast as column arithmetic.
  • for row in df iterates over rows of a DataFrame.

Assumes pandas 3.0, where Copy-on-Write guarantees that a row from iterrows() never writes back to the frame; in 2.x, depending on the dtypes, the row could be a view of the frame, so the docs warned never to rely on it.

open as a page

In pandas, why can df.info() report a DataFrame as far smaller than it really is, and how does memory_usage(deep=True) fix the measurement?

level: middleimportance: must knowfreq 55%

basics

~10 s

By default pandas counts only each column's array buffer, so an object column counts 8-byte pointers, not the Python strings behind them. memory_usage(deep=True) or info(memory_usage='deep') measures those objects too.

open as a page

When does pandas' DataFrame.eval() with the numexpr engine make a column expression faster, and when does it make it slower?

level: middleimportance: should knowfreq 28%

basics

~20 s

DataFrame.eval() helps when a compound arithmetic or boolean expression runs over large frames: numexpr evaluates it in one chunked, multi-core pass without full-size temporaries. On small frames, simple expressions or text, parsing overhead makes it slower than plain operators.

open as a page

A 50-million-row pandas events DataFrame takes 12 GB in memory; which dtype and read-time changes shrink it, and how do you confirm each one paid off?

level: seniorimportance: should knowfreq 50%

basics

~10 s

Rank columns with memory_usage(deep=True), then read only the needed columns with usecols and declared dtypes, downcast numbers with to_numeric(downcast=), and turn repeated labels into category; re-measure after each step and watch the load's peak.

open as a page

When should a team stop tuning a pandas pipeline and move the workload to a different engine, and what does that move cost?

level: principalimportance: should knowfreq 35%

basics

~20 s

Move when, after measured dtype, read, vectorisation and compiled-engine work, the job still exceeds one machine's memory or its time budget, especially when its core steps resist chunking. The move costs a rewrite, semantic differences, new skills and new operations.

open as a page

In pandas, what does engine='numba' change for rolling(...).apply() and groupby transform or agg, and why can its first call be slower than the default?

level: seniorimportance: nice to knowfreq 18%

basics

~20 s

engine='numba' JIT-compiles your function together with pandas' window or group loop, so it runs as machine code on NumPy arrays instead of calling Python per window. The first call pays compilation; later calls with the same function reuse the cached code.

open as a page