When does pandas' DataFrame.eval() with the numexpr engine make a column expression faster, and when does it make it slower?
answer
- one pass instead of many temporaries
- parsing has a fixed cost
- row count thresholds in the user guide
- plain operators already use numexpr
basics
~20 sDataFrame.eval() helps when a compound arithmetic or boolean expression runs over large frames: numexpr evaluates it in one chunked, multi-core pass without full-size temporaries. On small frames, simple expressions or text, parsing overhead makes it slower than plain operators.
solid answer
~40 s`df.eval("a * b + c - d")` parses the string and, when the optional `numexpr` package is installed, hands it to the `numexpr` engine (the default when it is available). numexpr evaluates the whole expression in cache-sized chunks on several cores, so it skips the full-size temporary that plain pandas builds for each intermediate. That pays off on compound arithmetic and comparisons over large frames; the user guide puts the gain above roughly 100,000 rows and says not to use `eval` below about 10,000. On small frames the parsing overhead makes it many times slower, `engine="python"` gives no speed-up, and parts on object or text columns run in Python. Ordinary binary operators already route to numexpr above one million elements, so the extra gain from `eval` comes from fusing several steps into one pass.
code
python · 17 linesimport numpy as np
import pandas as pd
rng = np.random.default_rng(0)
n = 2_000_000
df = pd.DataFrame(rng.random((n, 4)), columns=["a", "b", "c", "d"])
plain = df["a"] * df["b"] + df["c"] - df["d"]
fused = df.eval("a * b + c - d")
print(np.allclose(plain, fused))
small = df.head(100)
# In IPython, compare:
# %timeit small["a"] * small["b"] + small["c"] - small["d"]
# %timeit small.eval("a * b + c - d")
# %timeit df["a"] * df["b"] + df["c"] - df["d"]
# %timeit df.eval("a * b + c - d")go deeper
Recall that DataFrame.eval() takes an expression as a string and can be faster on large frames when numexpr is installed, but slower on small ones.
Explain the temporaries that plain operators create, how numexpr fuses an expression into chunked multi-core passes, and the row-count thresholds the user guide gives.
Show that plain operators already use numexpr above a million elements, so you measure before rewriting, and keep text-heavy expressions and user input away from eval.
Decide whether a team's large-frame code standardises on eval for hot compound expressions, weighing readability, the numexpr dependency and measured gains.
## What `eval` does `pandas.eval(expr)` and `DataFrame.eval(expr)` take an **expression as a string**, parse it into a tree, and evaluate the tree with one of two engines: - **`"numexpr"`**: hands the arithmetic and boolean parts to the optional **numexpr** package. It is the default when numexpr is installed (`engine=None` resolves to `"numexpr"` if it can be used, otherwise `"python"`). - **`"python"`**: evaluates the tree with ordinary pandas operations. The user guide says it gives **no** performance benefit and is mainly useful for testing the other engine. `DataFrame.eval` adds two conveniences: column names can be used bare, and an assignment such as `"total = price * qty"` returns a new frame with that column (the original is unchanged unless `inplace=True`). Its syntax rules, such as `@` for local variables and backticks for awkward column names, belong to method chaining; here the subject is speed. ## Where the speed comes from Plain pandas evaluates `df.a * df.b + df.c - df.d` one operator at a time. Each step allocates a **full-size temporary** array: `a * b`, then that plus `c`, then that minus `d`. On 50 million `float64` rows each temporary is 400 MB, and every step reads and writes the whole array through memory. numexpr compiles the expression into its own small program and runs it over the inputs in **cache-sized chunks**, using **several cores**. Each chunk goes through every operator while it is still in cache, and only the final result is written out. That gives two wins: - **Speed**, from fewer passes through main memory and multi-core execution. - **Lower peak memory**, because the intermediates exist only chunk by chunk. The gain grows with the number of operators in the expression and with the size of the frame. ## Where it is slower The parse and dispatch cost is fixed per call, so `eval` loses on small inputs. The pandas user guide states: 1. Do **not** use `eval` for simple expressions or small frames; it can be many orders of magnitude slower there than plain Python. 2. As a rule of thumb, use it only when a frame has **more than 10,000 rows**. 3. Its benchmark shows the numexpr engine's benefit only above roughly **100,000 rows**. Three more limits: - **Only certain syntax** is allowed: arithmetic (except `<<` and `>>`), comparisons, boolean operators, attribute and subscript access, and a fixed list of math functions such as `sin`, `log` and `sqrt`. Arbitrary function calls, `lambda` and comprehensions are rejected. - **Object, text and `NaT`-related parts** cannot go to numexpr. pandas evaluates those sub-expressions in Python while the numeric parts still use numexpr, so a mostly-text expression gains nothing. - **numexpr must be installed.** With `engine="numexpr"` requested and the package missing, pandas raises `ImportError`. ## Plain operators already use numexpr A frequent misconception is that `eval` is the only way to get numexpr. With numexpr installed and the option `compute.use_numexpr` at its default of `True`, pandas already sends a **single binary operation** to numexpr when the operand has more than **1,000,000 elements** and the dtypes are among `int64`, `int32`, `float64`, `float32` and `bool`. So `df.a + df.b` on a large frame may already run on numexpr. What `eval` adds is **fusion**: one pass over a compound expression instead of one numexpr call, and one temporary, per operator. | expression | small frame | large frame | |---|---|---| | `df.a + df.b` | plain operator | plain operator (already numexpr over 1,000,000 elements) | | `df.a * df.b + df.c - df.d` | plain operators | `df.eval(...)`: fused, fewer temporaries | | `(df.x > 0) & (df.y < 5) & (df.z == 1)` | plain operators | `df.eval(...)`, or `df.query(...)` to filter | | text comparison | plain operators | little gain: the text part runs in Python | ## Using it safely - **Measure.** Time both versions with `%timeit` on realistic sizes; the threshold depends on the machine and the dtypes. - **Keep strings out of user hands.** pandas' own docstring warns that `eval` can run arbitrary code and is open to code injection if it receives user input. Its parser is narrower than Python's built-in `eval`, but it is not a sandbox. - **Prefer clarity on small data.** For a few thousand rows, ordinary column arithmetic is both faster and easier to read.
- If plain pandas operators already use numexpr on large frames, why does eval still help?Plain operators send one binary operation at a time, each producing a full-size temporary. `eval` passes the whole compound expression to numexpr, which runs every operator on one cache-sized chunk before moving on, so there are fewer passes through memory and no full-size intermediates. The gain grows with the number of operators.
- Is DataFrame.eval safe to call on a filter string typed by a user?No. pandas' docstring warns that `eval` can run arbitrary code and is vulnerable to code injection with user input. Its grammar is narrower than Python's built-in `eval`, but it is not a security boundary. Build filters from validated column names and values instead of passing user text through.
Plain operators work like an assembly line that runs one step over every item, stacks the half-finished items in a warehouse, then starts the next step; eval with numexpr carries a small batch through every step before fetching the next, which pays off only when there are many items.
- df.eval() is always faster than plain operators, whatever the frame size.
- engine='python' gives the same speed-up as numexpr without the dependency.
- pandas only uses numexpr when you call eval() or query().
- eval() can call any Python function inside the expression string.
- eval() is safe on user input because pandas parses the string itself.