In pandas, what does engine='numba' change for rolling(...).apply() and groupby transform or agg, and why can its first call be slower than the default?
answer
- compile the function with the loop
- NumPy arrays in, not Series
- raw=True and a values, index signature
- compilation cached per function object
basics
~20 sengine='numba' JIT-compiles your function together with pandas' window or group loop, so it runs as machine code on NumPy arrays instead of calling Python per window. The first call pays compilation; later calls with the same function reuse the cached code.
solid answer
~40 sWith the default engine, `rolling(...).apply(f)` and `groupby(...).transform(f)` call the Python function `f` once per window or group. With `engine="numba"` pandas JIT-compiles `f` together with the loop, so the whole pass runs as machine code. The contract changes: `rolling.apply` requires `raw=True` (it raises `ValueError` otherwise) and passes NumPy arrays, and a groupby function must start with `values, index`, receives NumPy arrays and gets no fallback if Numba cannot compile it. `engine_kwargs` accepts `nopython`, `nogil` and `parallel` (defaults `True`, `False`, `False`). The first call compiles, which can take longer than the whole default run; pandas caches the result per function object, so defining `f` once and reusing it is what makes the engine pay off, typically on a million or more data points.
code
python · 25 linesimport numpy as np
import pandas as pd
def trimmed_mean(values):
s = np.sort(values)
return s[1:-1].mean()
def demeaned(values, index):
return values - values.mean()
rng = np.random.default_rng(0)
events = pd.DataFrame({
"user_id": rng.integers(0, 1_000, 1_000_000),
"latency": rng.random(1_000_000),
})
roll = events["latency"].rolling(30)
first = roll.apply(trimmed_mean, raw=True, engine="numba") # compiles
again = roll.apply(trimmed_mean, raw=True, engine="numba") # reuses the cache
centered = events.groupby("user_id")["latency"].transform(demeaned, engine="numba")
print(first.tail(3), centered.head(3), sep="\n")go deeper
Recall that some pandas methods accept engine='numba', which compiles a custom function for speed once the optional numba package is installed.
Explain the contract: raw=True for rolling apply, the values, index signature for groupby functions, NumPy arrays in, and compilation on the first call.
Show when it pays off: large data, a reused module-level function, Numba-compatible numeric code, and no built-in that already does the job.
Decide whether a team adopts Numba as a dependency for a few hot paths, weighing compile-time costs, debugging difficulty and the narrower code it accepts.
## The problem the engine solves Some calculations have no built-in pandas method: a custom score over each 30-event window, a trimmed mean per user, a weighted rule over a group. With the default engine, `Series.rolling(30).apply(f)` and `DataFrame.groupby("user_id").transform(f)` loop over the windows or groups and **call the Python function `f` once per window or group**. For 50 million rows that is tens of millions of interpreter calls, and the loop spends most of its time on them. `engine="numba"` asks pandas to compile `f` with **Numba**, a just-in-time (JIT) compiler that turns numeric Python and NumPy code into machine code, and to compile pandas' own loop around it. The whole pass then runs without returning to the interpreter between windows. ## Where it is available - `Rolling.apply` and `Expanding.apply`, and their groupby forms - `DataFrameGroupBy.transform`, `SeriesGroupBy.transform`, `.agg` and `.aggregate` - built-in window aggregations such as `rolling(...).mean(engine="numba")` The option `compute.use_numba` (default `False`) makes `engine=None` choose Numba wherever it is supported. ## The contract changes | | default engine | `engine="numba"` | |---|---|---| | `rolling.apply` input | a Series per window (`raw=False`) or an ndarray (`raw=True`) | an ndarray; `raw=False` raises `ValueError: raw must be True when using the numba engine` | | groupby function signature | any callable | must start with `values, index`, else `NumbaUtilError` | | what the function receives | pandas objects | NumPy arrays of the group's values and index | | unsupported code | runs in Python | compilation fails; no fallback | | options | none | `engine_kwargs` with `nopython`, `nogil`, `parallel` | The function must be code Numba can compile: arithmetic, loops and supported NumPy calls on numeric arrays. Pandas methods, string handling and arbitrary Python objects do not compile. ## Why the first call is slow Compilation happens on the first call. The pandas user guide's own example on a one-million-row rolling apply: 1. first call with `engine="numba"`: about **1.23 s**, mostly compilation 2. later calls: about **188 ms** 3. the same apply with `engine="cython"`: about **3.92 s** On a million rows even the first call beat the default engine. On a few thousand rows it does not: the default run takes milliseconds while compilation still costs on the order of a second, as in the guide's example. pandas caches the compiled loop, keyed on the **function object** and the `engine_kwargs`. Two consequences: - **Define the function once**, at module level, and pass the same object every time. A new `lambda` built inside a loop or a per-request function is a new object, so every call recompiles and the engine never pays off. - **Small inputs lose.** The guide says the engine performs well with a larger number of data points, for example one million or more. On a few thousand rows the compilation cost dominates. ## `engine_kwargs` `engine_kwargs` is passed to the `numba.jit` decorator. Its default is `{"nopython": True, "nogil": False, "parallel": False}`. - **`nopython=True`** fails loudly when code cannot be compiled to machine code, instead of falling into a slow object mode. - **`nogil=True`** releases Python's global interpreter lock inside the compiled code, useful when several threads each run their own calculation. - **`parallel=True`** lets pandas spread work across columns on several cores; the guide notes the gain applies to frames with many columns, not to a single long Series. ## Compiling your own function instead The pandas user guide also shows a second route for logic that is not shaped like a window or a group: decorate a plain function with `numba.jit`, pass it the arrays from `Series.to_numpy()`, and wrap the returned array in a Series with the original index. pandas is then only the container; the loop is yours and runs as machine code. The same rules apply: numeric code only, compilation on the first call, and a gain only on large inputs. ## When to reach for it Use the numba engine when all of these hold: - the logic cannot be written with built-in methods or column arithmetic - it is numeric and Numba-compatible - the data is large, and the same function is applied many times If a built-in such as `rolling(30).mean()` or `groupby(...).transform("sum")` expresses the calculation, use that: those built-ins are already compiled and need no warm-up.
- Why does a rolling apply with engine='numba' stay slow when it runs inside a loop over files?If the function is written as a fresh `lambda` or nested function on each iteration, it is a new object every time. pandas caches compiled code per function object, so each iteration recompiles. Define the function once at module level and pass that same object to every call.
- When does engine_kwargs={'parallel': True} help?pandas uses Numba's parallelism across the columns of a DataFrame, so it helps a wide frame with many columns on a multi-core machine. A single long Series gains little. The user guide also warns that some threading layers can crash with parallel=True, so pick a safe threading layer first.
- engine='numba' works with raw=False and passes each window as a Series.
- A groupby function for the numba engine may take any argument names.
- Numba makes the very first call faster than the default engine.
- If Numba cannot compile the function, pandas falls back to Python.
- Any Python code, including string methods, compiles under engine='numba'.