When shap.Explainer is handed a 20,000-row DataFrame as its masker, how many rows does it use, and how should you size that background?
answer
- the masker has a row ceiling
- the default is a round hundred
- max_samples on the masker changes it
- sample draws rows, kmeans builds centroids
- KernelExplainer only warns, never cuts
basics
~10 sIt uses 100 rows: the DataFrame is wrapped in shap.maskers.Independent, whose max_samples=100 sub-samples it with shap.utils.sample. Size the background on purpose, with max_samples or shap.sample, because model-agnostic cost grows linearly with it.
solid answer
~40 sA DataFrame passed as the masker becomes `shap.maskers.Independent(data)`, and its `max_samples=100` default sub-samples anything larger with `shap.utils.sample`, without replacement. shap 0.52 logs a message on the `shap` logger when it does this, which is easy to miss in a batch job. To choose the size yourself, build the masker: `shap.maskers.Independent(X_bg, max_samples=500)`, or pre-sample with `shap.sample(X_bg, 500)`. The trade-off is cost against the noise of a small reference set: every masked evaluation runs the model over every kept background row. `KernelExplainer` is different: it never cuts the background, only warns above 100 rows, and pairs with `shap.kmeans(X, k)`, which returns k weighted centroids rather than real rows.
code
python · 12 linesimport shap
# X_bg: 20,000 recent loads, 48 feature columns
explainer = shap.Explainer(partner_predict, X_bg)
print(explainer.masker.data.shape) # (100, 48), and a log message
# Decide the size yourself
masker = shap.maskers.Independent(X_bg, max_samples=500)
explainer = shap.Explainer(partner_predict, masker)
# KernelExplainer never cuts its data: summarise first
kernel = shap.KernelExplainer(partner_predict, shap.kmeans(X_bg, 20))go deeper
Remember that the default tabular masker keeps at most 100 background rows, and that a bigger background makes model-agnostic explanations slower.
Explain max_samples=100, the shap.utils.sample cut without replacement, the 0.52 log line, and when to use shap.sample against shap.kmeans and its weighted centroids.
Show how you would pick a background size empirically, make it explicit in a batch job, and keep KernelExplainer from silently running against tens of thousands of rows.
Treat the background as a versioned artefact with an owner: its size sets the compute budget, and its contents set the reference every explanation is read against.
## What the background is for, in library terms A **background** (the masker's data) is the set of real rows shap draws from when it needs a value for a feature it is "hiding". Which population those rows should represent, and why a different background changes the attributions, is a question about Shapley values themselves. The library questions are narrower: **how many rows** the explainer actually uses, **how it chooses them**, and **what they cost**. ## The default: 100 rows, chosen for you When `shap.Explainer(model, masker)` receives a pandas DataFrame or a 2-D array as the masker, it wraps it as `shap.maskers.Independent(data)`. The signature is `Independent(data, max_samples=100)`. If the data has more rows than `max_samples`: 1. the masker calls `shap.utils.sample(data, max_samples)`, which shuffles and keeps that many rows **without replacement**, with a fixed `random_state=0` by default; 2. shap 0.52 logs a message on the `shap` logger: "Background dataset has 20000 samples but max_samples=100. Subsampling to 100 samples...". So a freight brokerage that hands a 20,000-row table of recent loads to the explainer for its 48-feature partner score gets a 100-row background. The same wrapping happens inside `TreeExplainer` when you pass a DataFrame as `data`, and `shap.maskers.Partition(data, max_samples=100, clustering="correlation")` has the same ceiling. ## Why the size matters: cost For the model-agnostic explainers, each **masked evaluation** of one explained row runs the model over the kept background rows (fewer when some rows already match the row being explained). So: - model predictions per explained row is roughly masked evaluations times background rows; - doubling the background roughly doubles the run; - for `TreeExplainer` on the interventional path, the docstring likewise says run time is linear in the background size. ## Choosing the size deliberately | Goal | Call | |---|---| | keep the 100-row default but know it happened | check `explainer.masker.data.shape` or the `shap` log | | keep more (or fewer) real rows | `shap.maskers.Independent(X_bg, max_samples=500)` | | pre-sample before building anything | `shap.sample(X_bg, 500)` (default `nsamples=100`) | | summarise for KernelExplainer | `shap.kmeans(X_bg, 20)` | A reasonable process: start at the default, rerun a few rows at 500 or 1,000 rows, and keep the smallest size at which the attributions you care about stop moving. The masker docstring itself calls about 1, 10, 100 or 1,000 rows reasonable choices. ## shap.sample against shap.kmeans - **`shap.sample(X, nsamples=100, random_state=0)`** returns real rows, drawn without replacement (it sampled with replacement before 0.42). It works anywhere a background array or DataFrame is accepted. - **`shap.kmeans(X, k, round_values=True)`** runs k-means, snaps each centroid coordinate to the nearest value actually observed in that column, and returns a weighted `DenseData` object whose weights are the cluster sizes. Missing values are mean-imputed first. The weighted object is built for `KernelExplainer`, which uses the weights. The maskers and `TreeExplainer` expect an array, a DataFrame or a masker; `TreeExplainer` rejects the `DenseData` object as an unsupported masker type. ## KernelExplainer: no automatic cut `shap.KernelExplainer(f, data)` does not wrap its data in a masker, so it never sub-samples. Above 100 background rows it only logs a warning recommending `shap.sample(data, K)` or `shap.kmeans(data, K)`. Pass 20,000 rows and it will use all 20,000 for every sampled coalition, which is how an overnight job turns into a multi-day one. ## Checking what you actually got Because the cut happens inside a constructor, a batch job should make it visible rather than trust it: - read `explainer.masker.data.shape` after building a unified or model-agnostic explainer, and `explainer.data.shape` on a `TreeExplainer`; - route the `shap` logger into the job's logs, so the sub-sampling message is recorded with the run; - fix the random state: `shap.sample` and the masker's sub-sample both use `random_state=0` by default, so the same input gives the same rows, but a reshuffled input table changes them; - store the background you used (or the call that built it) next to the explanations, so a later rerun compares like with like. None of this decides which population the background should represent; that remains a modelling choice. It only makes sure the size and the rows are the ones you meant.
- When would you hand shap a Partition masker instead of Independent for the 48 quote features?When features come in correlated groups and you want attributions to respect that hierarchy. `shap.maskers.Partition(X_bg, clustering="correlation")` clusters the features and keeps the same `max_samples=100` ceiling. With `algorithm="partition"` the explainer returns Owen values that credit groups along the tree; with "auto", a Partition masker over 32 features goes to the permutation algorithm instead.
- Why is shap.kmeans not a drop-in replacement for shap.sample in front of TreeExplainer?Because it returns a weighted `DenseData` summary, not rows. `KernelExplainer` reads those weights; `TreeExplainer` accepts an array, a DataFrame or an `Independent` masker and rejects the `DenseData` object as an unsupported masker type. In front of TreeExplainer, use `shap.sample` or a masker with an explicit `max_samples`.
- The masker integrates over every row of the DataFrame you pass it.
- Background size only affects accuracy, not how long the explanation run takes.
- shap.sample draws background rows with replacement.
- KernelExplainer sub-samples a large background automatically, the way the maskers do.
- shap.kmeans centroids can be passed anywhere a background DataFrame is accepted.
Assumes shap 0.52, where a masker that sub-samples its background logs a message on the shap logger (added in 0.52.0); the max_samples=100 default is read from the 0.52 source.