skip to content

Explainers & Background Data

TreeExplainer for tree ensembles, Deep or Gradient for networks, Permutation or Kernel for any model, each fed background data. Interviewers ask because a wrong pick costs exactness or hours.

on this pageshow

questions

6

In the shap library, which explainer fits a LightGBM regressor, a PyTorch network and an opaque predict function, and why?

level: juniorimportance: must knowfreq 38%

answer

  1. what can the explainer see inside?
  2. fitted trees can be walked directly
  3. networks expose their gradients
  4. a bare function only answers calls
  5. agnostic cost: evaluations times background rows

basics

~20 s

Use TreeExplainer for the LightGBM model, DeepExplainer or GradientExplainer for the PyTorch network, and PermutationExplainer or KernelExplainer for the opaque function: the more of the model an explainer can read, the faster and more exact it is.

solid answer

~50 s

shap groups its explainers by how much of the model they can see. `TreeExplainer` reads the fitted trees of LightGBM, XGBoost, CatBoost and most scikit-learn tree models and runs Tree SHAP, which its docstring calls fast and exact, so explaining tens of thousands of rows is routine. For a network, `DeepExplainer` (DeepLIFT-style rules integrated over a background) or `GradientExplainer` (expected gradients) uses the network's own backward pass, and both need a PyTorch or TensorFlow model object. A function you can only call gets a model-agnostic explainer such as `PermutationExplainer`, which masks features, fills them from background rows and calls the function again and again, so its cost is masked evaluations per row times background rows. Every one of them except TreeExplainer's background-free path is handed a background dataset that stands in for a feature being unknown.

code

python · 15 lines
python
import shap
import torch

# LightGBM regressor: read the fitted trees directly
tree_explainer = shap.TreeExplainer(quote_model, data=X_background)
tree_sv = tree_explainer(X_quotes)

# PyTorch reranker: backward passes averaged over a tensor background
bg = torch.tensor(carrier_background, dtype=torch.float32)
deep_explainer = shap.DeepExplainer(reranker, bg)
deep_sv = deep_explainer.shap_values(torch.tensor(carrier_rows, dtype=torch.float32))

# Partner risk score: only a function, so mask features and call it
perm_explainer = shap.PermutationExplainer(partner_predict, X_background)
perm_sv = perm_explainer(X_quotes[:1000])

go deeper

for a junior

Recall the three families: TreeExplainer for supported tree ensembles, DeepExplainer or GradientExplainer for PyTorch and TensorFlow networks, and Permutation or Kernel explainers for anything you can only call.

for a middle

Explain why access sets the cost: Tree SHAP walks the trees in compiled code, while model-agnostic explainers pay masked evaluations per row times background rows, and passing model.predict silently loses the tree path.

for a senior

Show you would pick per model in a mixed system, keep the network as a model object so gradients stay available, and budget the black-box explainer separately because it dominates the run time.

for a principal

Frame the trade as access against cost and exactness: getting a partner to expose a model object, or accepting sampled approximations, is a contract decision as much as a library choice.

## Why the explainer depends on what you can see The **shap** library computes attributions for one prediction: how much each input feature pushed the output away from a reference level. How it computes them depends on how much of the model it is allowed to inspect. There are three broad situations, and they map to three families of explainer: - **You hold a supported tree model object** (LightGBM, XGBoost, CatBoost, most scikit-learn tree ensembles): shap can walk every tree. - **You hold a differentiable network** (a PyTorch `nn.Module` or a TensorFlow / Keras model): shap can run backward passes through it. - **You hold only a function** that takes rows and returns predictions: shap can only change inputs and call it again. A freight quote engine is a good test case: a LightGBM regressor prices each load, a small PyTorch MLP reranks candidate carriers, and a partner's risk score is reachable only as a Python `predict` function. Each one gets a different explainer. ## Tree ensembles: TreeExplainer `shap.TreeExplainer(model, data=None, model_output="raw", feature_perturbation="auto")` parses the fitted trees and runs **Tree SHAP**, which the class docstring describes as a fast and exact method for tree models. It calls compiled code, either inside the model package or shap's own C extension, so it does not need to call `predict` thousands of times per row. That is why it is the first choice for the LightGBM quote model, and why sending a tree model anywhere else throws away both speed and exactness. Two details matter in practice: 1. Pass the **model object**, not `model.predict`. A bound method is not a tree model, so the unified `shap.Explainer` cannot recognise it and falls back to a model-agnostic algorithm. 2. A background is **optional** here. Without one, TreeExplainer uses the training counts stored in the trees; with one, it integrates over the background rows instead. ## Networks: DeepExplainer and GradientExplainer For the reranker, shap offers two calls that use the network's own gradients: - `shap.DeepExplainer(model, data)` applies DeepLIFT-style rules to each supported layer and averages over the background tensors. Its cost grows linearly with the number of background samples; the docstring suggests 100 to 1,000. - `shap.GradientExplainer(model, data, batch_size=50, local_smoothing=0)` computes **expected gradients**, sampling a background row and an interpolation point many times per explained row (`nsamples=200` on the call). Both detect the framework from the model object, so they need the PyTorch or TensorFlow model itself. A NumPy `predict` wrapper around the network hides the gradients and leaves you in the model-agnostic family. How these gradient methods differ as ideas belongs to the material on gradient attribution; the library question is only which call fits the model. ## Anything callable: the model-agnostic explainers The partner's score is a black box, so shap must **mask** features: replace some of them with values taken from background rows, call the function, and compare. The model-agnostic explainers are: | Explainer | What it does | Typical use | |---|---|---| | `PermutationExplainer` | walks random feature orderings forward and backward; call default `max_evals=500` | the general default for a callable with many features | | `ExactExplainer` | evaluates all 2^M masks for M varying features (with an `Independent` masker) | only a handful of features | | `PartitionExplainer` | follows a feature hierarchy from a clustering masker, giving Owen values | correlated feature groups, text, images | | `KernelExplainer` | the older standalone path: weighted regression over sampled coalitions | legacy code, or when you need its specific options | All of them pay for the same thing: **model calls**. Each explained row costs a number of masked evaluations, and each masked evaluation runs the function on up to one copy of the row per background row. That is the reason a model-agnostic run is slow, and why the background is usually cut to around 100 rows. ## The background every explainer needs Apart from TreeExplainer's background-free path, every explainer is handed **background data**: real rows that supply the values a "hidden" feature takes. Which rows to use, and why different backgrounds give different attributions, is a modelling question covered by the material on Shapley values. The library-level facts are simpler: the background is an argument (`data` or the masker), its size drives the cost, and shap's tabular maskers cut a large one down to `max_samples=100` rows unless told otherwise. ## Putting it together For the quote engine: - **LightGBM quote model**: `TreeExplainer`, with or without a background. - **PyTorch reranker**: `DeepExplainer` or `GradientExplainer`, given the `nn.Module` and a tensor background. - **Partner risk score**: `PermutationExplainer` (or `shap.Explainer`, which picks permutation for a 48-feature callable), budgeted carefully because it is by far the most expensive of the three.

  • Why not send all three models through KernelExplainer, since it accepts any function?
    Because it only sees predictions. For each explained row `KernelExplainer` evaluates `nsamples` coalitions ("auto" is `2 * features + 2048`, so 2,144 for 48 features), each one over every background row, and it does not cut a large background down itself. On the LightGBM model that replaces TreeExplainer's fast exact values with a slower estimate, and on the network it ignores the gradients that make Deep and Gradient explainers cheap.
  • What happens if you pass `quote_model.predict` instead of `quote_model` to `shap.Explainer`?
    The bound method is not a tree model object, so TreeExplainer's support check fails and `algorithm="auto"` treats it as an ordinary callable. With a 48-column background wrapped as an `Independent` masker, more than 10 features means the permutation algorithm. The run still works, just far slower and approximate, and nothing warns you. Pass the model object to keep the tree path.

  • KernelExplainer is the right default for every model because it is model-agnostic.
  • TreeExplainer can explain any model as long as you give it a background dataset.
  • DeepExplainer can explain a NumPy predict function that wraps the network.
  • A model-agnostic explainer is about as fast as TreeExplainer on a tree ensemble.
  • Background data is only needed when explaining neural networks.
open as a page

When shap.Explainer runs with algorithm="auto", how does it decide which explainer to build for a given model and masker?

level: middleimportance: should knowfreq 16%

basics

~20 s

shap.Explainer tries the linear, tree and additive explainers in turn; failing those, a callable model gets exact or permutation by masker type and feature count, or partition for text and image maskers. Kernel, Deep and Gradient are never chosen.

open as a page

When shap.Explainer is handed a 20,000-row DataFrame as its masker, how many rows does it use, and how should you size that background?

level: middleimportance: should knowfreq 14%

basics

~10 s

It uses 100 rows: the DataFrame is wrapped in shap.maskers.Independent, whose max_samples=100 sub-samples it with shap.utils.sample. Size the background on purpose, with max_samples or shap.sample, because model-agnostic cost grows linearly with it.

open as a page

In shap 0.52's TreeExplainer, what does feature_perturbation="auto" resolve to, and what changes when you pass a background dataset?

level: middleimportance: should knowfreq 12%

basics

~20 s

In shap 0.52, feature_perturbation="auto" means "interventional" when TreeExplainer gets a background and "tree_path_dependent" when it does not. A background makes the explainer integrate over those rows, which allows probability or log-loss outputs and costs time per row.

open as a page

An overnight shap job must explain 50,000 quotes from a 48-feature black-box predict function; how do you estimate its cost and cut it?

level: seniorimportance: should knowfreq 8%

basics

~20 s

Count model predictions: rows times masked evaluations per row times background rows. With shap 0.52's PermutationExplainer defaults that is 50,000 x 485 x 100, about 2.4 billion predictions; cut the background, the evaluation budget or the rows explained.

open as a page

For a PyTorch MLP explained with shap, when do you choose DeepExplainer over GradientExplainer, and what does each cost per row?

level: seniorimportance: nice to knowfreq 6%

basics

~20 s

Choose DeepExplainer when every layer is in its rule table: one backward pass over the background per row, additivity checked. Choose GradientExplainer for layers it cannot handle: 200 sampled gradient evaluations per row by default, with sampling noise.

open as a page