skip to content

Base Values & Plots

Each SHAP value pushes a prediction away from base_values, and a row sums to the output on the model's scale, often log-odds. Interviewers have candidates narrate waterfall and beeswarm plots.

on this pageshow

questions

6

How does `shap.plots.waterfall` build one patient's readmission explanation from the Explanation's `.base_values`, `.values` and `.data`?

level: juniorimportance: must knowfreq 38%

answer

  1. start at the average output
  2. one bar per feature, largest on top
  3. base plus bars lands on f(x)
  4. max_display=10 and the leftover bar
  5. one row in, never the whole batch

basics

~20 s

A shap waterfall takes one row of an Explanation: it starts at base_values, the average model output E[f(X)], adds each feature's SHAP value from .values as a bar labelled with its .data value, and ends at that patient's output f(x).

solid answer

~40 s

`shap.plots.waterfall(sv[i])` explains one prediction. The bottom tick, `E[f(X)]`, is the row's `.base_values`: the model's average output over the reference data the explainer used. Each feature then gets a bar of length `.values[j]`, red pushing the output up and blue pushing it down, sorted by magnitude and labelled `value = name` from `.data` (or `.display_data` when it is set). Because the values are locally additive, `base_values + values.sum()` lands exactly on the top tick, `f(x)`. With the default `max_display=10`, the nine largest features get their own bars and the tenth bar collapses the rest, so a 30-feature model shows "21 other features". For a readmission model explained on its log-odds margin, I read every bar in log-odds, not in percentage points of risk.

code

python · 5 lines
python
sv = explainer(X_discharges)   # Explanation for every discharged patient
row = sv[17]                   # one patient: 1-D Explanation
print(row.base_values)         # E[f(X)], shared by every row
print(row.base_values + row.values.sum())  # f(x) on the explained scale
shap.plots.waterfall(row)      # max_display=10: 9 bars + "21 other features"

go deeper

for a junior

Recall the three ingredients: base value at the bottom, one signed bar per feature from .values, and f(x) at the top, with the feature's actual value printed beside each bar.

for a middle

Explain the additivity that makes the plot close and the max_display cut: nine named bars plus a grouped bar by default, and why the axis is the model's explained scale.

for a senior

Show you can hand a waterfall to non-specialists safely: label the scale, explain what the base value is averaged over, and refuse causal or global readings of one patient's bars.

for a principal

Weigh what a team standardises before per-patient plots leave the data team: one explainer and reference set per model version, a fixed scale and a stated max_display.

## What one row of an Explanation holds Calling a shap explainer on a batch returns a `shap.Explanation`. Indexing it with one row number, `sv[17]`, gives the explanation of a single prediction, and a waterfall plot is a picture of exactly that object. The attributes it reads are: | Attribute | For one row it holds | Where it shows up in the waterfall | |---|---|---| | `.base_values` | One number: the model's expected output over the explainer's reference data | The bottom tick, `E[f(X)]` | | `.values` | One SHAP value per feature, on the model's explained scale | The length and direction of each bar | | `.data` | The patient's raw feature values | The `value = name` label beside each bar | | `.display_data` | Optional human-readable values (category names, unscaled units) | Used for the labels instead of `.data` when set | | `.feature_names` | Column names | The feature names in the labels | The **base value** is not the patient's personal starting risk. It is the average output the model gives across the reference distribution: the background the explainer was handed, or, for a tree model explained without one, the training data recorded in the trees. Every patient explained by the same explainer shares it. ## How the waterfall is drawn `shap.plots.waterfall(shap_values, max_display=10, show=True)` builds the picture in four steps: 1. It checks that it was handed a **single** explanation. A matrix of rows raises a `ValueError` that suggests `shap.plots.waterfall(shap_values[0])`. 2. It sorts features by the absolute size of their SHAP values, largest at the top. 3. Starting from `E[f(X)]` at the bottom, it stacks one bar per feature: **red** bars push the output up, **blue** bars push it down. 4. It finishes at `f(x)` at the top, the model's output for this patient on the explained scale. When the model has more features than `max_display`, only `max_display - 1` features get their own bar. The last bar is labelled "N other features" and carries the summed SHAP values of everything not shown, so the arithmetic still closes. With 30 features and the default of 10, that is nine named bars plus "21 other features". ## A worked readmission example Take a hospital's 30-day readmission classifier, an XGBoost binary model explained on its raw log-odds margin, and one discharged patient: - `E[f(X)] = -1.20`, the average margin across the reference patients - `prior_admissions_12m = 3` pushes up by `+0.62` - `length_of_stay = 9` pushes up by `+0.31` - `age = 78` pushes up by `+0.18` - `follow_up_booked = 1` pushes down by `-0.45` - the remaining features, shown and grouped, sum to `-0.31` The bars sum to `+0.35`, so `f(x) = -1.20 + 0.35 = -0.85`. That is a **margin**, not a probability: passed through the logistic function it is about 0.30, against about 0.23 for the base value. A care-management reader must be told the axis is log-odds, or they will read "+0.62" as "62 points of risk". ```python sv = explainer(X_discharges) # Explanation, shape (n_patients, 30) row = sv[17] # one patient print(row.base_values + row.values.sum()) # equals f(x) on the explained scale shap.plots.waterfall(row) # 9 named bars + "21 other features" shap.plots.waterfall(row, max_display=15) # 14 named bars + "16 other features" ``` ## Common misreadings - **Treating bars as global importance.** A bar is this patient's contribution only; another patient with the same feature value can get a different bar because the other features differ. - **Expecting the bars to sum to the predicted probability.** They sum to the model's output on the explained scale, which for this model is the margin. - **Reading "other features" as ignored features.** It is the summed contribution of everything below the cut, and it can be large when many small effects point the same way. - **Reading a bar as causal.** A bar says how the model's output moved, not what would happen to the patient if the value changed. ## Neighbouring one-row views - `shap.plots.force(sv[17])` draws the same additive story as a horizontal tug-of-war. It renders through JavaScript, so a notebook needs `shap.initjs()` first; without it the cell shows "Visualization omitted, Javascript library not loaded!". `matplotlib=True` draws a static version for a single row. - `shap.plots.bar(sv[17])` draws the same row as signed bars without the running total. The waterfall stays the usual choice for a per-patient handout because it shows the starting point, every step and the end point on one axis.

  • Why does `shap.plots.waterfall(sv)` fail when `sv` holds the whole batch?
    The waterfall plots a single explanation. A 2-D Explanation of rows by features raises a `ValueError` saying it can only plot one explanation and suggesting `shap.plots.waterfall(shap_values[0])`. Slice the patient you want, `sv[17]`, or use a many-row plot such as `shap.plots.beeswarm(sv)` for the whole batch.
  • What does the bottom "21 other features" bar contain, and when should you change `max_display`?
    It is the sum of the SHAP values of every feature below the cut, so the plot still adds up to `f(x)`. With 30 features and `max_display=10`, nine features are named and 21 are grouped. If that bar is as long as the named ones, many small effects share the push, and raising `max_display` shows them.
  • How do you show the same patient as a force plot in a notebook?
    Run `shap.initjs()` once, then `shap.plots.force(sv[17])`. The force plot renders through JavaScript; without `initjs()` the cell prints "Visualization omitted, Javascript library not loaded!". For a static image, `matplotlib=True` works for a single row but raises `NotImplementedError` for several rows.

A waterfall reads like a bank statement: the opening balance is the base value, each line is a deposit or withdrawal, small items are lumped into one sundry line, and the closing balance is the opening balance plus every line, all in the account's own currency, which here is log-odds.

  • The base value is this patient's own risk before any features are considered.
  • The bars of a waterfall add up to the predicted probability for every classifier.
  • The other-features bar holds features the model ignored.
  • A waterfall bar shows how important the feature is across all patients.
  • You can pass the whole batch to the waterfall to see every patient at once.
open as a page

A shap waterfall for a binary XGBoost readmission model shows f(x) = -0.85 while `predict_proba` gives 0.30 — why, and how do you reconcile them?

level: middleimportance: must knowfreq 18%

basics

~20 s

By default TreeExplainer explains a binary XGBoost model on its raw log-odds margin, so base value plus SHAP values equals -0.85 in log-odds. The logistic function maps it to 0.30, the predicted probability; individual bars have no fixed probability equivalent.

open as a page

How do you read a `shap.plots.beeswarm` of a readmission model, and what do its default ordering and `max_display` leave out?

level: middleimportance: should knowfreq 24%

basics

~20 s

In a shap beeswarm each row is a feature and each dot a patient: x is the SHAP value, colour the feature's value. Rows rank by mean absolute SHAP value, and by default only nine are named.

open as a page

Since shap 0.45, what shape does a three-class scikit-learn RandomForestClassifier's Explanation have, and how do you plot one class?

level: middleimportance: should knowfreq 13%

basics

~20 s

In shap 0.52 a three-class random forest returns one array of shape (rows, features, 3), with base_values of shape (rows, 3). Slice a class on the last axis, sv[:, :, 2] for a beeswarm or sv[0, :, 2] for one patient's waterfall.

open as a page

In `shap.plots.scatter` for a readmission model's length-of-stay column, what does vertical spread mean, and what does `color=shap_values` add?

level: seniorimportance: should knowfreq 9%

basics

~20 s

A shap scatter plots each patient's length of stay against its SHAP value. Vertical spread at one length of stay means other features change its contribution; color=shap_values colours dots by the feature shap's heuristic ranks as interacting most.

open as a page

TreeExplainer raises 'Additivity check failed in TreeExplainer!' on a readmission scoring batch — what is it telling you, and is `check_additivity=False` the fix?

level: seniorimportance: nice to knowfreq 5%

basics

~20 s

The error means shap's tree walk and the model's own prediction disagree for some row: base value plus SHAP values misses the model's raw output beyond a 1e-2 tolerance. Setting check_additivity=False only skips the check; the unverified values may not add up.

open as a page