Model Interpretability

The model has learned a non-monotone relationship: average predicted demand is highest at cold temperatures and at hot temperatures, and lowest in between. The curve summarises the model's average behaviour over the whole dataset, not any single building.

An interpretable-by-design model is one whose fitted structure is itself the explanation: a sparse weighted sum, a depth-3 tree, a short rule list. Post-hoc methods leave the model opaque and build a separate, approximate account of it.

Permutation importance shuffles one column's values across held-out rows, breaking that column's link to the label, then re-scores the already-trained model. The drop from the baseline score, averaged over several shuffles, is the feature's importance.

Demographic parity requires the same positive-prediction rate in every group, ignoring the true outcome. Equalized odds requires equal true-positive and false-positive rates in every group, comparing errors only among people who share a true label.

Three points. Pre-processing reweights or repairs the training data, in-processing adds a fairness penalty or constraint to the training objective, and post-processing changes the decision rule per group after fitting. Each needs different access and pays a different accuracy cost.

Compute each group's selection rate — the share of that group's applicants the screen passes — then divide the lowest by the highest. A ratio under 0.8 is the conventional flag for adverse impact and calls for justification.

An adverse-action notice must name the specific facts about the applicant that drove the decline, such as a revolving balance at 78% of the limit. A score and a cutoff describe the mechanism, not the reason.

Choose a grid of values for that feature. For each grid value, overwrite the feature with it in every row, score the whole dataset with the model, and average the predictions. Plot grid value against average prediction.

Because shuffling one of them leaves its twin intact, and the model recovers the same information from the twin, so the score barely moves. Each correlated column masks the other's credit, and both end up ranked near zero.

Faithful means the explanation reflects what the model actually computed for that row. Plausible means it matches what a human expects. An explanation can read convincingly while the model keys on something else.

A feature's Shapley value is its average marginal contribution: for every ordering of the features, measure how much adding that feature moves the prediction away from a baseline, then average those changes across all orderings.

A local surrogate perturbs the row being explained, labels each perturbation with the black box's prediction, weights it by closeness to that row, then fits a small interpretable model whose coefficients are the explanation.

A model card is a short document shipped with a model stating what it is for, what it must not be used for, who and what the training data represented, how it performs on relevant subgroups, and its known limits.

Both changes may flip the trust model's decision, but a counterfactual is only useful if the seller can act on it. Three fewer late shipments is a small change on one controllable feature; 400 extra lifetime orders may take years.

They measure different things. Built-in importance is read off the training-time structure of the fit in split-criterion units; permutation importance measures the drop in an evaluation metric on data you choose. Different quantity, different data, different ranking.

Perturbing features one at a time builds rows no real case resembles, so the model is queried where it never learned. Redundant columns also share the credit, and how it splits is a property of the method.

Efficiency makes the attributions sum exactly to the prediction minus the baseline; symmetry gives interchangeable features equal credit; dummy gives an unused feature zero; additivity makes attributions of summed models add. Together they pick out one rule.

When two groups have different base rates, a score meaning the same thing in both must produce different error rates. Equal predictive value and equal false-positive rates cannot both hold, so the two criteria genuinely conflict.

Rarely, and never as an engineering decision alone. Per-group thresholds act directly on the measured gap but read the protected attribute at decision time, which is explicit differential treatment. Ship one only with counsel's sign-off, a recorded group attribute, and monitoring.

Not established. With 38 people, a 95% interval on that group's pass rate spans roughly 30 percentage points, and the interval on the 12-point gap itself runs from about minus 4 to plus 29 points. Report intervals, not point estimates.

Stores respond differently and the responses cancel. Partial dependence is the pointwise average of the per-row ICE curves, so steeply rising stores and falling or flat stores average to a flat line. The feature matters — just not uniformly.

A global surrogate is trained on the black box's own outputs, so it may be quoted only when it reproduces that model closely on held-out inputs from the deployment population, and closely within each segment you discuss.

The rule is a true pattern in the data and a lethal policy. Asthmatic pneumonia patients were routed straight to intensive care, so they survived more often. The label reflects the treatment they received, not their underlying risk.

The model memorised agent ID on the rows it was fitted to. Shuffling it on training data destroys that memorised fit, so the score collapses; on unseen rows the memorisation was never worth anything, so scrambling it costs nothing.

Separate estimator noise from real ambiguity: rerun across many seeds, measure rank agreement, and raise the sample budget until each spread is small next to the gaps you rank.

Sample random feature orderings instead of enumerating all coalitions: average each feature's marginal contribution over the sampled orderings. The estimate is unbiased, and its error falls roughly as one over the square root of the number of draws.

The kernel width sets how local the explanation is. Too wide, and the surrogate drifts toward the model's average behaviour; too narrow, and nearly all weight falls on a few near-identical points, so a handful of samples decide the answer.

Not from those two numbers alone. Convert both sides into decision consequences and money, check the tradeoff curve for a cheaper knee, confirm the gap is real and not an artefact of biased labels, and put the call to a named accountable group rather than deciding it as an engineer.

Decide from what the decision requires, not the metric gap: ship the readable scorecard when the logic must be signed off, contested or hand-edited, after confirming the 0.02 AUC difference is real out of time.