Explainable AI: The Landscape of Model Explanations

A gradient-boosted model can rank credit applicants better than any scorecard a human would write, and still leave you unable to answer a simple question: why was this application rejected? Accuracy is a property of the model over a population. Justification is a property of a single decision, made to a single person, and the two are not the same thing.

Explainable AI, usually shortened to XAI, is the set of techniques that recover an answer to that question from a model that was never designed to give one. The demand comes from four directions: debugging (is the model right for the right reason?), trust (should a doctor override it?), regulation (can you document the decision?), and discovery (what did the model find in the data that we did not know?).

Each of those four wants a different kind of answer. That is why the field is a landscape, not a single method.

The first fork is whether the model explains itself.

An intrinsically interpretable model has a structure a person can read directly. A linear model gives you a coefficient per feature. A short decision tree gives you a path of conditions. A generalized additive model gives you one shape function per feature that you can plot. Nothing is recovered or approximated, because the explanation is the model.

A post-hoc explanation is built after the fact, on top of a model whose internals are either opaque or too large to read. A deep network with a hundred million parameters has no readable structure, so you probe it: perturb the input, take gradients, fit a small surrogate, and report what you learned.

The trade is usually framed as accuracy versus interpretability, but that framing is contested. It holds for perception tasks on raw pixels or audio. On structured tabular data, a well-regularized interpretable model is often within noise of a black box.

Two more axes finish the map.

Scope. A global explanation describes the model's behaviour over the whole data distribution: which features matter on average, what shape the relationship takes. A local explanation describes one prediction: why this applicant, this scan, this transaction. A feature can be globally useless and locally decisive, so a global answer never substitutes for a local one.

Access. A model-specific method exploits internal structure: TreeSHAP walks the split structure of a tree ensemble, Grad-CAM reaches into the last convolutional layer. A model-agnostic method treats the model as a function you can call, and only needs the ability to feed it inputs and read outputs.

Model-agnostic methods are portable across model families and let you swap the model without rewriting your explanation pipeline. Model-specific methods are faster and often more exact, because they use information the black-box view throws away.

The three axes compose into a decision you make before choosing any tool. First ask whether you can use an intrinsically interpretable model at all, because if you can, most of what follows is unnecessary. If you cannot, ask whether the question is about the model as a whole or about one prediction. Then ask whether you have access to internals.

The leaves are the methods this cursus covers. Permutation importance, partial dependence, and ICE answer global questions. LIME, SHAP, Integrated Gradients, and Grad-CAM answer local ones. Counterfactual explanations sit slightly apart, because they answer a different question again: not which feature mattered, but what would have to change.

The oldest global model-agnostic method is also the most intuitive. Break a feature and see how much the model suffers.

Concretely: measure the model's error on a held-out set. Then shuffle one feature's column, destroying its relationship with the target while leaving its marginal distribution intact. Measure the error again. The increase is that feature's importance. Repeat for every feature, and average over several shuffles because the shuffle is random.

Breiman introduced it in 2001 for random forests. Fisher, Rudin and Dominici generalized it to any model in 2019 under the name model reliance.

Two warnings. It measures importance to a particular fitted model, not importance in the world, so two models trained on the same data can disagree. And with correlated features it shuffles one and leaves its twin intact, so the model compensates and both look unimportant, an effect that splits credit unpredictably.

Permutation importance takes a few lines with scikit-learn, and the shape of the call tells you what the method needs: a fitted model, held-out data, and a scoring function.

from sklearn.inspection import permutation_importance

r = permutation_importance(
    model, X_test, y_test,
    scoring="roc_auc",
    n_repeats=30,          # average over 30 shuffles
    random_state=0,
)

for i in r.importances_mean.argsort()[::-1]:
    if r.importances_mean[i] - 2 * r.importances_std[i] > 0:
        print(f"{X_test.columns[i]:<20} "
              f"{r.importances_mean[i]:.4f} +/- {r.importances_std[i]:.4f}")

Two details carry the weight. Using X_test rather than training data is what makes the number about generalization instead of memorization. And the filter on mean minus two standard deviations discards features whose apparent importance is inside the noise of the shuffling itself, which for weak features is most of it.

Importance tells you a feature matters. It does not tell you which direction, or whether the relationship bends.

A partial dependence plot answers that. Pick a feature and a grid of values. For each grid value, overwrite that feature for every row in the dataset, predict, and average. Plot the average against the grid value. What you get is the marginal effect of the feature, with the other features integrated out. Friedman introduced it in 2001.

Averaging hides disagreement. If the feature raises the prediction for half the population and lowers it for the other half, the partial dependence curve is flat and you conclude, wrongly, that nothing is happening.

Individual conditional expectation plots, introduced by Goldstein and colleagues in 2015, fix this by drawing one line per row instead of the average. Fanning lines mean an interaction; parallel lines mean the partial dependence curve is trustworthy.

Permutation importance and partial dependence share a mechanism, and therefore share a failure.

Both work by replacing a feature value with a value taken from somewhere else: a shuffled row, or a grid point. Both then feed the resulting row to the model as if it were real. With correlated features, that row often is not real. Overwrite height with values from a grid while leaving weight untouched and you will ask the model to score a person two metres tall weighing forty kilograms.

The model still returns a number, because models always return a number. But that number comes from a region of input space with no training data in it, where the model was never fitted and its behaviour is an extrapolation artifact.

This off-manifold problem is not a quirk of two old methods. It recurs, in the same form, in LIME and in KernelSHAP, and it is the single most common reason a published feature attribution is wrong.

Two criteria are constantly confused, and separating them is the most useful habit in this field.

Faithfulness asks whether the explanation reflects what the model actually computed. It is a claim about the model, and it can in principle be tested: remove the features the explanation called important and the prediction should move.

Plausibility asks whether the explanation looks reasonable to a human. It is a claim about the reader.

The two come apart in both directions, and the dangerous direction is a plausible explanation of an unfaithful kind. A model that classifies wolves by detecting snow in the background can be handed an explanation that highlights the wolf, and a reviewer will nod. Optimizing an explanation method for human approval selects directly for this failure. A convincing explanation is evidence about the explanation method, not about the model.

In 2019 in Nature Machine Intelligence, Cynthia Rudin argued that explaining black boxes for high-stakes decisions is the wrong response to the problem, and that interpretable models should be used instead.

The argument has three parts. First, a post-hoc explanation is by construction an approximation of the model; if it were exact it would be the model. So it can be wrong, and it is wrong in ways nobody can detect from the explanation alone. Second, the accuracy cost of interpretability is often assumed rather than measured, and on structured data with meaningful features it frequently turns out to be small. Third, the availability of explanation tools removes the pressure to build something interpretable in the first place.

The counter-position is practical: many deployed models are not yours to redesign, and an approximate explanation beats none. Both positions are held by serious people. Knowing that the debate exists is part of using the tools honestly.

A short guide to the global tools, before the next lessons take on local ones.

Permutation importance: use to rank features and to catch leakage, when a suspiciously dominant feature turns out to encode the label. Cheap. Unreliable under correlation.

Partial dependence: use to see the shape and direction of one feature's effect. Always pair with ICE, because the average alone can be a lie.

Global surrogate, meaning fit a short decision tree to the black box's predictions: use for a rough map of the whole model. Report how well the surrogate matches, and treat a poor match as a signal to stop.

And the prior question that outranks all three: could an interpretable model do this job? Answer it with a measurement, not an assumption. Fit the regularized linear model or the small gradient-boosted tree with monotonic constraints, and compare.