LIME and SHAP: Attributing a Single Prediction

Global importance tells you that income matters across the portfolio. It does not tell this applicant why they were declined, and it is the second question that people, auditors and regulators actually ask.

A local explanation attributes one prediction to the features of one input. The usual output format is an additive one: a base value plus a signed number per feature, where the numbers sum to the prediction. Income pushed the score up by 0.12, recent delinquency pulled it down by 0.31, and so on.

That format is a strong commitment. It says the prediction can be decomposed into per-feature contributions, which is exactly what a nonlinear model with interactions does not do. Every method in this lesson is, at bottom, a different answer to the same question: given that the decomposition does not really exist, what is the most defensible one to report?

LIME, from Ribeiro, Singh and Guestrin in 2016, rests on one observation. A decision boundary can be hopelessly complicated globally and nearly linear if you only look at a small neighbourhood around a single point.

So do not try to explain the model. Explain the neighbourhood. Generate perturbed variants of the instance you care about, ask the black box to label all of them, weight each variant by how close it is to the original, and fit a sparse linear model to that weighted sample. The coefficients of that little model are the explanation.

The name unpacks the design: Local, because it only claims validity nearby; Interpretable, because the surrogate is a sparse linear model a human can read; Model-agnostic, because the black box is only ever called, never inspected. LIME will explain a random forest, a transformer, or a hand-written rule engine with the same code.

Five steps, in order.

First, choose an interpretable representation. For tabular data that is usually the features themselves, but for text it is the presence or absence of each word, and for images it is the presence or absence of each superpixel, a contiguous patch found by segmentation. The surrogate is fitted in these binary on/off coordinates, not in raw pixel space, which is what makes its coefficients readable.

Second, sample perturbations by switching components off. Third, run every perturbation through the black box to get its label. Fourth, weight each sample by an exponential kernel on its distance to the original instance, so near neighbours dominate the fit. Fifth, fit a sparse linear model, typically with a limit of five or ten nonzero coefficients, and report those coefficients.

LIME's flexibility comes from free parameters, and free parameters are where the trouble lives.

The kernel width decides what counts as the neighbourhood, and there is no principled way to set it. Widen it and the surrogate averages over territory where the boundary bends, so the explanation drifts toward the global picture. Narrow it and the sample thins out until the coefficients are fitting noise. Published explanations of the same prediction can reverse sign under a plausible change of width.

The sampling is random, so re-running LIME on the identical instance yields different coefficients. Stability has to be checked by repeating the call, not assumed.

And tabular LIME perturbs features independently, which puts it squarely in the off-manifold trap: it asks the model to score combinations that could not occur, then reads the answers as if they were meaningful.

The other family starts somewhere unexpected: cooperative game theory, from Lloyd Shapley in 1953.

Several players cooperate and produce a payout. How should it be divided? Shapley's answer is to consider every possible order in which the players could have joined the coalition, measure how much each player added at the moment they joined, and average that marginal contribution over all orders.

Map it onto a prediction. The players are the features. The payout is the prediction minus the average prediction. A coalition is a subset of features that are present, with the absent ones replaced by values drawn from a background dataset. The Shapley value of feature j is

phi_j = sum over subsets S not containing j of [ |S|! (M - |S| - 1)! / M! ] * [ f(S union {j}) - f(S) ]

where M is the number of features and f is the model's expected output given that subset.

The weighting looks arbitrary until you see what it buys. The Shapley value is the only attribution satisfying four properties at once, and each maps onto something you would want from an explanation.

Efficiency, called local accuracy in the SHAP paper: the contributions plus the base value sum exactly to the prediction. Nothing is unexplained and nothing is invented.

Symmetry: two features that add the same amount to every coalition receive the same value. Order of columns cannot matter.

Dummy, called missingness: a feature that changes no coalition's output gets exactly zero. An irrelevant feature cannot be credited.

Additivity, extended to consistency: if you change the model so a feature's marginal contribution never decreases, its attributed value cannot decrease either. Without this, a model that leans harder on a feature can be reported as leaning less, which is the failure that sinks several earlier heuristics.

Uniqueness is the point. Any method violating one of these is not a rival, it is wrong on that axis.

Lundberg and Lee's 2017 contribution was not the Shapley value, which is decades old. It was showing that a whole class of existing methods were secretly the same shape.

They defined additive feature attribution methods as those whose explanation is a linear function of binary present/absent indicators: g(z') = phi_0 + sum of phi_j times z'_j. LIME fits this form. So does DeepLIFT. So does layer-wise relevance propagation. Each was proposed separately, with its own justification.

The result is that within this class, exactly one choice of coefficients satisfies local accuracy, missingness and consistency, and it is the Shapley value. The other methods are then not alternatives to SHAP but approximations of it, each with its own way of departing from the axioms.

That reframing is what made SHAP the default. It replaced a pile of competing heuristics with one target and a set of estimators for it.

The formula sums over every subset of features, so an exact computation costs 2 to the power M model evaluations. At thirty features that is over a billion. The two standard estimators attack this differently.

KernelSHAP is model-agnostic. It reuses the LIME machinery, sampling coalitions and fitting a weighted linear regression, but replaces LIME's arbitrary proximity kernel with the specific SHAP kernel whose regression coefficients are provably the Shapley values. Cost is stated as order T times L times 2 to the M. Correct in the limit, slow in practice, and one instance at a time.

TreeSHAP is model-specific to tree ensembles. It exploits the fact that a tree only branches on a few features, so most coalitions cannot change the outcome. Complexity drops to order T times L times D squared, where T is the number of trees, L the maximum leaves in any tree, and D the maximum depth. Exact, and fast enough to explain a whole dataset.

The library separates the estimator from the plotting, and picking the right estimator is most of the work.

import shap

# Tree ensemble: exact and fast, explain the whole test set
explainer = shap.TreeExplainer(model, data=X_train)
sv = explainer(X_test)              # sv.values, sv.base_values

# Any model: sample coalitions, use a SMALL background set
background = shap.sample(X_train, 100)
explainer = shap.KernelExplainer(model.predict_proba, background)
sv = explainer.shap_values(X_test[:50])   # 50 rows, not 50000

shap.plots.waterfall(sv[0])   # one prediction, decomposed
shap.plots.beeswarm(sv)       # global view, built from local values

The background set is the load-bearing argument. It defines what "absent" means: absent features are filled from it. Pass the full training set to KernelExplainer and every explanation costs a pass over it. Pass an unrepresentative sample and every attribution is measured against the wrong reference point.

The axioms guarantee a fair split of the model's output. They guarantee nothing about the world, and four misreadings follow.

It is not causal. A SHAP value says how the model's output moves as a feature enters the coalition. It does not say what happens if you change the feature in reality. A model can lean on a proxy for a cause and SHAP will faithfully report the proxy.

It is not feature importance in the usual sense. It is importance to this prediction, against this background set. Change the background and the numbers change.

It is not immune to correlation. KernelSHAP samples absent features from their marginal distribution, so, exactly like permutation importance, it builds unlikely rows and can inflate attributions. TreeSHAP's path-dependent variant takes the opposite risk: it can assign nonzero value to a feature with no influence at all, purely because it correlates with an influential one.

And summing local values into a global ranking is a convention, not a theorem.

A direct comparison, on the dimensions that decide it in practice.

Theoretical grounding: SHAP has uniqueness under four axioms. LIME has a fitted local surrogate and no guarantee that contributions sum to the prediction.

Stability: SHAP is deterministic for TreeSHAP and converges with samples for KernelSHAP. LIME varies run to run and with the kernel width.

Speed: TreeSHAP on a tree ensemble is the fastest option available and scales to a full dataset. LIME needs a few thousand model calls per instance. KernelSHAP is the slowest.

Coverage: LIME's interpretable representation handles text and images naturally through words and superpixels. SHAP has parallel machinery, but the mapping is less direct.

A reasonable default: TreeSHAP for tree ensembles, which is most tabular work. LIME for a fast qualitative look at text or images. KernelSHAP when you need the axioms and the model is neither a tree nor differentiable.