Gradients, Counterfactuals, and Whether to Trust an Explanation

LIME and SHAP treat the model as a sealed box you can only query. For a neural network that throws away the most useful thing you have: the gradient.

A single backward pass gives the partial derivative of the output with respect to every input coordinate at once. For a 224 by 224 image that is 150,528 sensitivity numbers for roughly the cost of one forward pass, against the thousands of forward passes a perturbation method would need. The efficiency difference is not marginal, it is what makes attribution practical on images and text at all.

The price is that gradient methods are model-specific in the weak sense: they need a differentiable model and access to its backward pass. They will not explain a random forest or an external API.

What follows is the progression from the naive use of that gradient to the version with guarantees.

The obvious move, taken by Simonyan and colleagues in 2013, is to use the gradient magnitude directly. Large derivative, important pixel.

It produces recognizable but noisy maps, and it has a specific structural flaw called saturation. Consider a network computing something like 1 minus ReLU(1 minus x). As x rises from 0 to 1 the output rises with it. Past x equal to 1 the output is pinned at 1 and the gradient is exactly zero.

So at x equal to 2, plain saliency reports that x has zero importance. But x is the entire reason the output is 1. The gradient answers a local question, how the output responds to a tiny nudge here, and mistakes it for a global one, how much this feature contributed to getting here.

Any feature the model has already fully committed to becomes invisible. That is precisely the feature you most wanted to see.

Sundararajan, Taly and Yan proposed the fix in 2017. If the gradient at the input is uninformative because the model saturated, do not evaluate it only at the input. Evaluate it everywhere along the way.

Pick a baseline representing absence: a black image, an all-zeros vector, a sequence of padding tokens. Draw the straight line from baseline to your actual input. Walk along it, taking the gradient at each point, and integrate. Multiply the integral by the difference between input and baseline.

In the saturating example, the gradient is zero at x equal to 2 but was equal to 1 for the whole stretch from 0 to 1, so the path integral recovers the contribution the endpoint gradient missed. In practice the integral is a Riemann sum, typically over 50 to 300 interpolation steps.

Integrated Gradients was derived from properties rather than tuned to look good, and the paper's title, Axiomatic Attribution for Deep Networks, says so.

Sensitivity: if an input differs from the baseline in one feature and the predictions differ, that feature must get nonzero attribution. Conversely, a feature the model does not functionally depend on must get zero. Plain saliency fails the first half, as the saturation example showed.

Implementation invariance: two networks that compute the same function for every input must get the same attributions, whatever their architectures. This rules out methods that read internal activations in architecture-dependent ways, which several backpropagation-modifying methods do.

Completeness: the attributions sum exactly to the model's output at the input minus its output at the baseline. Nothing is lost or invented, and completeness implies sensitivity, so it is the stronger statement.

The parallel with SHAP is not coincidental. Integrated Gradients is the Aumann-Shapley value from game theory applied to a continuous input.

The whole method fits in a dozen lines, which is a fair test of whether you followed the derivation.

def integrated_gradients(model, x, baseline, target, steps=100):
    alphas = torch.linspace(0, 1, steps).view(-1, 1, 1, 1)
    path = baseline + alphas * (x - baseline)   # steps points on the line
    path.requires_grad_(True)

    out = model(path)[:, target].sum()
    grads = torch.autograd.grad(out, path)[0]   # gradient at every step

    avg_grad = grads.mean(dim=0)                # the Riemann sum
    return (x - baseline) * avg_grad

One line is a built-in test. Because of completeness, the returned attribution should sum to model(x)[target] - model(baseline)[target]. If it does not, your step count is too low. Checking that residual on every run is the cheapest correctness guard in explainability, and it is routinely skipped.

Completeness holds against whatever baseline you pick, which means the baseline silently defines what the attribution is relative to. It is not a technical detail.

The common default for images is a black image. It is a bad default, because black is a real colour: genuinely black pixels in your input sit at the baseline and receive near-zero attribution no matter how much the model relied on them. A white baseline has the mirror problem.

The alternatives each encode a different meaning of absence. A blurred version of the input isolates high-frequency content. Gaussian noise removes signal without privileging a colour. Averaging over several random baselines, which is the Expected Gradients variant, removes the dependence on any single choice at extra cost. For text, the padding or mask token is the natural absence.

The rule is to state the baseline whenever you report attributions. Two Integrated Gradients maps with different baselines are answering different questions.

Pixel-level maps are noisy because individual pixels are not what a convolutional network reasons about. Grad-CAM, from Selvaraju and colleagues in 2017, attributes at the level of the last convolutional layer instead.

The mechanism has two steps. Take the gradient of the target class score with respect to each feature map in that layer and average it spatially; this gives one weight per channel, a measure of how much that channel matters to the class. Then form the weighted sum of the feature maps, apply a ReLU to keep only positive evidence, and upsample the result to the input size.

The output is a coarse heatmap over regions, not pixels. That coarseness is deliberate. Late convolutional layers retain spatial structure while encoding semantic content, so the heatmap answers where in the image the evidence for this class lives, which is usually the question being asked.

Grad-CAM also passes the sanity checks in the next step, which many pixel-level methods do not.

Every method so far answers which features mattered. A person denied a loan usually wants something else: what do I need to change?

Wachter, Mittelstadt and Russell proposed counterfactual explanations in 2017, in the Harvard Journal of Law and Technology, arguing they fit the transparency aims of the GDPR without requiring the model to be opened at all. A counterfactual is the nearest input that flips the decision. Your application was declined; had your annual income been 41,000 instead of 38,000, with everything else unchanged, it would have been approved.

The advantages are concrete. It is actionable, giving a target rather than a diagnosis. It is contrastive, matching how people naturally explain things, by reference to what did not happen. It reveals nothing about the model's internals, so it can be given out without exposing intellectual property. And it is verifiable: run the counterfactual through the model and check the decision actually flips.

Generating one is a constrained optimization, and the constraints are where the design lives.

The optimization trades off several requirements, and dropping any one of them produces a technically valid answer that is useless to the person receiving it.

Validity: it must actually flip the model's output. Non-negotiable and easy to check.

Proximity: it should be close to the original input, so the change asked for is small.

Sparsity: it should alter few features. A counterfactual changing eleven things at once is not advice.

Actionability: it must only change features the person can change. Increase your income is advice; be five years younger is not, and features like age, race or place of birth must be fixed as immutable in the optimizer.

Plausibility: it must land on the data manifold. An income of 41,000 with a listed occupation of student may flip the model while describing a person who does not exist.

When a system generates counterfactuals for people to act on, the field calls it algorithmic recourse, and the surveys note that recourse is only meaningful if acting on it really changes the outcome.

Adebayo and colleagues published Sanity Checks for Saliency Maps at NeurIPS in 2018, and the test they proposed is almost embarrassingly simple.

The model randomization test: take a trained network, replace its weights with random ones, and recompute the saliency map. If the map barely changes, the method is not explaining the model. It is doing edge detection on the input.

The data randomization test: retrain the model on data with the labels shuffled, so it can only memorize. If the map still looks sensible, the method is not explaining what was learned either.

Several widely used methods failed. Some produced visually convincing maps that survived randomizing the weights, which means every paper that had validated them by looking at them had validated nothing. Grad-CAM and Integrated Gradients passed.

The general lesson outlives the specific results: an explanation that looks right is not evidence. Run the randomization test on whatever you deploy.

Sanity checks catch accidental failure. Slack, Hilgard, Jia, Singh and Lakkaraju showed at the AAAI/ACM AIES conference in 2020 that a determined adversary can manufacture a chosen explanation on purpose.

The attack, which they call scaffolding, exploits the one thing LIME and SHAP have in common: both explain by feeding the model perturbed inputs, and those perturbed inputs do not look like real data. So build a classifier that first detects whether an input is a genuine data point or a perturbation. On genuine inputs, run the biased model you actually want to deploy. On perturbations, run an innocuous model that uses only harmless features.

The deployed system stays biased on the real distribution. The explanation, computed entirely from perturbations, reports whatever the innocuous model does. The authors demonstrated hiding reliance on race in a scaffolded classifier.

The implication for governance is sharp. An explanation is not an audit, because a party with an incentive to pass can control what it says.

The methods are useful. Treating their output as ground truth is not. A defensible practice looks roughly like this.

Run more than one method and compare. When Integrated Gradients, SHAP and a counterfactual agree, that agreement is real evidence. When they disagree, and studies of practitioners find they often do, resist picking the one that matches your prior.

Check what is checkable. Completeness residual for Integrated Gradients. Model randomization for any saliency method. Validity for a counterfactual, by rerunning it through the model. Stability, by re-running stochastic methods on the same input.

State the reference. The background set for SHAP, the baseline for Integrated Gradients, the kernel width for LIME. An attribution without its reference point is not interpretable.

And keep the prior question from the first lesson alive. If the decision is high-stakes and the explanation is load-bearing, an interpretable model removes the need to trust any of this machinery.