interpretability

How an unsupervised model trained on activations pulls features back out of superposition, what it found when scaled to a production language model, and where the method still breaks down.

Why individual neurons are the wrong unit for understanding a neural network, and what the superposition hypothesis says is really going on inside the activations.

Two methods dominate local explanation, and both perturb the input. LIME fits a small interpretable model near one prediction. SHAP borrows the Shapley value from game theory and is the unique attribution satisfying local accuracy, missingness and consistency. This lesson builds both mechanisms, compares KernelSHAP with TreeSHAP, and is precise about what a SHAP value does not mean.

A model that predicts well can still be impossible to justify. This lesson maps explainable AI: interpretable-by-design versus post-hoc, global versus local, model-specific versus model-agnostic. It covers the global workhorses (permutation importance, partial dependence, ICE), faithfulness versus plausibility, and the argument that post-hoc explanation is the wrong tool for high-stakes decisions.

When a model is differentiable you can attribute a prediction with calculus instead of sampling. This lesson covers plain saliency, Integrated Gradients and its axioms, Grad-CAM, then counterfactual explanations, which answer what would have to change rather than what mattered. It closes with the evidence that explanations can fail or be deliberately faked, and a protocol for checking yours.

Why neurons are polysemantic, how the superposition hypothesis explains it, and how sparse autoencoders use dictionary learning to pull a model's activations apart into monosemantic, steerable features, plus the failure modes and the top-k and gated fixes.