Features, Directions, and Superposition

Most interpretability work answers the question which inputs mattered? Attribution methods like LIME, SHAP, and Integrated Gradients score input features against a prediction, and they are genuinely useful for auditing decisions.

Mechanistic interpretability asks a harder question: what algorithm is the network running? The goal is to recover the internal computation as something a human can read, name, and verify, in the way you might recover the logic of a compiled binary by reading its disassembly. Christopher Olah's group framed the ambition in the Distill article "Zoom In: An Introduction to Circuits" (2020): treat the network as an object to be reverse engineered rather than a black box to be probed from outside.

The difference matters for safety. An attribution map tells you a loan model leaned on income. It cannot tell you whether a model contains machinery for deception, because that machinery would not announce itself in the inputs.

The working unit of mechanistic interpretability is the feature: a property of the input the network has learned to represent, such as "this text is in French", "the current token is inside a quotation", or "the subject of the sentence is a US city".

The central empirical claim, usually called the linear representation hypothesis, is that features correspond to directions in activation space. If a feature is present, the activation vector has a large component along that feature's direction. Two consequences follow immediately.

First, the magnitude along a direction encodes how strongly the feature is present, so features are graded rather than binary. Second, and more importantly, a direction need not line up with any coordinate axis. Nothing forces the "French text" direction to be neuron 1,742. It can be, and usually is, a diagonal combination of thousands of neurons.

That single observation is what breaks naive neuron-by-neuron analysis.

If you open a language model and read individual neurons, you find a frustrating pattern. A few are clean: a neuron that fires on Python code and nothing else. Most are polysemantic, firing on collections of concepts with no obvious relationship, for example academic citations, Korean text, and images of cats.

This is not noise or a training failure. Polysemantic neurons are reliable, reproducible, and appear across architectures and scales. Early circuits work in vision models by Olah and colleagues documented them thoroughly, and they are the single biggest obstacle to reading a network directly.

The practical damage is severe. You cannot summarize a model by labelling its neurons, because most labels would be lists of unrelated things. You cannot safely ablate a neuron to remove a behaviour, because you would remove several unrelated behaviours at once. Any technique that assumes one neuron equals one concept is building on sand.

The obvious question is why training produces this mess. The answer turns out to be an efficiency argument.

The superposition hypothesis says a network represents more features than it has dimensions, by storing them in overlapping, non-orthogonal directions. Nelson Elhage and colleagues at Anthropic laid this out in "Toy Models of Superposition" (2022), building on the earlier circuits work.

The argument is about scarcity. A model layer might have 4,096 dimensions but need to represent millions of distinguishable concepts. Only 4,096 mutually orthogonal directions exist. But if you accept almost orthogonal directions, the capacity is enormous, and the interference between features stays tolerable as long as few features are active at once.

That last condition is the key: natural data is sparse. Any given token is not simultaneously French, a legal citation, a chemical formula, and a street address. Because features rarely co-occur, the model can overlap them cheaply and pay the interference cost only in the rare collisions.

Superposition is therefore not a bug. It is compression the network chose because it pays.

The capacity claim is not hand waving. It follows from a classical result, the Johnson-Lindenstrauss lemma, which says the number of nearly orthogonal directions available in a space grows exponentially with its dimension, not linearly.

You can verify the intuition in a few lines:

import numpy as np

d, n = 512, 20000            # 512 dimensions, 20k random features
V = np.random.randn(n, d)
V /= np.linalg.norm(V, axis=1, keepdims=True)

sims = V[:2000] @ V[:2000].T  # pairwise cosine similarities
np.fill_diagonal(sims, 0.0)

print(np.abs(sims).max())     # ~0.20: worst-case overlap
print(np.abs(sims).mean())    # ~0.035: typical overlap

Twenty thousand directions in a 512-dimensional space, roughly forty times more features than dimensions, and the typical pair still overlaps by only a few percent. A downstream layer reading one direction picks up a little contamination from the others, and if only a handful of features are active at a time, that contamination stays below the threshold where it changes the answer.

The chain of reasoning runs in one direction: the model has more concepts to represent than dimensions available, so it overlaps them, so any single neuron participates in many features at once, so reading neurons individually yields nonsense. The fix is not to read harder, it is to change the unit of analysis by decomposing activations back into features.

The Anthropic toy models made superposition concrete by training tiny autoencoders to compress more input features than they had hidden dimensions, then reading off exactly what the network did. Three findings stand out.

Sparsity is the control knob. With dense features, the model behaves as linear algebra says it should: it keeps the most important features orthogonally and discards the rest. As features get sparser, it progressively packs more of them in superposition. The transition is a genuine phase change, not a smooth ramp.

The geometry is structured. Features in superposition do not scatter randomly. They arrange into regular configurations, pairs, triangles, pentagons, and higher-dimensional analogues such as tetrahedra, that maximise the angles between them.

Importance shapes the outcome. Features the loss cares about most get cleaner, closer-to-orthogonal directions; marginal features get squeezed into whatever room is left.

A toy model is not a language model, but it demonstrated the mechanism is real and predictable rather than speculative.

One refinement matters in practice: not every part of a transformer is equally likely to hide features along the axes.

A privileged basis exists wherever an elementwise nonlinearity acts, because a function like ReLU or GELU treats coordinates individually and so gives the axes a special status. MLP hidden layers have a privileged basis, which is why neuron-level interpretations sometimes work there.

The residual stream has no privileged basis at all. Nothing in a transformer applies an elementwise nonlinearity directly to it; components only read from it and write to it through learned linear maps. You could rotate the entire residual stream basis and, with matching rotations of the weights, get an identical model. Expecting residual stream dimension 300 to mean something is therefore a category error.

The practical rule: in the residual stream, always work with learned directions rather than coordinates. Even in MLP layers, treat a clean-looking neuron as a lucky exception to check, not the default.

It is worth fixing the contrast between the naive view and the superposition view, because almost every technique in this field is a response to the right-hand column.

Two gotchas are worth stating plainly. First, a neuron that looks clean on the inputs you tried may still be polysemantic on inputs you did not try; selective evidence is the most common error in this area. Second, superposition is a hypothesis with strong support, not a proven theorem, and the field continues to test where it holds.

What the hypothesis buys you is a research programme: if features are overlapped directions, the job is to find the overlap and undo it.

Two threads run out of this lesson, and they organise the rest of the path.

The first is composition. Features are the vocabulary, but a vocabulary is not an algorithm. The next lesson looks at how attention heads and MLPs read and write features to each other through the residual stream, forming circuits, and works through the best-understood example, the induction head that underpins in-context learning.

The second is decomposition. If features are overlapping directions, recovering them is an unsupervised learning problem in its own right: find an overcomplete dictionary of directions such that any activation is a sparse combination of a few of them. That is exactly the sparse autoencoder, and it is the reason the field moved from hand-analysis of small models to extracting millions of features from production systems.

Both threads then have to answer the same challenge: an interpretation is a claim about causation, and claims about causation need experiments, not just plausible stories.