Circuits: Reverse-Engineering Transformer Algorithms

The previous lesson established the vocabulary: features are directions. This lesson is about grammar, how components combine those features into an algorithm.

The key reframing comes from "A Mathematical Framework for Transformer Circuits" (Elhage and colleagues, 2022). Stop thinking of a transformer as a stack of layers that transform a hidden state, and start thinking of it as components communicating over a shared bus.

Every attention head and MLP reads from the residual stream, computes something, and writes its result back by addition. Nothing is ever overwritten; the stream only accumulates. A component in layer 20 can therefore read information written by a head in layer 2, as long as no later component subtracted it.

This has a strong consequence. Two components that never appear in the same layer can still be functionally connected, because one writes a direction the other reads. The layer index is bookkeeping, not architecture. The real structure is who talks to whom.

The same framework decomposes each attention head into two independent pieces, which is what makes heads analysable at all.

The QK circuit decides where to look. It combines the query and key matrices and determines, for each destination token, which source tokens get attention. It controls the attention pattern and nothing else.

The OV circuit decides what to move. It combines the value and output matrices and determines what gets copied from the attended source position into the destination's residual stream. It controls the content and nothing else.

The separation is clean because these two paths do not interact inside the head. You can ask "which tokens does this head attend to?" and "what does it write when it attends?" as two independent questions, then answer each empirically.

Once you can characterise a head this way, composition becomes readable. A head whose OV circuit writes a direction that another head's QK circuit reads is a head that steers where the second one looks. That is a two-step algorithm, spelled out in weights.

A circuit is a subgraph of the network, a specific set of components and the connections between them, that implements an identifiable behaviour.

Three properties make the definition useful rather than decorative.

A circuit is sparse. It typically involves a few dozen components out of many thousands, which is why circuit-level descriptions are short enough for a human to hold.

A circuit is causal. The claim is not that these components correlate with the behaviour, but that intervening on them changes it. This has to be tested, and the next lesson covers how.

A circuit is partial. Naming a circuit for a behaviour does not mean nothing else contributes. Real networks contain backup components that take over when the primary ones are disabled, so "the circuit" is usually the dominant path rather than the only one.

The strongest evidence for the whole framing is that circuits found this way generalise: the same motifs turn up in different models trained on different data.

The best-understood circuit in any language model solves a simple problem: continue a repeated pattern. If a sequence contains ... [A] [B] ... [A], predict [B].

Concretely, if a document mentioned "Dr. Emily Rasmussen" earlier and now says "Dr. Emily", the model should predict "Rasmussen", a name it may never have seen in training. Pure memorisation cannot do this. The model must copy from context.

The solution, identified by Elhage and colleagues and studied in depth by Catherine Olsson and colleagues in "In-context Learning and Induction Heads" (2022), uses two heads in different layers.

The previous-token head comes first. At each position it attends to the token immediately before and writes that token's identity into the current position's residual stream. After it runs, every position carries a record of what preceded it.

The induction head comes second, and its query is the current token [A]. Thanks to the first head, the position holding [B] is now labelled "preceded by [A]". So the induction head searches for that label, finds [B], and copies it to the output.

The induction circuit is the canonical example of composition: one head creates the information a later head queries against. Neither head can do the job alone, and the connection between them exists through the residual stream rather than through adjacency in the layer stack.

Copying a repeated name is a small trick. The reason induction heads became central is the argument that they underpin in-context learning generally.

Olsson and colleagues assembled several lines of evidence. The most striking is a training-dynamics observation: transformers show a sharp, narrow phase change early in training where in-context learning ability jumps. Induction heads form at exactly that moment. The correlation holds across model sizes, and ablating the heads damages in-context performance.

The generalisation argument is that the induction mechanism is not restricted to exact token matches. A head implementing "find something similar to my current state, and copy what followed it" is doing fuzzy pattern completion, which is a plausible substrate for few-shot prompting. Show a model three translation pairs and ask for a fourth, and the abstract shape of the task is the same.

The honest caveat: induction heads are demonstrably a major mechanism behind in-context learning, and describing them as the complete explanation overstates what the evidence supports.

Induction heads are two components. The largest hand-analysed circuit for a natural task is bigger and messier.

Kevin Wang and colleagues, in "Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small" (ICLR 2023), reverse engineered how the model completes sentences such as "When Mary and John went to the store, John gave a drink to ___", where the answer is "Mary".

The algorithm the model implements has three stages. It identifies the names present, detects that one name is duplicated, and then inhibits that duplicate so the remaining name gets written to the output.

The implementation spans 26 attention heads in 7 functional classes, including duplicate-token heads, S-inhibition heads that suppress the repeated name, and name-mover heads that copy the surviving name to the prediction.

Two details are the real lesson. There are negative name-mover heads that actively push against the correct answer. And there are backup name-mover heads that do nothing normally but take over when the primary movers are ablated.

The workflow is consistent across the examples above, and it is worth seeing as a procedure rather than a narrative.

1. Pick a narrow behaviour with a clean metric.
   IOI: logit difference between the correct and repeated name.

2. Build a contrast pair.
   A prompt where the behaviour occurs, and a minimally
   different prompt where it does not.

3. Localise.
   Replace each component's activation with its value from the
   contrast prompt. Components whose replacement collapses the
   metric are in the circuit.

4. Characterise.
   For each surviving head, read its attention pattern (QK) and
   what it writes (OV). Name its role.

5. Verify.
   Knock out everything outside the circuit. If the behaviour
   survives, the circuit is sufficient, not just correlated.

Step 3 is activation patching, the subject of the next lesson. Step 5 is the one most often skipped, and it is the one that separates a demonstrated circuit from a plausible diagram. Arthur Conmy and colleagues later automated much of steps 3 and 4 in "Towards Automated Circuit Discovery" (2023), because doing it by hand does not scale.

Circuits gave the field its first genuine successes, and their limits are just as instructive.

The scale problem is the binding one. IOI took skilled researchers substantial effort to characterise one behaviour in a 124-million-parameter model. Frontier models are four orders of magnitude larger and have effectively unbounded behaviours to explain.

The MLP problem compounds it. Attention heads move information between positions and have a clean two-part decomposition. MLPs hold roughly two-thirds of the parameters, and superposition hits them hardest, so their neurons resist direct reading.

Both limits point the same way: circuit analysis needs a scalable method for turning MLP activations into interpretable units first. That is the sparse autoencoder.