# Paper Summary: Interpreting Language Model Parameters

## Language models and interpretability
Language models are some of the most remarkable computer programs in existence. They implement algorithms humans have tried and failed to write by hand for decades. Yet the "code" these algorithms are implemented by is "*neural* code", which is written, somehow, as enormous, inscrutable matrices of parameters.

As a result, the field of interpretability has focused mainly on trying to understand the models' activations — the models’ “thoughts”. But reading these thoughts doesn’t immediately explain the computations that gave rise to them. To understand them deeply, we should understand not just the inputs and outputs to their computations, but *the computations themselves*.

## Adversarial Parameter Decomposition (APD)
Ad**V**ersarial **P**arameter **D**ecomposition (VPD) is our technique for doing this in a language model. By splitting the model's parameters into simple, understandable pieces, we are able to directly study their structure and the algorithms they implement.

We used VPD to decompose the weight matrices in a 67M-parameter language model. Using this new view into the network's parameters, we can:

- **Identify algorithms implemented in attention layers**, even when they are distributed across attention heads;
- **Edit the original model's behavior with no training** — performing "brain surgery" directly on the model's "neural code";
- **Recover small subnetworks** responsible for specific, abstract behaviors.

We think this is a meaningful step toward a more bottom-up form of interpretability: one that explains computation in the model’s own terms, rather than imposing our own abstractions, top-down.

## VPD, or, How to take apart a neural network
To understand a complex object, it helps to think of it in terms of simpler pieces: cars in terms of parts; bodies in terms of organs; chemicals in terms of atoms. Unfortunately, language models seem messier. The obvious candidates for these pieces, such as attention heads or MLP neurons, often do not cleanly correspond to interpretable functional roles. If simple pieces do exist, they are somehow encoded into the model's millions to trillions of parameters. Our job, then, is to tease apart these pieces.

VPD is an unsupervised process for identifying these pieces using gradient descent. To do so, we need to imagine what properties these pieces would have, so that we can optimize for them.

Firstly, we'd like each one to be mechanistically simple, so that we can understand each in isolation and in combination with each other. To enforce this, we constrain each piece to be a simple (rank one) matrix. These matrices must sum together to recreate the model's original weights.

We'd also like each to perform a specific computational role. The weights of a model might encode that "Paris is the capital of France", or that "The sky is blue". But for prompts where this knowledge isn't needed, we should be able somehow to remove this knowledge from the parameters without hurting performance on those prompts. A good matrix subcomponent, then, is one which is needed only to fulfill a specific role, and can otherwise be removed from the model's parameters without hurting performance.

To define when subcomponents are causally important, we train an auxiliary model, the causal importance network. On any given prompt, the causal importance network predicts the minimum number of subcomponents that are causally important to reproduce the model's behavior on that prompt. Our decomposition should approximate the behavior of the target network regardless of what we do with causally unimportant subcomponents.

## Our Decomposed Language Model
After applying VPD, the obvious first question is whether the resulting parameter subcomponents are actually interpretable objects. Indeed, we find that they often activate in coherent semantic or syntactic contexts.

There are also relatively few subcomponents: Decomposing all 24 matrices in the network, we identify only ~10,000 subcomponents.

### Interpreting Attention Layers
Notably, VPD can decompose attention layers. VPD works on attention natively, finding subcomponents in attention weights the same way it does in any other layer. In our model, this is enough to recover interpretable attention algorithms directly in weight space, including previous-token behavior and a syntax-boundary routing behavior.

### Model Editing
If our parameter subcomponents have cleanly isolated the true mechanisms of the model, we should be able to use them to perform clean, targeted changes. We found a subcomponent responsible for recognizing the eyes of emoticons. We altered this subcomponent to directly output the model’s unembedding vector for `o`, producing a model which predicts all emoticons as shocked faces!

### Describing Model Behaviours with Subnetworks
We can use VPD to study specific model behaviours in terms of the parameter subcomponents the model uses to perform them. For a given next-token prediction, we can identify the necessary subcomponents and compute an attribution graph — a rough approximation of how strongly the subcomponents interact — to trace how the model computes the output.

## Conclusion
Language models' parameters have widely been considered irreducibly, impenetrably complex. We now have evidence that this is not the case; that there is interpretable structure that we can find and understand. Ultimately, we would like to understand neural networks well enough to be able to intentionally design them, so that they have more of the properties we want and fewer of those we do not.