Transcoders find interpretable LLM feature circuits

Thank you for taking the time to review our work. We are very glad to see that you recognize the importance of the input-invariant/input-dependent factorization, along with the power demonstrated by transcoders in reverse-engineering feature circuits in an actual model (GPT2-small). We will now address your questions in turn.

Circuit Analysis: Assuming Transcoders perfectly emulates MLP (which it tries to but as long as faithfulness loss > 0 it doesn't) the circuit analysis graph is between Transcoders in different layers but in the actual model there is multiple-head attention in between each MLP layer which is not accounted for its effect in equation 6.

We show in Appendix D.3 how attribution works through attention heads. We mention this in Line 190 (section 4.1.2), but we agree that this deserves to be moved to the main body of the paper In particular, the contribution of token $s$ at layer $l$ through attention head $h$ to transcoder feature $i’$ at layer $l’ > l$ is given by:

$\text{score}^{(l,h)} \left( x_{pre}^{(l,t)}, x_{pre}^{(l,s)} \right) \left( \left( \left( W_{OV}^{(l,h)} \right)^T f_{enc}^{(l’, i’)} \right) \cdot x_{pre}^{(l,s)} \right)$

Where $\text{score}^{(l,h)}$ denotes the scalar “scoring” function that weights the importance of each source token to the destination token (i.e. the pattern in the QK circuit of attention).

Evaluation: The evaluation was weak at best. The paper did compare with SAE but the only actual quantitative results were the difference between SAE and transcoders in terms of sparsity and faithfulness which only shows the transcoders are on par with SAE. The more important experiment is the interpretability experiment since this is why both methods were developed in the first place for these experiments were not systematic and many of the choices were not justified or explained properly.

Our primary goal in our work was to demonstrate that transcoders enable input-invariant circuit analysis that is not possible with current methods using SAEs. To that end, our sparsity, faithfulness, and interpretability experiments were indeed intended to show that transcoders are on par with SAEs when evaluated on these metrics. Because transcoders also provide additional benefits over SAEs in circuit analysis, this means that you can obtain all of the benefits for circuit analysis with no penalties compared to SAEs.

If there are any specific other experiments you would like to see comparing SAEs and transcoders, we are happy to carry them out during the discussion period. But we would like to reiterate that the most important benefit of transcoders is in their circuit analysis capabilities.

For the blind interpretability experiments why was layer 15 in particular chosen?

We provide our rationale in Appendix F: “Layer 15 was chosen largely heuristically, because we believed that this layer is late enough in the model to contain complex features, while not so late in the model that features are primarily encapsulating information about which tokens come next.”

From the following sentence: "We recorded for each feature whether or not there seemed to be an interpretable pattern, and only after examining every feature did we look at which features came from where." How did you identify that there is an "interpretable pattern"? How many people performed this experiment?

Interpretability scores were assigned on the basis of whether it was possible to form a hypothesis about what the feature was doing on the basis of a sample of activating examples. Due to budget constraints that prevented wider-scale experiments from being carried out, only one person performed the interpretable pattern experiment. We would like to emphasize that features from SAEs and transcoders were shuffled together in this experiment, and the subject was unaware of which was which. Please feel free to refer to Figure 6 in the appendix if you’re interested in seeing examples of interpretable, partially interpretable, and non-interpretable features.

For attribution and graph calculation for a given feature i' at layer l' are all others feature at previous layers considered in the graph or only the same feature at the previous layer?

Equation 6 shows how any upstream feature affects any downstream feature. Because of residual connections in the transformer, this equation is valid for computing the effect of a feature in any previous layer to a feature in any later layer. (In particular, the index $l$ denoting the earlier-layer feature can be any layer less than $l’$ denoting the later-layer feature.) It is also worth noting that because transcoders are trained independently on each MLP layer, and because there is not any inter-layer consistency of MLP features anyway, there is no concept of the “same feature” across different layers.

Again, thank you for taking the time to write your detailed review.

Thank you for taking the time to write your thoughtful review and for recognizing the importance of transcoders to the broader mechanistic interpretability research program. We are also very excited to continue working on transcoders and seeing how far they can go.

Now, we would like to respond to your various questions.

In Appendix E line 1050, they state that the baseline SAEs they compare their results against were trained on MLP inputs rather than MLP outputs. I think this is the wrong comparison point. [...] Training the SAEs on the MLP output activations instead would be a more appropriate comparison. I think this is the biggest weakness of the paper.

This is a piece of feedback that we received from other readers as well post-submission, and we have responded by repeating these experiments for SAEs trained on the MLP output activations. The results were similar: transcoders performed on-par with or better than the SAEs by the sparsity/fidelity metrics. Our updated figure is attached to our author rebuttal in our PDF.

In section 4.3 and figure 5, they use neuron-based analysis as a baseline to compare their transcoder feature-based analysis to. I think this is a very weak baseline. Using e.g. activation PCA components would be more appropriate. This is the main result they have comparing their technique to standard techniques outside the cluster of techniques based on SAEs/sparse decoding, so I think not using an appropriate baseline here is not great. I don't think it is a very major flaw, because this is mainly a paper that takes it as given that SAEs are interesting and attempt to improve on them, rather than a paper trying to justify the sparse decoding approach to interpretability.

Thank you for the suggestion. We avoided looking at PCA components on the activations because to some extent, they suffer from the same problem as SAEs when applied to this task: namely, they do not address the computation carried out by the MLP, but only deal with the intermediate activations of those computations. In carrying out our investigation of the “greater-than” circuit, we wanted to understand how MLP10 in GPT2-small maps year tokens represented in the MLP input space to MLP outputs that boost certain logits. Because PCA (just like SAEs) only handles the intermediate activations of this computation, it is unable to explain this mapping in an input-invariant way. (Note that the original “greater-than” circuit paper also does not use PCA, presumably for similar reasons.)

They describe the attribution method between transcoder features they use as a 'new technique', which I don’t think is justified. It's effectively the attribution patching technique [...] Transcoders just have the advantage of making it work more elegantly. Any ReLU MLP network attribution patching is applied to would yield equation 6 as well.

We are aware of this, and had already modified the paper post-submission to include a footnote explaining that this is a special case of the classic “input-times-gradient” method for computing attributions. (As you note, this ends up being equivalent to attribution patching when looking at a single transcoder feature, rather than a vector of feature activations.)

What we consider to be a contribution of ours is our recognition that input-times-gradient, when applied to ReLU MLPs such as transcoders, does factorize into an input-dependent and an input-invariant term – i.e. the gradient is constant. Furthermore, something that we like to underscore when thinking about Equation 6 is that for transcoders (and this is not true in general for ReLU networks), the input-dependent term (the feature activation) is interpretable (because feature activations are largely interpretable).

Based on your comment, we can modify the paper to further highlight the relationship between Equation 6 and existing attribution methods.

In section 3.2.1 and table 1, manual comparison on fifty features isn’t a very large sample size for this sort of analysis. Supplementing this with e.g. a larger set of LLM generated autointerpretability scores would have been better, as was done in https://arxiv.org/abs/2309.08600. Personally, I don’t think not investing more effort into this is a major flaw, because I think reconstruction error and L0 are more important metrics than human interpretability scores.

Thank you for the suggestion. Indeed, we did recognize that fifty features each from a transcoder and SAE was a small sample size, but unfortunately, resource limitations prevented us from carrying out LLM-generated feature scoring on far more features. That said, note that our initial interpretability comparison was primarily intended to provide initial evidence for transcoders being approximately as interpretable as SAEs, and as such, able to be used in furthering our main goal of performing circuit analysis with them.

In Figure 2, they present reconstruction scores for transcoders and SAEs in terms of raw CE-loss recovered scores. I think it would have been good for the presentation to also show the recovered performance in terms of the compute required to train a model with the same CE score, as suggested in https://arxiv.org/abs/2406.04093. Raw CE scores can make the performance gap to the model look smaller than it is, since the difficulty of further reducing CE loss grows as models become more performant.

Thank you for this suggestion – this is a very interesting idea that we had not previously considered. We can update the figures to include an additional set of y-axis ticks reflecting the amount of compute that a Chinchilla-optimal model would require to achieve the same loss.

Questions: …

We believe that all of the questions have been addressed above; please let us know if you have any more.

Again, thank you so much for your review.

TL;DR: Thank you so much for taking the time to review our work. We were glad to see that you enjoyed our blind case studies and that you recognized the importance of the input-invariant/input-dependent decomposition of attributions. We performed some experiments in response to your questions about learned features, but we’d like to emphasize that our primary goal in this work was to use transcoders to perform input-invariant circuit analysis that SAEs can’t do.

We’ll now address your concerns and questions in order.

The concrete comparisons to traditional SAEs are not sufficiently thorough [...] There are, however, other questions that are more concrete but are unaddressed. e.g. how easy/stable are transcoders to train compared to SAEs?

We did not notice any meaningful differences in the difficulty of training transcoders compared to SAEs. (Note that we primarily follow the same training procedure as used for training SAEs, as our pipeline is adapted from SAE training pipelines.) The reason why we do not address this in the paper is because we did not find the experience of training transcoders to be noteworthy compared to training SAEs, and because training transcoders was instrumental to our primary goal of utilizing transcoders for circuit analysis.

Do transcoders suffer issues with "dead neurons" when the width is scaled up? Do transcoders show similar "feature splitting" effects when the width is scaled up?

In response to your question, we quickly analyzed the number of dead neurons in transcoders versus MLP-out SAEs with similar average sparsities (L0s), and found no clear winner when it comes to whether transcoders or SAEs have more dead neurons. (See the author rebuttal PDF for a graph.) As stated earlier, this investigation of training dynamics was not a core focus of ours. In any event, any dead neurons in the transcoders that we trained did not seem to affect fidelity or interpretability.

As for feature splitting, it seems that there are currently no rigorous metrics agreed upon by the mechanistic interpretability community for quantifying feature splitting. That said, we did train a pair of GPT2-small transcoders with expansion factors 32 and 64, and found that ~27% of live smaller-transcoder features were very similar (cossim > 0.95) to at least one feature in the larger transcoder. This seems to indicate some amount of feature splitting, but we find it difficult to interpret this number further.

While performing a blind test of the interpretability of different features is novel, I worry that it is prematurely rigorous. One of the implicit assumptions of the blind interpretability experiment is that the features learned by transcoders and SAEs are different [...] If the authors did not investigate the extent of the overlap between the features that SAEs and transcoders learn that is unfortunate.

Please note that our intent with these blind tests was to show that transcoder features are “up to par” with SAE features in interpretability, regardless of whether or not the features are different. This is because our main goal with transcoders is to use them to perform input-independent circuit analysis that can’t be carried out with SAEs, so we simply wanted to make sure that in achieving this, no penalties to interpretability were accrued. Importantly, this means that we weren’t intending to make any claims about whether transcoders learn “better” features than SAEs; rather, we intended to show that whatever features the transcoders do learn are equally interpretable to those learned by SAEs, without making any assumptions on the types of features being learned by SAEs versus transcoders.

What is the effect of different levels of L1 penalty on the features that the transcoder learns?

We did not investigate differences in features learned by transcoders at different sparsity levels, in part because we are primarily interested in the circuits learned by transcoders – and because of a lack of accepted methods for rigorously characterizing transcoder/SAE features learned at different sparsity levels. To our knowledge, this is a current lacuna in the mechanistic interpretability community’s collective understanding.

Does the input invariant / dependent construction not work for the features that SAEs learn?

This construction does not in fact work for SAEs; we believe that this is the primary advantage of transcoders. This is not because of the features that SAEs learn per se, but because SAEs fail to bypass the MLP sublayer whose nonlinearity prevents such a construction from being applied. In contrast, when computing attributions, transcoders explicitly bypass the MLP sublayer.

More formally, the input-invariant/input-dependent factorization for transcoder attributions is a special case of the classic “input-times-gradient” attribution method. For transcoders, the “input” term is input-dependent, but the “gradient” term is constant. But for SAE attributions, both the “input” term and the “gradient” term are input-dependent. (We would like to go into more mathematical detail here, but the rebuttal character limit prevents us from this.)

When looking at [sic]

It seems that this comment may have been cut off. If you can recall what you intended to write, we will be happy to reply during the discussion period.

Do SAEs and transcoders trained on the same layer and same data learn the same or similar features?

We performed some brief exploratory experiments a while back investigating this question on one-layer models. However, we have not continued to pursue this, as it was tangential to our primary goal of using transcoders for circuit analysis, although we do find it very interesting.

Have any attempts been made to perform a similar analysis for other circuits? e.g. the IOI or Gendered Pronoun circuits?

We have not carried these out yet, but we agree this is a promising direction for future research.

Before this phase of the discussion period ends, we wanted to check in with the reviewer on whether we have addressed your concerns with our work?

TL;DR: Thank you for taking the time to write this detailed review. We are glad to see that you enjoyed the presentation of our work, along with our interpretability study and blind case studies. Our initial paper was insufficiently clear that the main goal of transcoders is to enable input-invariant circuit analysis, which is impossible with any existing SAE-based feature circuit methods. We believe that most of your concerns stem from this, and as such, if we have addressed them, we hope that you might be willing to raise your score.

We’ll now respond inline to specific concerns:

Expand on challenges in MLP interpretability. This is the bulk of the motivation of the paper and, from what I can gather, is grounded in off-hand references in two prior works. I don't think those works thoroughly describe what the challenges in fine-grained circuit analysis of MLP sublayers are. [...]

Thank you for pointing this out---we realize now that we might not have properly explained the motivation behind the problem. In essence, the primary problem in fine-grained circuit analysis through MLPs is a lack of input-invariant methods for distinguishing between local and global behavior. The two prior works that we cited represent different attempts at performing this fine-grained circuit analysis, both of which fail to address this problem. Marks et al. 2024 use causal methods (such as attribution patching), which only yield the importance of features on a single input. Dunefsky et al. 2024 attempt to approximate MLPs with their gradients, but their method has the same shortcoming, because MLP gradients are very different on different inputs. To our knowledge, no current works on performing circuit analysis involving MLPs with SAEs avoid this problem, and we fear that this is an issue inherent to all such methods.

To make all of this clearer to future readers, we plan to add to the paper an explicit, mathematically-motivated explanation of how these SAE-based methods fail.

In fact, I would argue that your own results with SAEs show similar interpretability may be achievable as found via Transcoders (since SAEs perform similar in terms of model loss). From that perspective, it isn't established if the question being addressed is sufficiently a challenge.

We hope that it is now clearer, in the context of our answer to the previous question, why this is not the case. In particular, since our goal is to use transcoders to perform input-invariant circuit analysis, we view it as a positive that transcoders and SAEs have similar interpretability/model loss – because this means that we can substitute transcoders for SAEs and obtain all of the benefits of input-invariant circuit analysis without incurring any penalties.

Analysis is correlational, lacking a quantitative confirmation and causal interventions. While I really like the blind evaluation pipeline, there are two missing elements in the results per my opinion [...] First, once a hypothesis is formed, one should ideally run an algorithmic test to see on what proportion of a dataset where the hypothesis predicts a feature will activate does the feature activate.

The main reason why we did not perform any algorithmic tests of our feature hypotheses is that the features that we were investigating were mostly too complex to admit any such proxies. Whereas, for example, the original Towards Monosemanticity paper was able to cherry-pick features (such as a base64 feature or Arabic feature) that could easily be captured in such a manner, our blind case study pipeline meant that we did not know if a feature would be easily “proxied” until after having carried out the entire case study.

Indeed, while proxies can be useful for quantitatively evaluating the interpretability of an individual feature, it is worth noting that our goal with blind case studies was broader than this: we view blind case studies as an example of a nontrivial circuit-level interpretability task. To that end, we think that blind case studies act as a useful evaluation of how well a circuit analysis method would fare in a more real-world setting.

Second, the experiments are correlational, not causal. Ideally, one would systematically intervene on the input and see if the claimed hypothesis breaks in a predictable way. For example, when one identifies a feature activates for semicolons being present in the input, then if that input is kept the same but the semicolon is dropped or another relevant punctuation is added, does the feature not activate or at least activate less?

Although space constraints prevented us from highlighting this in the main body, we have performed such tests of our hypotheses which can be found in the appendices. Please refer to the end of Appendix H.1.1 and Appendix H.1.2 for examples of this in two of our case studies.

Figure 2. I found Figure 2 to be rather unintuitive [...]

Thank you for pointing this out. In general, the orientation of our axes was inspired by papers such as the DeepMind Gated SAEs paper. However, we didn’t realize that our axis label sizes would cause a problem, and as such, we will make them bigger.

From what I can gather, the "de-embedding" tool is equivalent to logit lens? Can you expand on what's different?

The de-embedding is a “reverse” logit lens: in the de-embedding, we multiply the encoder vector of the feature by the transpose of the model embedding matrix, in order to understand which tokens in the input vocabulary space cause the feature to activate.

I am still struggling to understand the benefit of Transcoders over SAEs. Can you expand on this more? It's good to define alternative tools, but unsure if this was a sufficiently different tool and led to much benefits.

We hope that the preceding discussion has clarified our position on this, which is that we think the main benefit of transcoders is the input-invariant circuit analysis they enable.

Thank you to the authors for their response.

I have a quick follow up question. Authors say in the paper that their goal was to design a tool for MLPs' interpretability, with statements that remark upon difficulties in analyzing them. In the rebuttals response though (both for my comments and in the global response), authors say their goal in this paper was to perform an input-invariant circuit analysis. I am struggling to reconcile these two points. It would help if the authors can expand on this bit. To be more precise, I would like to know:

1. Why is MLPs' interpretability deemed difficult? Why can we not use SAEs for this purpose, and why do we need a new tool to this end?

2. If I do analyze MLPs' features via SAEs, what blockades will I face such that input-invariance becomes a desirable property?

3. If the argument is input-invariance is a generally useful property (which I would buy), then why should we not use Transcoders for representations extracted from any unit in the model? Why is the authors' pitch that Transcoders are specifically motivated to address some challenge (unclear which challenge) faced in MLPs' interpretability?

Some further thoughts on your first question that we hope may be helpful: In circuit analysis work, a key goal is to decompose layers, large and complex objects, into independent components, so we can find a sparse set of components that matter for a given task. For attention layers, the layer naturally breaks down into independently varying attention heads. In prior work finding circuits, researchers often analyse individual heads, eg induction heads [1] or name mover heads [2]. Crucially, this works because attention heads often seem to be interpretable, at least in the context of a given task.

In MLP layers, however, things are harder. Though MLP layers decompose into independently varying neurons, these neurons are much harder to work with for circuit analysis than heads, as neurons are often polysemantic [3], i.e. activate for many seemingly unrelated things, and often many neurons seem relevant for a given task [4]. Prior work on circuit analysis that looks at MLPs [5, 6] has largely failed to find more granular decompositions just studying an entire MLP layer (though some works have made some progress [7]). Thus, by default, circuit analysis needs to either include or exclude an entire MLP layer, rather than being able to zoom in further. MLPs represent at least 60% of the parameters in models like GPT-2, so this lack of fine-grained analysis is a major roadblock to circuit analysis.

SAEs help by decomposing the output of the MLP layer, which can help causal intervention based circuit analysis [8], but SAE features are often dense combinations of many neurons [9], so we must still consider many neurons in the MLP layer, whose behaviour will vary depending on the input, preventing input-invariant circuit analysis.

Transcoders solve this problem by replacing the MLP layer with a sparser and more interpretable replacement layer. Transcoders features are easier to work with for circuit analysis, because they break the MLP layer down into fine-grained computations that do not depend on MLP neurons, which are often interpretable and can be studied independently as part of a circuit analysis. Transcoder features are computed directly from the MLP input (projecting the MLP input onto the encoder vector followed by a bias and a ReLU), allowing us to decompose the MLP layer better for circuit analysis in an input-invariant way.

Please let us know if there is anything further that we can clarify. If we have successfully addressed your concerns, we hope that you may consider raising your score.

[1] In-context Learning and Induction Heads. Olsson et al. https://arxiv.org/abs/2209.11895

[2] Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. Wang et al. https://arxiv.org/abs/2211.00593

[3] Softmax Linear Units. Elhage et al. https://transformer-circuits.pub/2022/solu/index.html

[4] Finding Neurons in a Haystack: Case Studies with Sparse Probing. Gurnee et al. https://arxiv.org/abs/2305.01610

[5] Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla. Lieberum et al. https://arxiv.org/abs/2307.09458

[6] Fact Finding: Attempting to Reverse-Engineer Factual Recall on the Neuron Level. Nanda et al. https://www.alignmentforum.org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall

[7] How does GPT-2 compute greater-than? Hanna et al. https://arxiv.org/abs/2305.00586

[8] Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models. Marks et al. https://arxiv.org/abs/2403.19647

[9] Open Source Replication & Commentary on Anthropic's Dictionary Learning Paper. Neel Nanda. https://www.alignmentforum.org/posts/fKuugaxt2XLTkASkk/open-source-replication-and-commentary-on-anthropic-s