Thank you for the positive feedback! We are glad that you found our method timely, scalable, and clearly presented. We address your concerns and questions below.

…[U]sing this approach would still require contrastive sample pairs that is hard to get and / or define for tasks such as long-form generation and algorithmic reasoning… [I]t'd be great for the authors to discuss this type of dataset limitations and future directions to automatically construct contrastive pairs for more challenging tasks.

Note that we do not necessarily require contrastive sample pairs! See our experiments on unsupervised circuit discovery, where no contrastive pairs were used, and moreover no human needed to make decisions about task definition and data selection.

Regarding long-form generation, you are correct that this paper focuses on circuits which explain the prediction of particular tokens. However, the core ideas should generalize: instead of using the log-probability of a single token as the metric to explain, one could instead use the log-probability of a multi-token completion. This substitution would be a drop-in replacement—nothing else about the method would need to change.

As the authors have mentioned, most of these results are qualitative, and it would be great to establish a more quantitative benchmark to supplement the other interesting behaviors that the authors were able to mine from their unsupervised approach.

We agree! We have work in progress on this; stay tuned!

Thank you for the positive feedback! We are glad you found our work to address core challenges in interpretability, and that you found our experiments to be effective and clear. We address your concerns and questions below.

Weaknesses

No code and data released yet.

Please note that our code and data are submitted as supplementary material. These will be publicly released after the review period.

…some formal characterization of when SAE features capture meaningful concepts vs memorized patterns would strength the paper.

To clarify: By "memorized patterns", are you referring to (i) features that are used by the model, but which are uninteresting because they capture memorized token associations, or (ii) cases where the SAE itself has memorized training data in a way that does not correspond to something learned by the language model? We note that features of type (i) may in fact play an important role in a model’s computation (e.g. the association between “Michael Jordan” and “basketball” is inherently a memorized one), so should rightfully appear when explaining model behaviors. Thus, we will assume that you are concerned about (ii).

For this, causal verification provides a way forward. If we annotate a feature as being related to concept C, then we can confirm that modifying the network’s computation by intervening on the feature affects the model’s output in a predictable way related to C. While we don’t do a thorough analysis of this for all features in our SAEs, our SHIFT experiment provides some validation: we see that intervening on networks by removing their gender-related features causes their activations to encoder less gender-related information.

SAE training requires significant compute (2 billion tokens) with potential instability (dead features). The current implementation limited to decoder-only models (Pythia-70M and Gemma-2-2B).

We agree. We will note that, based on SAE scaling laws [1] that were made available after this paper’s preparation, our Pythia-70M SAEs were actually overtrained by a factor of 10. So it would have actually been possible to produce SAEs of a similar quality with substantially fewer training tokens. There’s also been significant progress on reducing the fraction of dead features [2, 3].

Subject-verb agreement circuits focus on relatively simple grammatical phenomena and this might not be an issue with scale but no mention of anything more complex in the paper.

We agree that it would be interesting to apply sparse feature circuits (SFCs) to more complex tasks. If this concern is primarily on the diversity of tasks, we believe that the unsupervised interpretability pipeline partially addresses this point: it reveals that SFCs allow one to tractably interpret very diverse behaviors in LMs. (Please feel free to explore these diverse behaviors at our demo at feature-circuits.xyz.) The Bias in Bios experiment also reveals that sparse feature circuits are effective not just in revealing interpretable mechanisms underlying syntactic agreement tasks, but also those underlying a semantic classification task.

Questions

The paper shows strong results with linear approximations of indirect effects. Have you checked when/why these approximations break down? This seems especially relevant for early layers where you note integrated gradients improves accuracy.

In general, the linear approximations were worse in earlier layers, and especially for the layer 0 MLP; see figure 25 in App. H. Intuitively, we expect that linear approximations are worse for earlier layers because there are more nonlinearities between early layers and the model’s output. (As an extreme case in the opposite direction, the model’s output is a linear function of its final layer, so linear approximations are perfect for the final layer.)

While not surprising, we’ll note explicitly that approximation error doesn’t seem to be any worse due to using SAE features. Our results on the efficacy of linear approximations (and when they fail) are generally in line with prior work, and Figure 25 does not show different trends for SAE features vs. error terms.

For the automated circuit discovery, how robust are the discovered clusters to different choices of projection dimension and clustering hyperparameters?

As with our other results on unsupervised circuit discovery, we lack the metrics to quantitatively analyze the thousands of circuits produced by our unsupervised pipeline. However, we did try six different approaches to clustering, and we’re happy to give some qualitative takeaways. If you’d like, you can browse the clusters and corresponding feature circuits for all of these variants at feature-circuits.xyz.

One axis of variation was clustering based on activations vs. based on linear indirect effects (i.e. activations * gradients). It generally seemed like the former method produced clusters based on similar inputs, whereas the latter produced some clusters based on similar outputs (e.g. clusters typified by the model’s output always being a particular token).

Another axis of variation was how to aggregate activations/linear effects across various tokens. The options we explored were: only take the activation/indirect effect over the final token; over the final 5 tokens; or sum over all tokens. The difference between final token vs. final 5 tokens was what you might expect: the final-token clusters seemed to more strongly depend on the last token of the input. Surprisingly, for linear effects, we didn’t observe much of a difference between just using the final token or summing over all tokens; it seems that (consistently with our circuits), when doing causal attribution for the next-token-prediction, activations in the final token position consistently have the largest causal effects.

We also replicated clustering via parameter gradients as in [4]. These clusters looked qualitatively similar to our last-token activation or linear effect clusters.

We didn’t experiment with different choices of projection dimension. We chose the projection dimension to approximately preserve the unprojected cosine similarities, which seemed like a principled choice.

References

[1] Lindsey, et al. (2024). Scaling Laws for Dictionary Learning. https://transformer-circuits.pub/2024/april-update/index.html#scaling-laws

[2] Gao et al. (2024). Scaling and Evaluating Sparse Autoencoders. https://arxiv.org/abs/2406.04093

[3] Conerly et al. (2024). Update on how we train SAEs. https://transformer-circuits.pub/2024/april-update/index.html#training-saes

[4] Michaud et al. (2023). The Quantization Model of Neural Scaling. https://openreview.net/forum?id=3tbTw2ga8K

Questions

How do the authors ensure consistency in identifying spurious features across different annotators or applications? Are there any plans to standardize the interpretation criteria?

We had two authors independently annotate each sparse feature. We then compared annotations and resolved differences via discussion. In practice, it is likely true that different annotators will arrive at distinct annotations. However, as discussed above, we believe that if this were reduced to the binary classification task of simply assessing whether a feature is relevant or spurious to the task, then there would be significantly more inter-annotator agreement.

Could the authors clarify how the performance of SAEs affects the interpretability and effectiveness of feature circuits? Have they tested their approach on varying SAE architectures to assess robustness?

The interpretability, faithfulness, and completeness of discovered feature circuits crucially rely on SAE quality, as discussed above. Please note that the Pythia-70M and Gemma-2-2B SAEs we use have different architectures, with the former being a simple rectilinear encoder-decoder and the later being a JumpReLU autoencoder; see the top of page 3 for more information about the SAEs we use.

For large-scale models with complex circuits, what optimizations or techniques do the authors recommend to maintain interpretability while managing the increased number of features?

The circuit discovery algorithm we use is already relatively scalable for discovering circuits in larger models (assuming that trained SAEs exist for the model). Thus, the remaining challenge is that of interpreting the features. Here, a common technique is to use language models to interpret SAE features; for example, Neuronpedia provides LLM-generated labels for all features in the Gemma-2-2B SAEs we used (see the box in the upper-left at https://www.neuronpedia.org/gemma-2-2b/16-gemmascope-res-16k/0 for an example). While these automatic interpretations are not always reliable, they are a very active area of research and we expect rapid improvement.

Additionally, a subset of the authors have pilot experiments using agents to automatically decide which sparse features in a given circuit are spurious and can be safely ablated. We have found that it is effective to condition on a description of the task when generating the feature labels, and that this improves the agreement between the agent’s and humans’ judgments on which features are spurious for a given task. While we do not yet have fully-formed evidence, our pilot experiments give us hope that we will be able to automate away much of the human labor involved in interpreting the causal mechanisms underlying task-specific behaviors.

References

[1] Bills et al. (2023). “Language models can explain neurons in language models.” OpenAI. https://openaipublic.blob.core.windows.net/neuron-explainer/paper/index.html

[2] Lin & Bloom (2023). “Analyzing neural networks with dictionary learning.” Neuronpedia. https://www.neuronpedia.org

[3] Gao et al. (2024). “Scaling and evaluating sparse autoencoders.” https://arxiv.org/abs/2406.04093

[4] Rajamanoharan et al. (2024). “Improving Dictionary Learning with Gated Sparse Autoencoders.” https://arxiv.org/abs/2404.16014

[5] Arora et al. (2024). “CausalGym: Benchmarking causal interpretability methods on linguistic tasks.” ACL. https://aclanthology.org/2024.acl-long.785/

We are grateful that you found our technique to be innovative, useful, and scalable! We address your concerns and questions below.

Weaknesses

The framework lacks a clear evaluation on the repeatability and consistency of human-selected spurious features in SHIFT, especially when applied by different users or in different contexts.

This is a valid point. We have added a statement to the limitations about this.

Although SHIFT leverages human judgment for feature interpretation, this reliance could hinder scalability in complex or high-dimensional models, as manual inspection becomes increasingly challenging. The paper does not provide clear guidelines on how to efficiently scale this process for larger models or more diverse tasks.

There has been some work on automatically labeling sparse features using LLMs [1,2]. While not perfect, we believe that this line of work has significant potential to automate away much of the human labor involved in performing SHIFT. In particular, the annotation task involved in our SHIFT experiments was actually relatively simple, in the sense that it was almost always very easy to tell whether a feature was related to gender; see the example features in Figures 19 and 20. Once a human has identified gender as an unintended signal of interest from inspecting a few features, we think it’s likely that a well-prompted LLM would be effective at systematically searching through the full collection of features for gender-related ones.

The method’s success is heavily tied to the availability and quality of sparse autoencoders (SAEs). If the SAEs are suboptimal, the interpretability and causal circuit discovery might suffer, particularly if they fail to capture meaningful latent structures. This dependency on pretrained SAE quality raises questions about the method’s consistency and applicability across different model architectures or under domain shifts.

This is a valid point. We rely on high-quality SAEs to discover useful and interpretable sparse feature circuits. That said, the SAE training techniques available during this paper’s preparation were already good enough to enable applications like the ones we demonstrate, and there has been rapid progress since then [3,4].

As a secondary point, because our technique computes circuits containing SAE error nodes and quantifies the causal importance of these nodes, it is well-suited to be adapted into a metric for guiding future progress in SAEs. We leave exploring this direction to future work.

This limited evaluation could affect confidence in the generalizability of the results. Expanding evaluations to include complex, semantically challenging tasks (e.g., multi-turn dialogue or reasoning tasks) would provide stronger evidence of the method’s robustness.

We agree that it would be interesting to apply sparse feature circuits (SFCs) to highly capable models on difficult tasks. If this concern is primarily on the diversity of tasks, we believe that the unsupervised interpretability pipeline partially addresses this point: it reveals that SFCs allow one to tractably interpret very diverse behaviors in LMs. (Please feel free to explore these diverse behaviors at our demo at feature-circuits.xyz.) The Bias in Bios experiment also reveals that sparse feature circuits are effective not just in revealing interpretable mechanisms underlying syntactic agreement tasks, but also those underlying a more semantics-based classification task.

If this concern is instead on the difficulty of the tasks, we agree that it would be interesting to analyze the mechanisms underlying performance on datasets used to benchmark state-of-the-art models. For example, a mechanistic analysis of how Llama 3.1 (405B) accomplishes ARC and MMLU could likely be its own paper!

Thank you for the feedback! We appreciate that you found the paper to address an important topic and the results to be promising. We address your feedback and questions below.

Weaknesses

…[I]dentifying specific circuits among those formulated by these features can be challenging.

They can indeed. Hence, we propose a causally efficacious technique that scales effectively with the number of components (and combinations thereof). Is the concern regarding the time complexity of the technique, or perhaps the difficulty of locating the exact circuit? We would love to address your concern, so if you have a chance to clarify, we would be happy to discuss further.

SAE errors cannot be fully interpreted… it should be discussed more in the paper regarding how to perform surgical edits on such errors.

We agree that this is a fundamental limitation of our method. Better SAEs will lead to more interpretable circuits, whereas worse SAEs will put much more of the indirect effects in the error terms.

…Simply removing gender-related features and observing a drop in classification performance is not universally convincing as a causal intervention approach.

Thank you for the comment. Following prior approaches to studying causality, we establish a causal connection between specific SAE features and the probe's performance by intervening on the SAE features and observing a change in the way the probe classifies gender (which was the predicted effect, since the SAE features ablated were selected for appearing related to gender). If you have remaining concerns about establishing that the circuits identified are causal, would you please elaborate on your concerns? We'd love to address them.

For example, one possible concern might be that ablating any SAE features generically interferes with gender classification (but not, for some reason, profession classification). We have added an experiment that addresses this: we now have a baseline for ablating random SAE features instead of SAE features which were selected as being related to gender. We show that this has next to no impact on probe performance (see the revised table 2), indicating that it was the specific SAE features we selected which were having the effect. Please let us know if you would like to see other experiments.

Other concerns

Thank you for pointing out these formatting errors! These have been fixed in the revised PDF.

Questions

Can you add explanations regarding the efficiency of sparse feature circuit finding with the vast amount of possible feature circuits with respect to neurons?

Please see L139-157. Using the most efficient linear approximation for estimating indirect effects, we can identify causally relevant features using just two forward passes and one backwards pass per input in our dataset. When using the more accurate but less efficient integrated-gradients-based approximation, this grows by a factor of our model’s serial depth (i.e., roughly the number of layers). So overall, identifying the nodes in our circuits is as expensive as inferencing the model over our data a constant number of times; in particular, it does not depend on the number of features. We work with datasets of 100 examples, and this takes a few seconds for both Pythia-70M and Gemma-2-2B.

Computing the edges in our circuits is more expensive: it requires an additional forwards and backwards pass per feature in our circuit (the number of which depends on the chosen threshold for including nodes) and per datapoint. For the circuits we present in this paper, this takes around 10 minutes for Pythia-70M and around one hour for Gemma-2-2B.

We have added some information discussing the time efficiency of our method.

Can you share your thoughts on extended experiments on more feature ablating studies and how would the results be?

As discussed above, we have added experiments for random feature ablations. Are there other experiments that you would be interested in seeing?

Can you help better understand how to interpret SAE errors as part of the sparse feature circuits?

Certainly! In this work, we use SAEs to decompose latent activations x as sums \sum_{i=1}^{d_{SAE}} f_i(x) \mathbf{v_i} + \epsilon(x) where the f_i(x) are feature activations and \epsilon(x) is a (high-dimensional) error term. This decomposition allows us to view an LLM as consisting of a computational graph with nodes corresponding to the f_i’s and \epsilon’s. Concretely, you could imagine representing the LLM as a computational graph with a node for each residual stream, attention output, and MLP output; and then breaking each of these nodes apart into d_{SAE} + 1 nodes each. Note that even though these nodes come in two types (nodes representing one-dimensional features and nodes representing high-dimensional errors), there is no difference between these types of nodes from the perspective of the causal structure of the graph. In particular, we can define the causal effect of a node on the model’s output in exactly the same way for these two types of nodes: as the change in the model’s output after intervening on the node by patching it to a counterfactual value. So from a formal perspective, you can view the error terms as components in a circuit in exactly the same way as for sparse features.

However, there is one key difference between sparse features and error terms: it is possible to annotate the nodes corresponding to sparse features with human-understandable interpretations. But this is not possible for the error terms.

We’ll note the causal importance of the error terms, as measured by the change in the model’s training-set perplexity after ablating these errors, was already a central metric of the quality of an SAE prior to this work (though not usually phrased in the language of “error terms”). Thus, better SAEs produce compositions whose error terms are less important for model outputs (and conversely, the interpretable sparse feature nodes explain more of the model’s outputs). You can see this reflected in our work when comparing feature circuits computed with our Pythia-70M SAEs (which have quite important error nodes) to feature circuits computed using the Gemma Scope SAEs for Gemma-2-2B (where the error nodes are substantially less important). One interesting consequence of our work is that—since feature circuits are computed for specific tasks—the weight of error nodes in feature circuits gives a task-specific indication of SAE quality (in contrast to using loss recovered on the full training set).