Sparse Autoencoders Do Not Find Canonical Units of Analysis

Thank you for your positive review and thoughtful questions. We appreciate your assessment that our “motivation and methods are explained clearly and intuitively, with helpful examples” and that we provide “detailed experimental information”. We address your main points and questions below:

Including a discussion on the potential limitations of the proposed approaches would be valuable.

We have expanded the conclusion section, highlighting the limitations of our work.

Is there any particular reason to omit the bias term of the decoder 1 in equation 5?

For a given base language model, the decoder biases of all the SAEs are very similar. For GPT-2, the minimum cosine similarity between any two pairs of SAE decoder biases is 0.9970, and the magnitude of the vectors differs by less than 0.01%, so we arbitrarily use b^{dec}_0. We've updated the text around the equation to clarify this.

How does the stitching SAE handle shared reconstruction latents during the swap process? For example, in the case of colored shapes, how would the model swap the "square" or "blue" latent if these are entangled in composite latents in the larger SAE?

In case of one-to-many, many-to-one, or many-to-many relationships between reconstruction features in two SAEs, they are indeed swapped simultaneously. Please see Appendix A.5 for some examples of the sub-graphs of swaps that were performed on the two smallest GPT-2 SAEs. In the case of the “blue square”-example, this would result in a sub-graph containing all the latents in both SAEs.

Would it be possible to introduce some form of supervision in meta-SAEs?

We believe the unsupervised nature of both SAEs and meta-SAEs is actually a key advantage, as it allows us to discover what compositional structure the model naturally represents rather than imposing our assumptions about what concepts should exist. We agree that future work with supervised datasets could be valuable for validating (meta-)SAE performance and understanding how discovered decompositions align with human concepts.

For BatchTopK, have the authors examined how varying values of k impact the semantics of the learned latents? Do the "concepts" learned under a stronger sparsity constraint become more abstract (as they need to explain a given activation with fewer latents) for a fixed dictionary size?

Our paper primarily focuses on the effect of dictionary size on the abstractness of features, rather than the role of sparsity (k). However, we agree that analyzing the relationship between sparsity and semantic granularity would be valuable future work, both in BatchTopK SAEs and other SAE architectures.

Have the authors extracted any intuition on how should the dictionary size be adjusted to tailor the SAE to the requirements of an specific analysis?

We have addressed this in the top-level comment.

Thank you for your thoughtful review and helpful suggestions. In particular, we were happy with your assessment that our work “provides thorough and solid experiments” and “answers an important question”. We appreciate your feedback on the positioning of the paper within the literature, and have updated the manuscript to be more precise about the modality of the experiments, and added experiments to clarify the relevance of our results to interpretability. We address your main points and questions below. If you have further comments or questions we are keen to respond to them, and otherwise appreciate an increase in your support for our paper.

In the introduction and in the abstract, the paper is presented as if the research topic pertains to an analysis of a decomposition of the activations of a "general" neural network into features, while in fact the SAE introduced by [Bricken et al] and [Cunningham et al] is a probing tool for a specific 'organism' that is LLM.

SAEs have been used on a range of modalities, including image and audio models. However, you are correct that our work specifically advances the understanding of SAEs as interpretability tools for LLMs. We have updated the abstract and introduction to specify this.

The research does not relate the claimed novelty to the SAE's probing "capability"

Please see the top-level comment.

Figure 4, it is said that "Features with cosine similarity less than 0.7 tend to improve MSE". Do you mean 0.6 or less? In the similar regard, this tendency is much less clear in Figure 18.

We found that different model architectures and training regimes lead to different optimal thresholds (0.7 for GPT-2, 0.4 for Gemma). By "work well" we mean that using these thresholds allows qualitatively similar results (such as the interpolation between SAE sizes), though the exact patterns differ between models. We have updated the text to clarify this. Note that by changing the threshold, we can adapt the trade-off between false positives and false negatives, as shown in the ROC plot in Figure 16.

What are "active latents"?

We have added a definition to the Glossary of Terms (Appendix A.1) - these are latents with non-zero activation values after applying the sparsity-inducing activation function.

Would you please clarify "on what SAEs" the MetaSAE were applied in these experiments? For example, from SAE of what size "the Meta SAE with 2304 latents" was obtained?

The meta-SAE with 2304 latents was trained on the decoder directions of the GPT-2 SAE with 49152 latents. These experimental details can be found in line 403 in the paper.

Before this phase of the rebuttal period ends, we wanted to ask the reviewer whether we have addressed your concerns with our work?

Thank you very much for trying out the best to resolve the concern. It was very important to me that the experiment is being done on the benchmark that are available for all reviewers (including myself) to see. I am raising the score. Please notify to all reviewers that this modification on your manuscript has been done (including the github link).

Thank you for clarifying the [Upcoming Work], and I am very sympathetic to the situation. While so, I have never seen this case in my moderately long research career so far, particularly the case in which the reference is being made to a work that has not been even unofficially published (e.g. ArXiv) so that no one can verify its validity. While I do feel the sympathy, please understand that I must also play my part in conducting a fair review and that the problem addressed in this part of the additional experiment is very important in my opinion. By any chance, would it be possible to point out any preceding example in which something like this has been done before, so that I feel more confident in raising the score? Meanwhile, I will try to look for a such example myself within my capability.

Thank you for your positive and insightful feedback, in particular for recognizing that our work “addresses an important question about SAE features” and that “the presentation in the paper is clear and the experiments are original and well-executed”. We address your main points and questions below:

I could not find reported values of the reconstruction loss that meta-SAEs obtain in reconstructing the base 49k-latent SAE latents. If they are only a very weak approximation, what would that say about the hypothesis that large-SAE latents are linear combinations of more atomic latents?

Thanks for bringing up this point. The meta-SAE with 2304 meta-latents (and an average L0 of 4) explains 55.47% of the variance of the 49k latents in the SAE. We have included this information in Section 5 of the updated manuscript. Although this only indicates partial linear decomposability, it still provides concrete evidence against the hypothesis that SAEs converge to atomic units. With meta-SAEs, as with SAEs in general, we think the most important result is that any kind of decomposition is possible, not that it is perfect. The fraction of activation variance unexplained for our SAEs ranges from 50% to 10%, in comparison with 45% to 10% for our meta-SAEs, depending on choice of L0 and number of (meta-)latents.

Some minor grammatical and presentation issues: "vertexes" -> vertices, the left quotation marks in the meta-SAE section should be fixed, etc.

Thank you for pointing these out, they have been fixed in the updated version.

It's interesting that the reconstruction error falls as much as it does when the reconstruction latents are stitched in.

This is a good observation. The reason that the reconstruction tends to fall more steeply during reconstruction latent stitching vs. novel latent stitching has to do with the order in which we do these. If we swap reconstruction latents before adding in the novel latents, the reconstruction MSE tends to go up a little bit when swapping the reconstruction latents (see Appendix A.7 Figure 17 for the individual effect of swapping reconstruction latents). The steeper fall in MSE during reconstruction latent stitching in Figure 5 occurs because we've already added in the novel latents, which are optimized to work best in combination with the other latents of the larger SAE.

I wonder if a feature-manifold explanation might also be worth considering here (Engels et al. 2024), where reconstruction features are more densely covering a feature manifold that corresponding latents in the small-SAE are more coarsely covering.

Your hypothesis about the feature manifold explanation from Engels et al. 2024 is interesting. It aligns with our finding that sometimes multiple reconstruction latents have high cosine similarity with multiple small-SAE latents. This could mean that the small SAE makes a lower-dimensional approximation of a higher-dimensional feature manifold that the large SAE represents. We would be excited to see future work in this direction.

Samuel Marks, Can Rager, Eric J Michaud, Yonatan Belinkov, David Bau, and Aaron Mueller. Sparse feature circuits: Discovering and editing interpretable causal graphs in language models. arXiv preprint arXiv:2403.19647, 2024.

Lieberum, T., Rajamanoharan, S., Conmy, A., Smith, L., Sonnerat, N., Varma, V., ... & Nanda, N. (2024). Gemma scope: Open sparse autoencoders everywhere all at once on gemma 2. arXiv preprint arXiv:2408.05147.

Leo Gao, Tom Dupr´e la Tour, Henk Tillman, Gabriel Goh, Rajan Troll, Alec Radford, Ilya Sutskever, Jan Leike, and Jeffrey Wu. Scaling and evaluating sparse autoencoders. arXiv preprint arXiv:2406.04093, 2024

Senthooran Rajamanoharan, Arthur Conmy, Lewis Smith, Tom Lieberum, Vikrant Varma, János Kramár, Rohin Shah, Neel Nanda. Improving Dictionary Learning with Gated Sparse Autoencoders. arXiv preprint arXiv:2404.16014, 2024

Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, et al. Towards monosemanticity: Decom- posing language models with dictionary learning. Transformer Circuits Thread, 2, 2023

Isn't it clear that (local optima aside) a larger SAE will always find features that are missed by smaller SAE? I'm not sure I follow the argument from line 355 onwards.

While it might seem intuitive that larger SAEs would always find novel features, it wasn't clear whether additional capacity would be used to find new features versus just representing existing features more sparsely. In Figure 2, we provide the “blue square” example of a small SAE that has learned the 6 ground truth features in a dataset, and a large SAE with 9 latents that has learned compositions of those features to further reduce sparsity. Our results empirically demonstrate both effects occur.

Please add more clarity around treating the latents W_i^{dec} (why is W in boldface)? W_i^{dec} is a scalar quantity, the ith component of the d-dimensional vector W^{dec}. What does it mean to take scalars "training data for our meta-SAE"? The directions W don't convey information about which directions are simultaneously activated. I would have thought it would make more sense to treat the feature vectors f(x) as the entities for learning a meta SAE.

Each decoder direction W_i^{dec} is actually a vector, not a scalar. W^{dec} is a matrix, and we are taking the i'th column, the decoder vector for the i'th SAE latent. The meta-SAE learns to reconstruct these vectors. We have added further clarification of the shapes of these entities in Section 2.

The BatchTopK function is fully defined. In line 465 it states that the function selects the top K activations, suggesting that this is a mapping from activations to indices. I suspect that the authors mean that the function should return zero for all activations that are not in the top K highest positive values, and is the identity function otherwise.

Yes, your interpretation is correct - BatchTopK zeroes out all activations except the top K highest positive values, for which it acts as the identity function. We've added this explicit definition to Section 6.

The introduced batch method is used only during training, with a different non-linearity used during "inference". This seems quite strange and it's not clear to me how to justify this. A potential issue not mentioned is that it's quite possible that the batch approach means that some input sequences will have entirely zero activation.

During training, we estimate the threshold on single random batches for efficiency purposes. However, this means that the threshold varies depending on the samples in the batch, resulting in dependencies between samples in the batch. During inference, in order to break this dependency, we use a single threshold estimated over many training batches. This also lets us do inference on small batches (eg a single prompt), where an estimate for the threshold would be very noisy. Our experiments show that this works well. Using different functions during training/inference is common, for example in dropout and batch normalization. If all activations are zero, the reconstruction is equal to the decoder bias, but we have not seen this in practice (see Figure 9). This is the same for ReLU and JumpReLU SAEs, but not for TopK SAEs.

It's hardly surprising that the BatchTopK SAE has lower SAE than TopK SAE since BatchTopSAE imposes fewer constraints on the objective. I'm not sure why this would be seen as "outperformance".

The goal of papers introducing new SAE training methods (Gao et al. 2024, Rajamanoharan et al. 2024, etc) is to find a method that is better at finding sparse reconstructions, to act as a useful tool for researchers. The standard method is showing better reconstruction performance at similar sparsity levels and number of latents. Our contribution with BatchTopK is to provide a novel method with better performance. We agree that, once thought of, it's unsurprising that BatchTopK is better. Though, ReLU SAEs also impose fewer constraints on the objective than top-k SAEs, but attain worse reconstruction performance Gao et al. 2024

I'm also unclear as to why BatchTopK SAE is being discussed. Is this method used in all the previous experiments in the paper, or is this a separate piece of work orthogonal to the other contributions of the paper?

BatchTopK was developed specifically to enable training meta-SAEs with very low sparsity (4 active latents per input on average), which existing methods struggled with. It is only used for the meta-SAE experiments and to compare it to existing methods, not in the earlier stitching experiments.

Figure 11 isn't well explained. Please explain what is being shown here.

Figure 11 (now Figure 12) shows paired examples of latents from GPT2-768 and GPT2-1536 that have high cosine similarity (0.99). For each latent, we show their top activating inputs and the logits they influence most strongly. This demonstrates that similar latents in different sized SAEs capture similar semantic features. We've expanded the text in Appendix A.4 to clarify the figure.

Thank you for your extensive review! We appreciate your feedback in helping us make this paper more appealing and clear to readers from the broader community. We have run some additional experiments, and address your questions and critiques below. If you have any further questions or feedback we are keen to hear them, and otherwise we would appreciate you to increase your support for our paper.

I don't really agree with the phrase "circuit". To me this suggests some end to end explanation, whereas the reality is that a small part of a transformer is being examined (the activations in a single layer).

We use the term "circuit" only in the related work section where we reference papers that examine activations across multiple layers (e.g., Marks et al.'s "Sparse Feature Circuits"). While our work focuses on single-layer analysis, we maintain this terminology when discussing prior work to be consistent with the established literature.

I can't find any information on how the SAE is trained. What is the optimiser and how are the hyperparameters (eg \lambda) set?

Thanks for pointing this out. We have added the training details for the GPT-2 SAEs to Appendix A.6 The Gemma-2 SAEs are described in Lieberum et al.

Why choose layer 8 of GPT2?

We followed the lead of Gao et al. (2024) in choosing the layer 8 residual stream. This clarification has been added to Appendix A.6 along with the training details.

Equation 5 has an asymmetry. Why is the bias from decoder 0 used, rather than decoder 1, or a combination of the two?

For a given base language model, the decoder biases of all the SAEs are very similar. For GPT-2, the minimum cosine similarity between any two pairs of SAE decoder biases is 0.9970, and the magnitude of the vectors differs by less than 0.01%. We've updated the text around the equation to clarify this.

The motivation for stitching isn't clear to me. The features and decoders learned are optimised for each SAE separately. Why would combining them in a suboptimal way mean anything? Why not fix the features f0, f1 from each encoder but then learn optimal decoder weights W_{01} for the combination of these feature (this is a simple quadratic optimisation problem). Similarly, in the subsequent discussion, I don't follow why an increase or decrease in MSE is of any significance.

While learning optimal decoder weights is possible, our goal is to understand relationships between features learned by SAEs of different sizes, not to optimize their combination. By keeping original decoder weights, we can directly measure whether larger SAE features provide novel information or reconstruct information already captured by smaller SAEs. The MSE changes directly indicate whether larger SAEs learn novel features or just different representations of the same information.

If one wishes to understand whether latents in larger SAEs are finer grained versions of latents in smaller SAEs, would it not be feasible to look at a sentence that coactivates two features, one feature from the smaller SAE and one from the larger SAE?

From a single sentence it would be hard to distinguish such a relationship between features in SAEs of different sizes as many features from each SAE are likely to be active on that sample. Bricken et al. (2023) use coactivation as a measure of latent similarity, however we empirically found a high correlation between this metric and the decoder similarity metric (see Appendix A.4 Figure 11), and decoder similarity has lower computational cost. Furthermore, decoder directions define the effect of an active latent on the reconstruction, rather than just the magnitude of the activation.

In figure 5 it's not clear to me what the average L0 means -- what is being averaged over here?

The L0 is averaged over a number of input sequences in our dataset. We've now clarified this in the figure caption.

As far as I understand the SAE objective is non-convex, meaning that there is no guarantee of finding the global optimum. It's therefore also quite possible that different SAEs are simply finding different latents simply because of finding different local minima.

Thank you for raising a valid point about local optima. To investigate this, we have added an extra experiment to Appendix A7 (Figure 19) where we compare the number of reconstruction/novel latents between SAEs of the same size. We find that an average of 94% of latents are reconstruction latents with regard to SAEs of the same size, compared to a maximum of 68% for SAEs of different sizes.

Before this phase of the discussion period ends, we wanted to check in with the reviewer on whether we have addressed your concerns with our work?