Are Sparse Autoencoders Useful? A Case Study in Sparse Probing

Thank you for taking the time to review our work! We are grateful for your time and help, especially related to missing discussion of related work and questions about the automated interpretability techniques we use. We were especially glad to hear that you appreciated the depth to which we studied evidence and counter-evidence.

---

And whether SAEs extract interpretable concepts remains essentially unsolved.

.. but this does not seem to provide clear evidence in support of or against interpretability of SAEs…All questions revolve around the interpretability of SAEs. The main question is: Are SAE neurons interpretable?

This is an important point that we would like to clarify. In this paper, we do not directly investigate whether SAEs discover an interpretable basis of latents. This statement is difficult to falsify or prove because there is no ground-truth for SAE latents, as you note. Instead, the goal of our work is to attempt to evaluate how “good” the basis of features that SAEs discover is by studying how helpful they are on the downstream task of probing. This question is both more useful for practitioners and easier to answer than pure interpretability metrics. Thus, our work also provides a more objective (if indirect) measure of SAE latent interpretability.

The details about the autointerpr method are missing and they should be discussed at least in the supplementary. I am not even convinced that using this method is the best practice: it looks a bit circular to study the LLM concepts with another LLM, and in the best possible scenario that would have been done with a user study.

Thank you for bringing this up! We have added the following description of autointerp in Section 4.1: “For this and all subsequent experiments, we generate autointerp labels using Neuronpedia, which leverages a language model to produce natural language explanations for a latent based on its top activating tokens (the Neuronpedia autointerp implementation is based on [4]).” Autointerp is a standard procedure for evaluating SAE latents [1, 2, 3, 4 ], although you are correct in stating that it is somewhat circular to study an LLM with an LLM! More specifically, our procedure is dependent on the efficacy of autointerp. To remove this confounder in Section 4.1 (about probe pruning), we ran an additional experiment where two of the authors independently labeled latents and ranked their relevance for all three tasks, with no change to the results.

[1] "Language models can explain neurons in language models."

[2] "Towards Monosemanticity: Decomposing Language Models With Dictionary Learning"

[3] "Scaling and evaluating sparse autoencoders."

[4] "Automatically interpreting millions of features in large language models."

In the discussion of Section 4.1, the authors focus on the latent 122774 of the SAE, whose supposed meaning is "mentions living room". This "concept", however, is not invariant to language variations (the French example), so is it really encoding the concept of "presence of living room"? Is it activating also on unrelated sentences?

This is correct! We use latent 122774 as an example of a latent that doesn’t represent the true living-room feature (which we expect to be the same across languages). This indicates that the underlying SAE is imperfect, and is a possible reason why SAE probes did not generalize as well to covariate shifts. This latent generally activates only when living-room is in the sentence, but other latents are less precise.

In section 4.2, the authors further suggest that some latents are correctly labelled and some others are mislabelled by autointerpr… Despite that, is it the case that latents are disentangled from other potentially unrelated concepts? Can they activate in other tasks as well? Also, from the conclusion in lines 372-384 it is not clear what the authors mean.

Thank you for mentioning this! If SAEs worked perfectly, then we would expect latents to specialize and only fire on a single concept (which could be quite complex itself). However, since SAEs are imperfect, many latents remain polysemantic (are active on multiple concepts). We have adjusted the language of lines 372-384, we agree that it was complex! Please see our response to reviewer 3 (gmH6), which has the revised paragraphs.

Also, the code is not available and reproducibility cannot be verified.

Thank you for pointing this out! We have uploaded our code anonymously here: https://anonymous.4open.science/r/SAE-Probes-B404 We also have added a link to the de-anonymized github repo in the non-anonymous version of our paper.

Thank you for providing the additional citations, we will add them to our related work.

---

Thank you again for taking the time to review the paper and providing helpful feedback! Do the above actions address your concerns with the paper? And are there any further clarification or modifications we could make to improve your score?

We are thankful for your time and help, especially related to your points about the quiver of arrows and the clarity of our interpretability experiments. We were glad to hear that you appreciated the breadth of our experiments and found the problem we are investigating important.

---

However, the method by which SAE-based probing is evaluated ("quiver of arrows") is nonstandard. The claim that such an approach is needed for robustness is not convincing to me, as it is widely practiced to compare the individual accuracies of different models.

Thank you for bringing up this point! The quiver of arrows approach we introduce is non-standard in the literature, but we adopt it to make the strongest possible case for SAEs. Since we select the best method using validation AUC, we expect to choose SAEs only for tasks where they perform best. To verify this, in the three settings where we employ the quiver of arrows—standard conditions, data scarcity, and class imbalance—we compare its performance to that of using a single SAE across all tasks, as shown below.

Clearly, the quiver of arrows serves as an upper bound on the performance of any individual SAE. As an alternative counterfactual, instead of comparing against a single SAE probe, we select the best SAE for each task using validation AUC (a “quiver of SAEs”) and compare this to the overall quiver of arrows with both baselines and SAEs. Again, the baseline + SAEs quiver outperforms the just SAE quiver.

At a higher level, the claim that SAE-based probing does not have advantages for interpretability is limited by the fact that there are not comparisons to baselines for the most standard interpretability task (interpreting latents). The paper claims that "these findings may be possible using baseline classifiers," (line 373), but this is limited to theoretical speculation. An evaluation here would be more convincing. In general, the other evaluations in Section 4 seemed to be somewhat ad hoc / anecdotal.

Thank you for pointing out that line 373 was confusing! In section 4.3, we zoom into two datasets as case studies and show that this is not just a theoretical worry: baseline classifiers can find spurious correlations and noisy labels as well as SAEs. We have modified this paragraph to be as follows:

The spurious latent category seems especially promising because finding a spurious latent may help us identify spurious features in the dataset. However, in a cast study in \cref{sec:ai_vs_human}, we find that similar findings may be possible using baseline classifiers: we apply a logistic regression probe to model hidden states on tokens from the Pile \cite{pile} and show that maximally activating examples also exhibit the spurious correlation. 

However, a practical advantage for SAEs is that the infrastructure to perform autointerp is pre-existing through platforms like Neuronpedia, and a theoretical advantage is that the baseline classifier can only identify the single most relevant coarse-grained feature, while the decomposability of SAE probes into latents allows for identifying many independent features of various importance.

For your first point, we are not sure what you mean by comparisons to baselines for interpreting latents. Because SAEs are an unsupervised method, it is not clear what a baseline for interpreting a latent would be, as other methods do not have an equivalent to latents (i.e. units a probe can be broken down into). This decomposition is an advantage of SAEs, but it is unclear how much value it gives. In section 4.2, we are investigating what we can learn about different SAE latents by examining the datasets that they are discriminative for (as opposed to auto-interp or human interpretability of latents, which typically looks at top activating examples).

---

Thank you again for taking the time to review the paper and providing helpful feedback! Do the above actions address your concerns with the paper? If not, what further clarification or modifications could we make to improve your score?

Thank you for your time and help! We were very glad to hear that you appreciated our use of the quiver of arrows technique and found our evidence and claims clear.

---

While I do observe evidence that SAE probes contribute less to downstream use cases than other probes, the gap is not incredibly large, especially in Fig. 5.

Thank you for pointing this out! Figure 5 shows the result of the quiver of arrows approach, and so the SAE quiver achieving the same performance as a baseline quiver only shows that SAEs do not actively make the practitioner worse off. We have added a new table in the appendix which directly compares each SAE to logistic regression.

However, I was unconvinced by the claims in Section 5 (see strengths/weaknesses)...

Thanks! We agree and have moved it to the appendix.

However, I would like to see more elaboration of why the imbalanced dataset settings give the SAE’s inductive bias an advantage.

Thank you for pointing this out! We have created a new Appendix section to detail the general SAE probing intuition and for each section. We paraphrase below:

‘’’ “..We argue that if SAEs are successful at this task [creating a sparse interpretable model basis], requiring a probe to only use a sparse set of directions in this basis should serve as a beneficial inductive bias to prevent overfitting with limited data.” Class Imbalance: Because SAE latents are sparsely activating, choosing SAE latents that are positive on the minority class and negative on the majority class may generalize well. ‘’’

Section 5 is a little confusing to me. Bricken et al. also use max pooling on their baseline probes… Thus, it is hard to argue they present an “illusion” of SAE probes being better…

Thank you for pointing this out! We attempt to argue that max pooling activations is not the strongest baseline possible. That being said, we agree that our language regarding an “illusion” is too strong, and we have toned down the language in the paper.

Additionally, it is hard to understand if your results in the third graph of Figure 11 came from softmax-pooling or the quiver approach… Also, why do the win rates not sum to 100%?

We apologize that we were not more clear here! The graph compares two quivers, quiver(SAE max pool, SAE last token) vs. quiver(activations softmax,activations last token). The win rates do not sum to 100% because we consider test AUCs within 0.005 to be tied, and thus count as a win for neither method. We have added additional clarification to the text and figure caption.

In 2.3, when introducing your probing methods, you seem to be listing the hyperparameters for each method…The writing should be improved here.

We agree, thank you! We have now formatted this section as a table.

…”we train probes using the largest L0”...What is the “largest l0”?

Thank you for noting this! For any given SAE width, we use the largest available L0 from GemmaScope (and noted this in the paper).

…the “Quiver of Arrows” setup is a core part of your model… I would appreciate seeing this discussed in the main body.

We agree! We moved it to the main body.

…consider flipping the x-axis of Figure 19…

Thanks, we just made this change!

…~25%… There is a typo here with the ~

Good catch! We have updated the paper to replace it with the correct value (22%)

1)…Did your GemmaScope and LlamaScope SAEs not come with already-interpreted latents? 2

2. Also, the autointerp process you used is not described in your paper.

1)Gemma/LlamaScope SAEs do not come with already-interpreted latents, they just contain the SAEs themselves. 2) we have now added the following language to Section 4.1: “For this and all subsequent experiments, we generate autointerp labels using Neuronpedia, which leverages a language model to produce consistent natural language explanations for a latent based on its top activating tokens.” Thanks for the update!

In section 4.2 when you consider the top 128 latents is this again by mean difference across positive and negative samples in your binary classification tasks?

Yes, we have fixed this in the text, thank you!

---

Thank you again for taking the time to review the paper and providing helpful feedback! Do the above actions address your concerns with the paper?

Thank you for your insightful questions and comments! We are especially grateful for your suggestions on better latent selection methods, and we are glad you feel our paper is a valuable contribution.

---

However, since SAE latents (e.g. in TopK SAEs, Gao et al. 2024, L086 right) are not on the same scales, it seems like this could be misleading.

We agree your method of choosing latents is better, thank you! We find this technique improves test AUC when k is small (<32) but not when k is large (see https://anonymous.4open.science/r/SAE-Probes-B404/rebuttal_plots/comparing_new_and_old_mean_diff_auc.png). Intuitively, your method better finds the “correct” latents for small k, but the old method is “good enough” for large k.

We reran our main experiments with this technique. Unfortunately, the baseline + SAE quiver still fails to improve over baselines. Investigating further, we find that we select k = 128 probes from the quiver 80% of the time, so quiver performance is dominated by k = 128 probes, which do not improve that much.

If the SAE was "chosen" based on its performance for 14 tasks (L184-188 right), shouldn't [Figure 4] have 14 points above the diagonal?

When an SAE is chosen the point may be on or below the diagonal, since the test AUC might decrease or stay about the same. We’ve added “Datasets not directly on the diagonal signify that an SAE method was chosen from the quiver” to the Figure 4 caption.

In Section 4.2 (and Section 4) in general, results from autointerp are assumed to be "ground truths". It would be helpful if this could be evaluated, e.g. using humans for a small subset of latents.

This is a valid point, thank you! To remove this confounder in Section 4.1, two of the authors independently labeled and ranked latents, with no change in results. However, manually labeling the latents for all datasets in Section 4.2 is labor intensive (and thus a good use case for autointerp).

could it be possible… that SAEs would perform well at a different layer?

This is a great point that we did not think of, thank you! We checked the width=16k largest L0 SAEs on the 4 layers in Figure 12a (layers 9, 20, 31, and 41), and found that layer 20 was also the best for SAEs as well, see https://anonymous.4open.science/r/SAE-Probes-B404/rebuttal_plots/comparing_sae_test_auc_by_layer.png. We have added this plot and a discussion to Appendix D.

Results in Section 4, and particularly Section 4.3, are on specific handpicked datasets. Do they generalize? Alternatively, is there a reason for picking these specific datasets?

We apologize for not being clearer here; we intended for Section 4.3 to be a set of case studies complementing Section 4.2. We selected these datasets after an investigation of five datasets whose top latent representations exhibited strong performance. During this analysis, we discovered that both 87_glue_cola and 110_aimade_humangpt3 contained labeling errors. We have altered the introduction to section 4.3 to reflect this.

Why is a different SAE config used for experiments with label noise (L263) and covariate shift (L294)?

For label noise, we use the SAE with smallest width (16k) and maximal L0, which we found to be most performant in standard conditions (we have added this justification to the paper). We would have used this SAE for the covariate shift domain as well, but there is no support for generating autointerp explanations through Neuronpedia for this SAE, so we instead use the width = 131k, L0 = 114 SAE. We have added additional clarification around this choice in the paper, thank you for this comment!

In Section 4.1, why is k=8 used, when k=16 and k=128 have been used everywhere else?

We do so because the probe pruning experiment is a proof-of-concept experiment and is significantly simpler with smaller k.

L158, right: why only use logistic regression for SAE probes?

This is a great question! We think this is a good choice because logistic regression is common in practice and is the best baseline activation methods. This is still a valid concern, so we have added the following to our limitations section: "Finally, it is possible that further optimization of the SAE probe baseline might increase performance such that it beats baseline methods. For example, we only tried logistic regression on SAE probes, and it is possible that other probing techniques could perform better."

L027-033: sentence is hard to read

We agree and have rephrased to: “However, although SAEs occasionally perform better than baselines on individual datasets, we are unable to ensemble SAEs and baselines to consistently improve over just baseline methods.”

---

Thank you again for taking the time to review the paper and providing helpful feedback! Do the above actions address your concerns with the paper, especially with regard to the better top k latent selection? If not, what further clarification or modifications could we make to improve your score?