Automatically Interpreting Millions of Features in Large Language Models

We agree with this comment by the reviewer. Our generated interpretations only interpret the activations of individual latents, and are far from full explanations of their behaviour and downstream impact. Intervention based methods, like the one we proposed, are the ideal candidates to probe the causal role of latents. We believe that a combination of both is the most informative.

(2) Why were Surprisal and Embedding scores included despite their weak correlation with human evaluations? Do you see specific scenarios where these scores are more useful than Fuzzing, Detection, or Simulation?

We propose surprisal as a seemingly natural method to score interpretations, with the hope that it inspires future work in that direction, and that future work is aware of the limitation of this technique. Although we probably won’t use surprisal scoring much in the future, we believe it to be valuable for the community to know that it was tried.

On the other hand, embedding scoring is much faster and cheaper than all the other techniques, even though such comparisons are less apples-to-apples than the comparison between fuzzing/detection with simulation. This is because, not only are the input tokens lower but the embedding model used can be as small as 700M parameters. This makes for a very compelling reason to use embedding scoring, if one wants to quickly evaluate the interpretability of latents, potentially on the fly while training.

The fact that latents with low embedding scores are generally low scoring in fuzzing/detection can be used to filter out incorrectly interpreted latents or find latents which are generally not interpretable.

We thank the reviewer for this feedback.

Lack of direct numerical comparison with existing methods

We compare our scoring techniques with the standard scoring technique at the time of writing, simulation scoring. In Table 1, we compare the costs of both scoring techniques, and we perform human score correlations, lines 322 left. We also perform inter-score correlations, Tables A1 and A2.

Is there a specific comparison that the reviewer would like to see?

Following 1, it is unclear how much interpretability quality improves or is sacrificed for scalability.

Improvements in scalability were mostly on the generation of scores, which we discuss in Table 1. This means that the proposed techniques do not sacrifice interpretability quality for scalability. Indeed we believe that the proposed guidelines for generating and evaluating latents improve interpretability quality instead of sacrificing it. Some of our proposed scores correlate well with human given scores.

Missing the analysis of failure cases. It is important to be aware of the safety boundary of the proposed method in implementation.”

We currently provide one failure case in Figure A2 of the appendix, where the explainer model incorrectly identifies the explanation, one that is more easily gathered from top activations - as we write this we notice that the caption of the figure is missing, and that it indeed discussed this. We discuss this failure mode in the text. It is hard in general to discuss failure cases in interpretability, and even harder to define safety boundaries. We wonder if the reviewer has specific comments on how to improve on this.

Thank you for your helpful comments.

While being a useful tool for a broad community (both interpretability community and people who want to use the interpretability toolsets), the overall insightful findings and novelty in the method are limited. The automated interpretability pipeline is not new.

At the time of writing, evaluating SAE interpretability focused on evaluating the top activating examples. We show that this is misleading for multiple reasons. Firstly, this gives an illusion that SAEs are more interpretable and monosemantic than they are (we show this in figure 2), ignoring the full distribution. There was also the impression that the lower activations of SAEs were completely uninterpretable and less valuable, which again we show not to be the case. We think this to be a valuable insight.

Our scoring techniques improved on the state of the art, and are now the standard instead of simulation scoring. We think that our discussion on interpretability scoring is also helpful for the community, and such an analysis is not common. Intervention scoring was a type of scoring that was missing in the current framing of SAE interpretability.

We would love to be able to produce more rigorous results that would convince the reviewer that the paper strongly merits acceptance. If there are specific examples that the reviewer would like to be done more rigorously we would take those into consideration.

We thank the reviewer for your helpful comments.

The authors only used one LLM to interpret SAE in this article. It would be more convincing if the authors could consider more LLMs and perform prompt sensitivity analysis.

In the current version of the article, we evaluate the explanation given by Llama 3.1 70b, Llama 3.1 8b and Claude 3.5 Sonnet, as shown in table A8. We discuss prompt sensitivity analysis in the last reply of this rebuttal

The author claims in the title to explain millions of features in LLM, but it seems that this article explains the massive units in SAE latent space.

In this work we use features to mean interpretable directions in both the residual stream and the MLP outputs. SAEs were developed because directions in the residual stream and MLP (by looking at neurons for instance) were often polysemantic and hard to understand. The interpretability community uses feature to mean interpretable concepts, and so we believe the claim of explaining millions of features to be correct, being similar to the one in Templeton et al., 2024. We have added to the introduction the following sentence: "Throughout this work we are going to be using latent to mean a specific pair of encoder-decoder indices in SAE, and feature to mean the concept that this latent is representing.”

What if we let an LLM explain its own SAE? What if we use an additional LLM to explain another LLM's SAE? The author can explore this.

We did both of these in this work. We used Llama 8b to generate explanations for the Llama 8b SAE, and in general have used other LLMs – Llama 70b – to generate the explanations for the SAEs trained on LLama 8b and Gemma 2 9b. Llama 8b is worse at generating explanations than Llama 70b and at the time of writing there were no Llama 70b SAEs, which was the smaller model that gave reasonable explanations, so these were the only experiments we could perform.

Can this method identify some latent units that do not have obvious conceptual information?

Some units have simple conceptual interpretations (they fire on the verb to be, or on chair) and some have more complicated patterns, like firing in text between parenthesis. We are not sure if this is what the reviewer meant to ask.

The method in this paper is very interesting, but the experiments seem to be insufficient, for example, there is a lack of some sensitivity analysis, specific ablation experiments, or more potential downstream applications, etc.

We have ablated model size (for both explainer and scorer), number of examples shown, number of tokens per example shown, origin of the examples shown, using chain-of-though or no and, showing token activations or. We also investigated the interpretability of different layers, whether the interpretability of the residual stream SAEs and of SAES trained the output of the MLP, as well as the dependence of interpretability on the SAE expansion factors. Is there a specific ablation that the reviewer has in mind?