Revising and Falsifying Sparse Autoencoder Feature Explanations

We thank Reviewer LBic for the constructive review. We address your concerns as follows.

---

W1. There is no human evaluation to confirm whether the proposed methods improve automatic interpretability of SAEs in a meaningful way. Furthermore, using LLM as a judge potentially introduces biases that could affect evaluation reliability, particularly for concepts across different languages or domains.

A1. Regarding human evaluation, evaluating all features would be prohibitively expensive. However, we have performed extensive case studies across various aspects. We provide examples of semantically similar negatives falsifying SAE explanations in Appendix C, tree-based explainer generating higher-quality explanations in A1 of our response to Reviewer gHHG, and structured explanations capturing polysemantic concepts in A4 of that same response. These case studies confirm our methods generate more accurate explanations for SAEs. We have also provided code that facilitates reproduction and manual inspection of our results.

While LLM judges can introduce bias, we don't use them in our main experiments. Explanation performance in Section 5.2 is measured by the correlation coefficient between simulated and ground-truth activations, consistent with previous work [1]. We only use an LLM judge when measuring feature complexity, where we use carefully-written few-shot examples and confirm through manual inspection that the LLM's judgments align with human intuition.

References:

[1] Bills et al. Language models can explain neurons in language models. OpenAI blog.

---

W2. All experiments are conducted on an uncopyrighted subset of the Pile, which may not fully represent all possible concepts that a model might encode. A more detailed discussion on how the dataset choice might affect the results would be beneficial.

A2. Dataset choice can indeed affect results. However, the Pile is large, and our 100,000 samples should capture the vast majority of concepts in SAE features. If a concept encoded by an SAE feature were absent from the dataset, we would observe no activations—yet this never occurred in our evaluated features. All features in our experiments showed a significant number of activating sentences, making our results meaningful.

---

W3. While features can be polysemantic as defined in L46 and L201, more thorough explanations of how this can be different from the property of features in SAE latent space that is known to be more monosemantic and interpretable (L31) is needed, to provide readers with clarity.

A3. Thank you for this suggestion. In L31, SAE features are described as monosemantic and interpretable relative to neurons. The large latent dimension of SAEs allows us to decompose polysemantic neurons into more monosemantic concepts. However, the typical latent dimensions in our experiments (16k to 64k) are still far fewer than the concepts encoded in the LLMs' latent space. Therefore, SAE features must still be inherently polysemantic, making it essential to break down this polysemanticity for accurate explanations. By representing polysemantic concepts in a structured format, we obtain a clear measure of feature polysemanticity and improve explanation quality.

---

Q1. Typos and missing citations.

A4. Thanks for pointing these out. We'll fix the typos and add the missing citations.

---

Q2. L200: You mentioned that the explainer LLM is prompted such that each component corresponds to a monosemantic concept. Is there any guarantee for that?

A5. While there is no hard guarantee, we observe that during the tree-based explainer's iterative refinement, it tends to break down polysemantic concepts into monosemantic components. Explanations with clear, monosemantic components achieve higher validation scores and survive the pruning process. Through case studies (e.g., A4 in our response to Reviewer gHHG), we find this pattern holds across all inspected features.

---

Q3. Do you have any conjecture regarding how different width of SAEs might affect the quality of explanation for polysemantic feature?

A6. This is an insightful question. SAEs are trained with a sparsity loss in the feature space. As SAE width increases, they tend to decompose polysemantic features into more monosemantic components, a phenomenon known as feature splitting [1]. Therefore, we believe SAEs with higher widths would exhibit lower polysemanticity. However, since the latent dimension of SAEs cannot exceed the number of concepts in natural language, polysemanticity will always exist in SAE features, necessitating better handling methods. Our proposed structured explanations better capture this polysemanticity, resulting in consistent performance improvements.

References:

[1] Chanin et al. A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders. arXiv preprint arXiv:2409.14507.

---

We hope our response addresses your concerns. If you have further questions, we're happy to address them in the discussion period.

Dear Reviewer LBic,

Thank you once again for your thoughtful and constructive feedback on our submission. We deeply appreciate your engagement during the review process and are grateful for your positive assessment of our work.

As the discussion phase is drawing to a close, we wanted to kindly confirm whether there are any remaining concerns or clarifications we can provide. We are happy to respond to any further questions you may have.

Thank you again for your time and support.

Best regards,

The Authors

We thank Reviewer i1y8 for the constructive review. However, we believe there are some misunderstandings regarding our paper. We'll address your concerns as follows.

---

W1. However, it is not clear how the proposed method helps in understanding LLMs compared to Bills. […]

A1. Thank you for the comment. We respectfully disagree that our paper does not help with understanding LLMs. In fact, our work improves this understanding in several important ways:

1. Improving explanation accuracy matters. SAE research aims to understand LLMs by interpreting their features. But if those explanations are inaccurate—too broad or missing polysemanticity—they hinder, rather than help, our understanding. Many current auto-interpretability methods suffer from these issues [1,2]. Our work directly addresses them.
2. We are the first to tackle overly broad explanations. Prior methods (like [1]) use evaluation metrics that don’t penalize overly inclusive explanations. This allows inaccurate, misleading outputs. Our method introduces semantically similar negative sentences to detect and fix overly broad explanations. Figure 3 shows that our tree-based explainer reduces this problem through iterative refinement.
3. We are the first to systematically capture polysemanticity. Polysemanticity—when a feature activates on unrelated concepts—has been a known problem in XAI. We use a structured explanation format that not only captures this property but also lets us quantify it. In experiments, structured explanations consistently outperform unstructured ones across datasets and settings (see responses to Reviewer q24V and A4).
4. We analyze how feature complexity and polysemanticity evolve. Using our methods, we examine how these properties change across layers and models (Figure 5). For example, middle layers tend to encode more complex concepts, and stronger models (GPT-2 < Gemma 2 < Llama 3.1) have more complex and polysemantic features. These patterns deepen our understanding of LLM internals.

Regarding the reviewer’s point about L316-317: those lines refer only to the limits of holistic activations and training negatives, which we report as negative results. The new methods we introduce—semantically similar negatives, structured explanations, and the tree-based explainer—directly address these problems and show clear performance gains (Figures 3 and 4).

In short, our paper directly tackles major open issues in SAE interpretability and provides tools that improve both explanation quality and our understanding of LLMs.

References:

[1] Bills et al. Language models can explain neurons in language models. OpenAI blog.

[2] Paulo et al. Automatically Interpreting Millions of Features in Large Language Models. In ICML.

---

W2. The contribution of the paper is marginal as it is mainly the approach of Bills et al. with a new prompting strategy.

A2. Our approach goes well beyond just a new prompting strategy. The tree-based explainer automatically improves explanations using feedback from validation sets. It produces structured, intuitive explanations and introduces a way to detect overly broad features using semantically similar negative sentences—something not done before. These methods are part of an iterative refinement process that significantly improves explanation accuracy and helps us better understand LLMs.

Also, our contribution is not marginal compared to Bills et al. That work focuses on generating and evaluating SAE feature explanations but does not address major issues like overly broad features and polysemanticity. As a result, their explanations are often inaccurate. In contrast, we directly tackle these issues and show clear improvements (see our response to Reviewer gHHG). Finally, while Bills et al. is mainly methodological, our paper also analyzes how complexity and polysemanticity change across layers and models, offering deeper insight into LLM behavior (see Figure 5).

---

W3. After reading the paper, I'm not sure about the take away of the paper except from what was already know from Bills et al. […]

A3. We address how our paper goes beyond Bills et al. in A1, but to summarize: our work is the first to systematically tackle the issue of overly broad and polysemantic SAE features. Unlike Bills et al., which focuses mainly on methodology, we use our tools to study the internal structure of LLMs. For instance, we show that middle layers tend to encode more complex and polysemantic concepts, and that stronger models (e.g., GPT-2 < Gemma 2 < Llama 3.1) have more such features. This directly connects to the reviewer’s question—polysemanticity helps models represent more concepts per neuron, making them more capable.

We also understand the concern about using one black box to explain another. In our main experiments, though, we don’t rely on LLM judges—explanations are evaluated using correlation with ground-truth activations, following Bills et al. We do use an LLM judge to estimate feature complexity, but we design high-quality few-shot prompts and find that its judgments align well with human intuition.

---

W4. Certain choices of design decisions are not well-justified.

A4. We justify our design choices as follows:

• We use maximum activation to rank sentences so that a few repeated activating words don’t dominate the score.
• We sample randomly from the top quantile instead of just taking the top-activating sentences to better capture polysemantic features. Looking only at the top examples can make a feature seem monosemantic. For example, the 1st feature in Gemma 2 9B seems to focus on "instrument," but looking deeper reveals it also activates on "warn" and "warning." Sampling more broadly helps uncover this.
• As for how we define "top," we split the dataset into 1000 activation quantiles and sample from the top one. To address concerns about cherry-picking, we ran experiments on different quantiles on the 10th layer of Gemma. In all cases, the tree-based explainer consistently outperforms the one-shot one.

---

Q1. In the abstract it is stated the tree-based method "refines explanations". This is assessed in 5.3 (Feature Complexity) by using an LLM judge. How can we sure that LLM judge does a good job?

A5. Thank you for the question. We believe there may be a misunderstanding. Section 5.3 is not used to evaluate the quality of explanations from the tree-based explainer. That evaluation is done in Section 5.2, where we use the correlation between simulated and actual activations—just like in Bills et al.

Section 5.3 instead studies how feature complexity changes across layers and models. Here, we use an LLM judge, but we design high-quality, human-written few-shot prompts to guide it. Based on manual checks, the LLM’s ratings align well with human intuition.

---

Q2. What does "close counterexample" mean? How is closeness mathematically defined?

A6. "Close counterexamples" are sentences that are semantically similar to the top-activating sentences but on which the SAE feature doesn't activate. They are useful for exposing overly broad explanations, as shown in Figure 3. The definition involves semantic similarity which is challenging to describe mathematically.

---

Q3. The creation of the child explanations is without a score. How is it possible then to perform the pruning in step 3?

A7. The child explanations are scored on a validation set, and these validation scores are used for pruning. This process is iterative: at each iteration, we generate variations of current explanations, score them on validation sets, then prune the explanations using validation performance. We'll formalize this into pseudo code in our paper to improve clarity.

---

Q4. Does no ground-truth mean zero activation?

A8. Yes, no ground-truth means the SAE feature has zero activation on the sentence.

---

Q5. […] However, looking at Fig. 3 it is above 1. How is this possible?

A9. The value in Figure 3 refers to the number of false positives in the sentence. To get the ratio of false positive elements, we need to divide it by the length of sentences.

---

Q6. L306: Is there any evidence for this hypothesis?

A10. On L306, we hypothesize that the tree-based explainer is capable of packing polysemantic meanings into a single explanation. This is mainly derived from case studies. We provide examples of one-shot and tree-based explanations below, where the one-shot explanation misses polysemantic components and achieves a lower score, while the tree-based explainer successfully packs polysemantic concepts into a single string.

Example 1:

• One-shot (score: 0.738): instruments and related equipment or devices
• Tree (score: 0.894): The neuron is looking for mentions of specific types of devices, tools, or equipment, particularly when referred to by their common names or technical terms, such as "instruments", "instrument", or warning messages like "warnings" or "Warning".

Example 2:

• One-shot (score: 0.126): references to software licenses and technical documentation metadata, such as version numbers and programming terms.
• Tree (score: 0.270): The neuron is looking for specific keywords related to licensing, particularly "License" and "Apache License", as well as some LaTeX-related keywords like "article" and specific package names. It also activates on certain version numbers and technical terms like "time" in the context of Java imports.

---

We hope our response addresses the reviewer's concerns and would appreciate your consideration in raising the score. Should you have any further questions, we welcome the opportunity to address them during the discussion period.

[3.25] The word 'become' and its variations in different contexts.
[2.36] Common words or phrases indicating change of state, such as 'becomes'.
[3.06] The use of intensifying adverbs.
[1.27] Wright
[1.15] netto
[1.17] sie
[1.09] Webber
[1.02] Cliff Richard
[1.13] born
[1.73] ( date of birth )
[3.53] Mathematical expressions and formulas, particularly those containing symbols like $, \, and $.
[2.95] The character 'lambda' and related scientific concepts.
[2.23] Physical quantities such as length, radius, and size.
[2.47] The word 'contact' in various contexts, including forms, emails, and contact pages.
[1.73] Words related to forms, such as 'form' and 'forms'.
[3.00] The words 'different', 'differentiate', 'confused', or 'same' when in context of comparing two things.
[1.92] Specific biological terms like 'lymphocyte', 'B cell', 'antigen', or 'immune functions'
[1.80] Chemical symbols or molecular terms, such as 'beta', 'g', or 'F'
[1.74] Product codes or names with specific formatting, such as 'Y', or 'OHN'
[3.38] Immunology-related terms, specifically 'antigen', and concepts related to immune response
[2.03] Cell-related terms, like 'B cell', 'lymphocyte' or 'immune functions'
[1.83] The word 'year', especially when used in a temporal context (e.g., 'this year', 'per year')
[3.11] The phrase 'year' with adjacent positive or optimistic words
[1.42] The word 'season'
[1.27] free King The Kid ringtones
[1.20] DU Battery Saver
[1.86] World Conqueror
[1.31] VPN
[1.83] Google Speed Tracer
[1.27] The word 'want'

We thank Reviewer gHHG for their thoughtful review. Below are our responses to their concerns.

---

W1. The key weakness for me is that the paper lacks qualitative examples that show why the proposed method is useful. While quantitatively one can see that it is better at finding false positives (Figure 3) and getting better scores (Figure 4), how does this finally affect the kind of explanations that are obtained for SAE latents? For instance, are more granular explanations obtained? Showing examples of explanations for latents along with what inputs activate them would be very helpful (including negative examples). […]

A1. Thank you for this constructive suggestion. The tree-based explainer can generate more granular, accurate, and concise explanations compared to the one-shot explainer, thanks to its ability to iteratively refine explanations based on evaluation feedback. We demonstrate this through three examples where the tree-based explainer outperforms the one-shot explainer, plus one negative example showing its limitations.

Example 1: A birthdate feature

• One-shot explanation components (score: 0.062):
  • Names of people, specifically first and last names.
  • Date formats and structures, including days, months, and years.
• Tree-based explanation components (score: 0.621):
  • born
  • date of birth in parentheses
• Example activations (with activated tokens in bold):
  • Anabel Englund (born September 1 , 1992 ) is an American singer and songwriter
  • Kim Soo-yong (born September 23, 1929) is a South Korean film director
  • Simon Blair Doull (born 6 August 1969) is a New Zealand radio personality, commentator and former international cricketer

This feature activates on birthdates of people in parentheses. While the one-shot explanation broadly includes names and date formats, the tree-based explanation captures the specific pattern more accurately.

Example 2: An import and JavaScript keywords feature

• One-shot explanation components (score: -0.025):
  • Programming syntax related to imports, specifically import statements.
  • Code comments.
  • Specific programming keywords.
• Tree-based explanation components (score: 0.117):
  • JavaScript/React keywords and syntax
  • import and export statements
  • slider and view related terms
  • querySelector and document
  • formData and FormData
• Example activations:
  • import { NUM_BINS, PRESETS } from "./ config ";
  • export default Object.freeze({
  • const grid = document.querySelector('# Symbols .grid');

This polysemantic feature activates on various programming contexts, including import statements and JavaScript keywords. The tree-based explanation provides more detailed coverage of these activation patterns.

Example 3: A parenthesis feature

• One-shot explanation components (score: 0.100):
  • Mathematical and programming syntax, especially with context of abstract algebra and mathematical concepts.
  • Use of parenthesis '('
• Tree-based explanation components (score: 0.602):
  • '('
  • ')'
• Example activations:
  • create a ( mutable ) object first, and at certain scenario I need to make it immutable
  • I allow other (mac ) machines to access an IIS7 hosted MVC3 application on my (win7 ) machine
  • I'm looking for a formula to calculate the ( product ?) of an arithmetic series

This feature simply activates on parentheses. The tree-based explanation captures this concisely, while the one-shot explanation includes unnecessary context about mathematical concepts.

Negative example: A people name feature

• One-shot explanation components (score: 0.251):
  • Names of people, including first and last names.
  • Proper nouns, including names of songs, albums, movies, and places.
• Tree-based explanation components (score: 0.039):
  • Wright
  • netto
  • King
  • Cliff Richard
  • Lloyd Webber
• Example activations:
  • billionaire Nirav Modi, could have cost $2 billion
  • 1965 popular song with lyrics by Hal David and music composed by Burt Bacharach
  • "You have seen into the heart of my music," Andrew Lloyd Webber said to Gale Edwards in a trans

This feature activates on people's names, but the tree-based explainer overfits by producing a list of specific names. Such cases are rare in our experiments.

---

W2. A problem that has been observed with SAEs is feature absorption and feature splitting. Intuitively, one would expect that this approach would be able to detect such instances better. This could be a very useful evaluation to have to show the practical benefits of the approach, […]

A2. Thank you for this helpful suggestion. We hypothesized that feature splitting and absorption would result in more false positives, and tested this empirically. Following [1], we examined SAE features that activate on words starting with specific English letters. We measured feature splitting and false positives from the simulator (which activates on words beginning with particular letters). Here are the correlation scores at different feature splitting thresholds:

The high correlation ($r>0.5$) between feature splitting and false positive rate suggests our approach offers an efficient way to detect feature splitting without training multiple k-sparse probes.

We also found a correlation of 0.1042 between feature absorption and false positives—statistically significant but more subtle than feature splitting.

References:

[1] Chanin et al. A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders. arXiv preprint arXiv:2409.14507.

---

W3. The trends shown for explanation complexity and polysemanticity appear somewhat weak, and a more granular analysis would be helpful. For example, the results also depend on how good the trained SAE is, which would vary across models and layers (in terms of reconstruction loss, latent dimension size, sparsity, etc.) and could affect the conclusions made.

A3. You raise an important point about how SAE training quality affects feature complexity and polysemanticity. While we kept latent dimensions consistent across layers, reconstruction loss and sparsity varied. For a more detailed analysis, we measured correlations between complexity and reconstruction loss ($r_1$), and between polysemanticity and reconstruction loss ($r_2$), across different layers:

The positive correlation between these metrics and reconstruction loss makes sense—more complex and polysemantic features are inherently harder for SAEs to learn.

We also examined correlations between complexity and L0 ($r_3$), and between polysemanticity and L0 ($r_4$):

Llama was excluded as it uses top-k activation. The results show complexity correlates positively with L0, while polysemanticity shows no significant correlation.

---

W4. Additionally, similar to W1, having examples would also help here, particularly for polysemanticity. For example, are the explanations for the polysemantic neurons conceptually similar or very distinct?

A4. For polysemantic neurons, structured explanations better capture conceptually distinct components compared to unstructured explanations. Here are two illustrative examples:

Example 1:

• Unstructured explanation (score: 0.894): The neuron is looking for mentions of specific types of devices, tools, or equipment, particularly when referred to by their common names or technical terms, such as "instruments", "instrument", or warning messages like "warnings" or "Warning".
• Structured explanation (score: 0.967):
  • The word 'instrument' and its plural 'instruments'.
  • Warnings in code, such as 'Warning' and 'warnings'.
• Example activations:
  • Copyright (C) 2004 Texas Instruments
  • This is a follow up to Is -Wreturn-std-move clang warning correct in case of objects in the same hierarchy

This polysemantic feature activates on both "instruments" and "warning"—distinct concepts that structured explanations capture more clearly.

Example 2:

• Unstructured explanation (score: 0.493): The neuron is looking for phrases or sentences that express sincerity, emphasis, or encouragement, often marked by words like "mean", "seriously", "come on", or phrases that start with "let's be honest". It also seems to activate on certain punctuation marks like commas that precede or follow these phrases.
• Structured explanation (score: 0.589):
  • The word 'on' in various contexts.
  • The word 'mean' when used for emphasis.
  • Expressions of informality or emphasis like 'come on', 'hey', or 'seriously'.
  • Commas in certain phrases.
• Example activations:
  • Hang on for a minute...we're trying to find some more stories you might like
  • By other js files, I mean the files you include in popup.html
  • Come on in and stay for the video

This feature also activates on conceptually distinct patterns, which structured explanations capture more effectively.

---

We'll incorporate these additional results and discussion in our paper. We appreciate your feedback and are happy to address any further questions.

Dear Reviewer gHHG,

Thank you again for your valuable comments and detailed review of our paper. We have carefully considered your feedback and submitted a comprehensive rebuttal addressing all the points you raised.

As the discussion phase is approaching its end, we would like to kindly confirm whether we have adequately addressed your concerns. If any questions remain or further clarification is needed, please don’t hesitate to let us know—we are happy to respond promptly.

If you feel that our responses sufficiently address your concerns, we would be grateful for your consideration in revisiting your evaluation.

We sincerely look forward to your feedback.

Best regards,

The Authors

We thank Reviewer q24V for the constructive review. We address your concerns as follows.

---

W1. Focuses on top-activating examples, which are known to be significantly more interpretable than medium or low activation quantiles.

A1. Thank you for this observation. We focus on the top-activating quantile because SAE features only activate on a tiny fraction of tokens, making sentences in medium and low activation quantiles essentially random. Our evaluation approach on top-activating quantiles aligns with standard practices [1].

To verify that our methods remain superior across different activation levels, we tested on the 10th activation quantile (out of 1000 total). For lower quantiles, sentences would be uninterpretable due to the sparse nature of SAE activations. In this experiment, we combined sentences from this quantile with an equal number of random sentences to form training sets for generating explanations and test sets for evaluation. This setup mirrors our paper and [1], with the only difference being that we replaced top-activating sentences with those from another quantile.

Results for the 10th quantile are shown below:

These results demonstrate that structured explanations consistently outperform unstructured ones, and the tree explainer generally surpasses the one-shot explainer. However, we should note that sentences in lower quantiles tend to be more random, making scores from those quantiles less informative.

References:

[1] Bills et al. Language models can explain neurons in language models. OpenAI blog.

---

W2. Tree-based explanation method is computationally intensive.

A2. We agree that the tree-based explainer requires more computational resources than the one-shot explainer, particularly at larger depths and widths. However, this cost comes with a significant performance improvement for SAE feature explanations (~20% increase in correlation score). This represents a substantial advance in mechanistic interpretability approaches and enhances our understanding of LLMs. Given that SAEs are widely used for understanding and controlling LLM behaviors, accurate feature explanations are essential. Additionally, the tree-based explainer's depths and widths can be adjusted to balance efficiency and accuracy.

---

Q1. Have you tested your innovations on lower quantiles of the activation distribution? Do you anticipate that your results will be qualitatively different on, say, the middle tercile of the feature activations?

A3. As shown in A1, we tested our methods on the 10th quantile and consistently observed improvements over baselines. This confirms our approaches' superiority across different activation quantiles. For even lower quantiles, SAE activations are essentially zero, rendering the sentences uninterpretable.

---

We hope our response addresses your concerns. If you have further questions, we will be happy to address them during the discussion period.

Dear Reviewer q24V,

Thank you once again for your thoughtful and constructive feedback on our submission. We deeply appreciate your engagement during the review process and are grateful for your positive assessment of our work.

As the discussion phase is drawing to a close, we wanted to kindly confirm whether there are any remaining concerns or clarifications we can provide. We are happy to respond to any further questions you may have.

Thank you again for your time and support.

Best regards,

The Authors