Sparse Autoencoders for Hypothesis Generation

Thank you for your detailed, positive review. We are glad you find the claims convincing, the paper well-presented, and the method efficient.

To reply to your questions:

(1) “The method proposed here is also subjected to the context window limit[...]” Thanks for pointing this out; we will clarify in the paper. HypotheSAEs doesn't run into context window limit issues in practice because it applies LLMs only to interpret SAE neurons, for which the LLM requires only a few examples (we test this in C.2). In contrast, prior methods use LLMs to learn the interpretable features in the first place, which requires reasoning over many examples.

(2) “it claims the dimension of z_i to be [...] highly interpretable [...] without any supporting evidence” Thanks for this point. Supporting references are in Sec 1+2.2, and we will include them here as well. However, this sentence is meant to motivate using SAEs, rather than to prove this claim; our experiments on real datasets demonstrate it holds in our setting. One quantitative metric (see response to R2) is that interpreting SAE neurons yields much higher fidelity vs. interpreting embeddings directly (F1: 0.84 vs. 0.54).

(3) “Interpreting each single neuron of the activation matrix to be a nature language concept is problematic” Thanks; we agree this is a key challenge, which is why we conduct extensive experiments to maximize fidelity to the underlying neuron (Appendix C). Empirically, Figure 3 demonstrates that the loss in predictive performance due to using discrete concepts to approximate the neurons is quite small on average (-2.4%).

(4) “How stable are the interpretations when the examples are changed” Thanks for this question. We ran a new experiment to measure stability. For 100 random neurons on the Yelp dataset, we generated 3 interpretations with different random seeds, and computed text embeddings of all interpretations. The mean pairwise cosine similarity of two interpretations generated for the same neuron is high: 0.84 (vs. 0.34 for a pair of interpretations from two different neurons). This is despite two sources of randomness—sampling different examples and an LLM temperature of 0.7—showing that the underlying concept learned by each neuron is durable. Here is a set of interpretations for a random neuron (can share more upon request):

Neuron 50 (stability: 0.76): ['discusses the seasoning of food, either praising it as perfect or criticizing it as lacking', 'mentions seasoning or lack of seasoning in the food', 'mentions seasoning or lack thereof in the context of food flavor']

(5) “Interpreting the neurons with LLM requires per dataset manual efforts to tune the bin percentiles”; “How sensitive is the method to different sparsity levels (k in TopK)”. Thanks for these questions about hyperparameters. We ran some experiments and found that results are not particularly sensitive; they work well with defaults:

Headlines, using default bin [90, 100] instead of [80, 100] as in paper: AUC 0.70, 11/20 significant (still beats all baselines) Yelp, using default k=8 instead of k=32 as in paper: R^2 0.77, 14/20 significant (~identical to original)

We provide hyperparameter guidance for practitioners in Appendix B, with more detail in the Python package we released publicly (which, unfortunately, we aren't permitted to link here).

(6) “Lack of human evaluation for the interpretation quality of the LLMs and the generated hypothesize” Thank you; in light of this comment, we conducted a qualitative human eval where we asked three computational social science researchers (not involved with the paper) to evaluate all significant hypotheses on the Headlines and Congress datasets. We followed prior HCI work (Lam et al. 2024, “Concept Induction”) and asked them to annotate for “Helpful” and “Interpretable” hypotheses. We use the median of the three ratings. HypotheSAEs substantially outperforms baselines in terms of raw counts and percentages: 24/30 (80%) are rated helpful, and 29/30 (97%) are interpretable.

Results plot: https://imgur.com/a/qw6bt3s

We also spoke to domain experts about novelty; see our reply to R3. We hope these findings increase your confidence that our hypotheses are high-quality.

(7) “Does the H [...] equal the dimension M” No: M is the total number of SAE neurons, from which we select H predictive neurons to interpret. We choose it based on the validation AUC of how well the SAE neurons predict y (see B.1 + our repo).

(8) “Did you observe any hallucinations from the LLM when labeling neurons?” ~ We did not observe hallucinations in the usual sense, but some interpretations did not describe the neuron well, which is why we generate multiple interpretations and choose the highest fidelity one. In practice, our results are not very sensitive to this step (see B.2).

Given the strengths you highlight, and our experiments to address your comments, would you consider raising your score? If not, do you have further questions?

Thank you for the detailed review and suggestions. We're glad you found our method original, clearly presented, theoretically grounded, and computationally efficient, with the performance gains convincingly supported.

To respond to your questions and suggestions:

(1) “ ‘Broad applicability beyond text-based tasks claim’ is overstated”. Could you clarify which part of the paper you’re referring to (we couldn’t find this quote directly)? We agree that we do not provide evidence that the method generalizes beyond text-based tasks; we would be happy to revise specific areas where you thought this was implied.

(2) “What if we directly use pretrained LLMs [...] as one of the baselines?” We included a baseline, HypoGeniC, which uses pretrained LLMs directly. This method performed worse across 14 of 15 quantitative comparisons and was ~10x more expensive and ~30x slower, owing to the need to make many LLM calls to select candidate hypotheses. If you have specific suggestions for other ways to use pretrained LLMs, please let us know!

(3) “Novelty claim could benefit from additional validation, especially through expert evaluation.” Thank you; based on this suggestion, we reached out to two experts to assess the novelty of our findings on the Headlines dataset (Table 1 of the paper). In particular, we asked them whether the three hypotheses most negatively associated with engagement—"addressing collective human responsibility or action", "mentioning environmental issues or ecological consequences", "mentioning actions or initiatives that positively impact a community"—were novel or reported previously. They did not provide papers with these specific findings; for example, one expert noted, “I don't know of any papers specifically looking at [the hypotheses you mention]”. Both experts pointed us to theories that are broadly supported by these findings: e.g., Robertson et al. (2023) find that negativity drives news consumption, which is consistent with the third hypothesis.

In light of your and R4’s comments, we also conducted a human eval where we asked three computational social science researchers (not involved with the paper) to evaluate all significant hypotheses on the Headlines and Congress datasets. We followed prior HCI work (Lam et al. 2024, “Concept Induction”) and asked them to annotate for “Helpful” and “Interpretable” hypotheses. We use the median of the three ratings. HypotheSAEs substantially outperforms baselines in terms of raw counts and percentages: 24/30 (80%) are rated helpful, and 29/30 (97%) are interpretable:

Hypothesis Human Eval - Imgur

We hope these new findings increase your confidence that the results from HypotheSAEs are (1) novel, as per our correspondence with experts; and (2) helpful and interpretable, as per our human evals.

(4) You asked about “the necessity of the specific theoretical bound (proposition 3.1) in practical performance.” We agree the theoretical bound is not strictly necessary for practical performance, but rather serves as a broader motivation for the procedure, as you note. It also provides us a way to conceptually decompose hypothesis generation performance into (1) neuron predictiveness, and (2) interpretation fidelity. C.3 explores this empirically.

(5) You asked about “the difference between ... the paper and more complex scientific hypothesis generation scenario in the papers below.” Thank you for these references, which we will include in an additional related work paragraph titled “Automated Idea Generation”. We agree that these literatures are both in service of helping researchers conduct science, but they address different tasks:

Our work is focused on solving the problem: “given a dataset of texts and a target variable, what are the human-understandable concepts that predict the target?” In contrast, the literature you mention focuses on: “given a corpus of prior scientific papers, can we propose promising research ideas?” Practically, the former task emerges when a researcher knows what they are studying and have collected data, but they need tools to make sense of the data. The latter task emerges when a researcher is trying to decide what to study based on prior literature. Methodologically, the former task involves methods like clustering & interpretability, while the latter involves methods like prompting & RAG. We think these two literatures are distinct, but complementary; for example, a political scientist might use an idea generation method to decide to study partisan differences in social media posts, and then, after collecting a dataset, use HypotheSAEs to propose data-driven hypotheses for further validation.

(6) More formal language; appendix table of contents: Thank you! We will make these edits.

Given the strengths you highlight, and our experiments to address your comments, would you consider raising your score? If not, do you have further questions?

Thank you for the thoughtful and positive review. We’re especially glad to read that as an interpretability researcher, you found our work to be “highly significant because it is the most compelling example I have yet seen of sparse autoencoders beating baselines on a real task that people have actually tried to solve with other methods.” Indeed, we agree that recent literature is mixed on what value SAEs contribute, and we see the method as highlighting an area in which they have a real advantage: the ability (as you note) to find interpretable and unexpected patterns in data. We will further highlight this contribution in the paper.

We also thank you for suggesting simpler baselines, which we agree would clarify the importance of the SAE. We ran several of them on the three real-world datasets (Headlines, Yelp, Congress). Besides the feature generation step, the baselines are identical to our method.

Embedding: Run Lasso to select 20 dimensions directly from embeddings. PCA: Transform embeddings to PCA basis (k=256, which explains ~80% variance), then select 20 dimensions using Lasso. Bottleneck (suggested by R1): Fit a simple neural network which uses the embeddings as input, a hidden 20-neuron “bottleneck” layer, and predicts the target as output. Then interpret the 20 bottleneck neurons. (This is conceptually similar to NLParam, but simpler.)

Metrics are combined/averaged across the three datasets. Recall that count is the number of significant hypotheses in a multivariate regression; predictiveness is the AUC (or R^2 for Yelp) using all hypotheses together; fidelity is the F1 score for how well the interpretation matches the neuron activations.

Using the SAE produces many more hypotheses which are significant in a multivariate regression (45/60 across the three datasets for SAE vs. 30/60 for the next best baseline); slightly higher predictiveness; and much higher interpretation fidelity. This is consistent with our qualitative finding that SAE neurons fire on more specific, distinct concepts than embeddings. Specificity permits the high-fidelity interpretations, and distinctiveness results in the wide breadth of hypotheses. While these baselines perform reasonably well in terms of predictiveness, for downstream utility, predictiveness is only a prerequisite: we ultimately want hypotheses that are non-trivial and non-redundant (as is discussed in Sec 6.2).

We believe the results of these baselines clarify the value of the SAE in our setting, and we are excited to include them in the updated version of the paper.

Regarding your suggestion to fit an n-gram/bag-of-words model: in the paper, we perform a related analysis for the Congress dataset, using n-gram results from Gentzkow et al. (a seminal, widely-cited analysis). They fit an L1 regression using bigrams and trigrams, and report 600 predictive ones. Controlling for counts of the n-grams on this list, we find that our hypotheses improve AUC from 0.61 to 0.74 and 28 out of 33 remain significant. This suggests that n-grams aren’t sufficient to cover our hypotheses. Qualitatively, we find hypotheses which we think n-grams can’t easily capture, like “criticizes the allocation of resources or priorities by the government, particularly highlighting disparities between domestic needs (e.g., education, healthcare, energy) and actions abroad (e.g., Iraq)”.

Thank you also for the pointer to Matryoshka SAEs. We’ve been excited about these as well, and plan to add them to our codebase and try them. We agree they may reduce the need to stitch together SAEs of different sizes.

We agree that working with domain experts would be a valuable direction to increase confidence in our results. In the last week, we took some initial steps here (more detail in the response to R3): we (1) confirmed that two domain experts did not know of specific work producing several of our findings on the headlines dataset (the researchers also noted that some of these findings support theories in psychology) (2) conducted a human eval for whether our hypotheses are helpful and interpretable, which produced strongly positive results (https://imgur.com/a/qw6bt3s).

We share your desire to make our code easy to work with. In fact, we’ve released a public codebase with a pip-installable package, though unfortunately we are not allowed to link to it (even anonymously).

Thank you again for your suggestions, and we’re really happy to see that you found the paper to make a significant contribution to interpretability research.

Thank you for the thoughtful and positive review. We’re glad that you found the claims of the paper to be very clear and well supported.

We agree that a significant benefit of our method is that it reduces compute requirements by 1-2 orders of magnitude, and believe this will facilitate real-world applications. We would like to also emphasize what we see as an even more important benefit of the method: its performance in identifying predictive hypotheses. On synthetic tasks (where we know ground-truth hypotheses), the method outperforms all three baselines on 11/12 metrics. On real-world tasks, the method outperforms all three baselines on 5/6 metrics, generating 45 significant hypotheses out of 60 candidates, while the next best generates only 24.

We also appreciate your feedback about the phrase “triangle inequality.” Perhaps we could call it a “sufficient condition for hypothesis generation.” We would be curious for your thoughts on this phrase. An alternative could be to drop the phrase.

We also thank you for suggesting ablations, which we agree would clarify the importance of the SAE. On the three real-world datasets (Headlines, Yelp, Congress), we ran both ablations you mentioned—using embeddings directly and fitting a bottleneck—and a third suggested by R2, which selects features from an embedding PCA. We keep other steps of our method fixed.

Embedding: Run Lasso to select 20 dimensions directly from embeddings. PCA (Suggested by R2): Transform embeddings to PCA basis (k=256, which explains ~80% variance), then select 20 dimensions using Lasso. Bottleneck: Fit a simple neural network which uses the embeddings as input, a hidden 20-neuron “bottleneck” layer, and predicts the target as output. Then interpret the 20 bottleneck neurons. (This is conceptually similar to NLParam, but simpler.)

Metrics are combined/averaged across the three datasets. Recall that count is the number of significant hypotheses in a multivariate regression; predictiveness is the AUC (or R^2 for Yelp) using all hypotheses together; fidelity is the F1 score for how well the interpretation matches the neuron activations.

Using the SAE produces many more hypotheses which are significant in a multivariate regression (45/60 across the three datasets for SAE vs. 30/60 for the next best baseline); slightly higher predictiveness; and much higher interpretation fidelity. This is consistent with our qualitative finding that SAE neurons fire on more specific, distinct concepts than embeddings. Specificity permits the high-fidelity interpretations, and distinctiveness results in the wide breadth of hypotheses. While these baselines perform reasonably well in terms of predictiveness, for downstream utility, predictiveness is only a prerequisite: we ultimately want hypotheses that are non-trivial and non-redundant (as is discussed in Sec 6.2).

We believe the results of these baselines clarify the value of the SAE in our setting, and we are excited to include them in the updated version of the paper; thank you for suggesting them!

We believe that we have now addressed the only weakness you raised (comparison to additional baselines) and were hoping, in light of this, that you would be willing to raise your score. If not, please let us know if there are further concerns we could address or experiments we could conduct. Thank you again for your suggestions, which have strengthened the paper!