Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control

Dear Reviewer,

Thank you for your detailed and thoughtful review. We especially appreciate your careful analysis of the paper's structure and technical content.

Regarding additional tasks

• See our top-level comment on additional tasks, where we generalize to new models, datasets and tasks.

Regarding presentation and readability: We have:

• moved some crucial formulae from the appendix to the main text (e.g., the necessity formula from Section 7.4)
• put the information important to our paper about the new SAE variants (TopK and Gated) and how they're trained in the main body
• extended the limitations section and added some directions for future work
• moved the related work section to directly follow the introduction
• added pointers from Figure 1's caption to relevant sections of the paper
• reorganized the interpretability sections for better coherence
• fixed the minor mistakes you pointed out
• moved one of the exploratory figures on interpretability to the appendix

Regarding technical issues and experimental results:

• On sufficiency in Figure 3: We appreciate your point about averaging potentially hiding per-example changes. We have added a note of the distribution of changes in logit differences to address this concern; briefly, we observe empirically that logit differences are well-concentrated around their means.
• On necessity: We agree this important formula should be in the main text and have moved it there. However, we disagree with your characterization that showing necessity necessarily means the difference between 1) and 2) (in your terminology). Specifically, with our setup, a necessity score close to 1 indicates that removing our reconstructions degrades performance similarly to removing all task-relevant information via mean ablation, suggesting our features capture the essential computation. We think this is the meaningful way to evaluate whether the SAE error term contains information relevant for the task.
• On probing accuracy in Control: We have revised this section by adding results (figures contained in Appendix) that substantiate the claims made in this part of the paper.
  • You said: "This is because the probe is linear, which itself can "disentagle" the attributes. ". However, this indicates a misunderstanding of the experiment. What we check here is two things:
    • whether a probe trained to predict e.g. IO will "change its mind" when we edit the IO attribute in an activation;
    • Whether probes for the other attributes will keep their original predictions
  • In other words, we use probes merely as one tool to measure the information encoded in activations. We use this to evaluate whether our interventions have the desired effect on the information encoded in the activations.

Regarding your questions:

• About the bias term: yes, this is by design -
  • It is well known that activations in LLMs may have non-zero average values over a dataset, especially when this dataset is very narrow/specialized compared to the full pre-training distribution. The bias term is a way to account for this, and let us focus on how activations differ between examples on the IOI task.
  • As another justification, SAEs also have a similar bias term added to reconstructions. In this way, our supervised dictionaries imitate this design choice.
• About F1 score: you are correct; we have added more explanation about this (with examples) to the relevant parts of the paper.

We are very grateful for your many detailed observations about our paper. Please let us know if these revisions address your concerns. We remain available to answer any additional questions during the discussion period.

Before this phase of the discussion period ends, we wanted to ask the reviewer whether we have addressed your concerns with our work?

We thank you for the valuable feedback, especially regarding presentation, readability, and the structure of this work, and we believe we have now resolved these issues.

Thanks to the authors for addressing the comments. I especially appreciate the inclusion of the necessity formula being moved to the main text, adding some limitations and future work, as well as moving the exploratory figures to the appendix. I think the current presentation is better than before - but there are still places of minor improvements.

• F1 score. I still don't think the F1 score description in section 3 is clear enough. If I understand correctly, the F1 score is a measure for interpretability, which I think in the original comment it is out of place. The description of "A" is still not clear. Is it done automatically, or does it depend on the choice of human-defined subsets as in section 5.5? More comments on the sections on interpretability follows.

• Figure 3. The supervised/Task SAE/Full distribution SAE are introduced. The Task SAE and Full distribution SAE was slightly introduced in the introduction. I think adding a sentence reminding what Task SAE and full distribution SAE in the beginning of section 4.3 (just like the beginning of 5.1) will do.

• Sufficiency: I apologize that I did not raise this last time. I think for clarity, one should describe it in terms of formulae. For each p, we have a logit difference logitdiff(p) given by the difference in model logprobs for IO and S names. Thus, if we perform the intervention, we will have a different logit difference, say $logitdiff_{intervention}(p)$. From the text, it seems like what you were doing was to do $E[logitdiff_{intervention}(p)] / E[logitdiff(p)] \approx E[logitdiff_{intervention}(p)] / 3.3$. (Also, are there absolute values any where?) Also, thanks for addressing the concerns of the per-example change.

• Necessity: Thanks for addressing the concern. I think the presentation has improved a lot. It would be nice to say it even clearer, if I understand correctly, as

  • $a_{intervention}(p)$ = E(..) + error term,
  • $a_{mean ablation}(p)$ = E(..),
  • $a_{clean}(p) = a(p)$.

Then the subscript in logitdiff will be the same as the "a"'s. In this case, we also don't have to use the left arrow to indicate assignments. Regarding the "difference between 1 and 2", I agree that the formula is a reasonable measure for necessity.

• Probing accuracy: Thanks for addressing the comments. I think I understand what you are doing in this section.

• Bias term: Thanks for addressing it in the comments.

Upon a second reading, regarding the interpretability sections including the section at the end of section 4 as well as section 5.5,

• I think the interpretability at the end of section 4 is very interesting - and I think that alone can be a separate paper, with more experiment results. Thus one could remove this section and put it in the future work. The reason I think it is out of place is because a) this section is not related related to any of the other parts of the paper; b) the results are only shown in the appendix; and c) both the methods and the results are interesting on their own.

• Section 5.5 is less interesting (to me) compared to the end of section 4, but it is more related to the other parts of the paper. It is still slightly out of place because the methods and experiments in this section are not clearly described. (They are of course much more clearly described in the Appendix)

• Both sections are labelled "interpretability" under "evaluating Sparse Feature dictionaries" and "evaluating SAE", but they correspond to different methods. This is also confusing.

My original suggestion was to remove both sections - or put it in the appendix. I still think this should be the case. However, in case the authors strongly prefer to keep these sections, I think they should at least rename the "interpretability" section at the end of section 4 to "Feature-level Interactions" instead.

I also appreciate the addition of more discussions on limitations and future work. The removal of these interpretability sections also can benefit a more thorough discussion on the results and the limitations. For example,

• section 5.3, "We find that vanilla task SAEs are good at sufficiency/necessity, but full-distribution SAEs are not.". Are there any discussions or explanations on why this is the case?
• Expand on how to extend the proposed methods to datasets other than IOI. Thus, can we compute supervised feature dictionaries without knowing the set {IO, S, Pos}? Can we evaluate sufficiency and necessity using similar methods for SAE features?

The above two questions are just suggestions of discussion items if the interpretability sections are removed.

Dear reviewer,

Thank you for your detailed and thoughtful comments.

We note that a central concern in your review is the restricted nature of our evaluation (considering only the IOI task with a single model and dataset). To this end, we have applied our framework with minimal changes to a new setup combining a new model, new pre-training dataset, and new linguistic task, for the purpose of evaluating SAEs trained on the full pre-training dataset of the LLM being analyzed. Please refer to our top-level comment on generalizability of results for more details.

To address your other concerns:

• While we agree that focusing on IOI may seem narrow, we believe it serves as a valuable proxy for evaluating SAEs more broadly. Practitioners often need to evaluate SAEs across hyperparameter sweeps or compare new architectures, and the IOI task better represents realistic downstream applications than standard proxy metrics. Importantly, even though IOI is a special case where prior work identified relevant features, evaluating full-distribution SAEs on this task provides evidence for their effectiveness on tasks where ground truth is unknown.

• Regarding how attributes are chosen: you say that "In particular, when there are many choices of attributes, what is the manner of choosing between them without using another interpretability method, which would then prevent the need of using an SAE in the first place?" - however, this objection runs orthogonal to the goal of our paper.

  • Specifically, our goal is to, through whatever means, find a situation in which we can have an independent ground-truth evaluation of SAE quality. The prior work on IOI gives us (through lots of manual effort) a good understanding of the IOI circuit, which allows us to set up an independent evaluation by comparing features in the IOI circuit (found via non-SAE means) to SAE latents.
  • In our newly added task, we have shown that we can construct supervised features like we do for IOI (via conditional expectation over attribute values), and we check that the supervised features disentangle two relevant properties for the task. This serves as validation that the supervised features are relevant to the model's computation (similar to IO, S and Pos).

• You say "Some examples of /possible/ interpretations are provided in Appendix 7.13-7.14, but if I understand correctly these are not the actual labels of the SAE features." - indeed, the labels of SAE latents in figure 5 are the ones from Appendix 7.13; we are sorry for the misunderstanding!

• You say "How would one do this searching or interpretation without access to the counterfactual activations? Wouldn't it be more realistic to interpret or label each SAE feature and then use the features that are labelled to be relevant to the task at hand?" - we are sympathetic to your objection; some qualifications:

  • The reason we set things up in this way is that we first ran an extensive validation showing that IO, S and Pos are indeed meaningful for the model's computation on IOI. In this sense, it makes sense from a human perspective to want to see how well SAE latents can control these attributes.
  • We agree this is somewhat biased, and have tried to be agnostic to whether SAEs represent these features in a 1-to-1 way or some more complex pattern. Still, this does not fully address your concern. We hope that future work can move towards an even more agnostic evaluation.

Please let us know if these responses address your concerns. We remain available to answer questions throughout the discussion period.

Dear reviewer,

Thank you for your comments. We appreciate your focus on strengthening the paper's motivation and empirical validation. We have made several significant changes to address your concerns:

Motivation and Novelty: We have substantially reworked the introduction and related work sections to better motivate our contribution (see top-level comment). The key insight is simple: while SAEs are increasingly used for model interpretability, existing evaluations rely only on indirect metrics. Our work provides the first ground-truth evaluation framework using supervised features of the same mathematical form as SAEs.

Additional Empirical Results: We have strengthened the empirical validation by:

• Adding a complete new case study applying our framework to a different model, dataset, and linguistic task (see the top-level comment on generalizability of results)
• Demonstrating that our supervised features successfully disentangle relevant task attributes in this new setting
• Showing how our evaluation methodology transfers with minimal changes

English Language: We have carefully revised the manuscript for clarity and readability, with particular attention to:

• Technical explanations and terminology
• Flow between sections
• Motivation of methodological choices

Please let us know if you would like us to clarify or expand on any of these changes. We remain available for questions throughout the discussion period.

Before this phase of the discussion period ends, we wanted to ask the reviewer whether we have addressed your concerns with our work?

We believe we have addressed the concerns you raised regarding motivation, clarity of language, and including sufficient technical detail.

Dear reviewer,

We warmly thank you for your enthusiastic reception of our paper and your thoughtful comments. Regarding the relevance of our work to the literature (where you noted your expertise might be limited): ours is the first work to provide a thoroughly validated approximation of ground-truth features to evaluate SAEs against. We believe our evaluation framework provides important results for the IOI task, and we have also shown it can be readily generalized to other linguistic tasks. We have reworked the introduction and related work sections to more clearly situate our paper within the field of SAE evaluations, which we hope will help readers better understand our contribution.

For a concise summary of the main changes in the revision, please also consult our two top-level comments:

• On novelty/motivation: we describe briefly and clearly our contribution and how we re-worked the paper to present it better
• On generalizability of results: we give a concrete example with another task where we show that our approach for computing supervised dictionaries and evaluating SAEs works too.

We have also incorporated your helpful suggestions:

• Moving the related work section to directly follow the introduction
• Adding pointers from Figure 1's caption to relevant parts of the paper

Thank you again for your constructive feedback. We remain available to answer any questions.