Thank you for your thoughtful review and detailed feedback on our paper. Below, we address your points and clarify the changes and additions we have made in response to your comments:

Comparison with existing methods

In addition to the paragraph in Related Work summarizing past work on interpreting ICL, we have added a more extensive review in the revised paper as Appendix H. This section compares our method to various existing approaches in cases where comparison is possible and draws some connections to past work. In particular, we find similarities with work on anchor tokens from Wang et al. 2023, find similar findings regarding approximation errors in Bansal et al. 2022, and highlight our focus on mechanistic interpretability for larger models on real-world discrete tasks.

Generalization to more complex ICL scenarios

From our experiments with algorithmic tasks, we hypothesize that more complex tasks may involve less interpretable features. For example, features in the algo_last task were still somewhat interpretable, as they activated on word repetitions. However, features for tasks like algo_first and algo_second were less clean and harder to interpret from manual inspection of features of Gemma Scopes – they did not seem to activate on examples where a second instance of an object was selected. Despite this, these features still exhibited noticeable and selective effects as shown by steering experiments, indicating that they play distinct and meaningful roles in the ICL process. Since our findings and investigation methods are considerably more in-depth than prior work on task vectors, we are unsurprised that there are some difficulties further scaling our method. However, we think there are no fundamental limitations with SFC and task features and it is a promising area for future research to address.

Details on hyperparameter selection and sensitivity

We have conducted $L_1$ coefficient sweeps for multiple models and SAEs, including Gemma-1, Gemma-2 2B (16k and 65k sparse autoencoder widths), and our own Phi-3 sparse autoencoders. These results have been added to the Appendix D.1. They show that our method performs generally well across those models and SAEs, being able to reduce active SAE features by 50-70%, while preserving task vector performance.

Approximation Errors

We have clarified the limitations section (the paragraph in Section 6: conclusion) to better explain the implications of the approximation errors in our analysis of attention heads versus MLPs. When formulating the experiment, we assumed that attention alone was sufficient to describe the causal link between task-detection and task-execution features. However, our experiments revealed that this link required running the MLP on top of direct attention outputs. Moreover, Figure 6 uses frozen attention patterns with ablations on pre-OV residuals instead of SFC nodes, moving the causal link analysis even further from the SFC subcircuit. Since SFC uses transcoders instead of MLPs, this may also introduce additional approximation errors, as this experiment does not account for transcoder effects and relies solely on MLP outputs.

Hello reviewer JwLJ, we only have 23 hours left to be able to respond to your questions and concerns. We think that we've addressed them as detailed in our rebuttal. Let us know if you have any further questions

We thank the reviewer for their thoughtful feedback and valuable suggestions. Below, we address the points raised:

Limited evaluation on additional models

We would like to clarify a potential misunderstanding regarding our experimental setup. This study primarily uses the Gemma-1 2B model, and we trained our own SAEs specifically for this research. Our research started before the release of SAEs for Gemma-2, which is why our main results use our Gemma-1 SAEs. However, we have since expanded our analysis to include $L_1$-regularization coefficient sweeps for SAEs from multiple configurations: our Gemma-1 2B (32k), our Phi-3 (50k), and Gemma-2 2B (16k and 65k), as can be seen in Figures 9-13. Our results show that we can reduce the number of active features in task vectors to a fraction of the original, while maintaining performance across setups. We also provide a positive steering chart for cleaned Gemma 2 2B 16k and 65k SAE features (Appendix F.2) together with some of their max activating examples (Appendix I). While we acknowledge the importance of applying our method to larger models such as Gemma-2 9B, these models require more complex infrastructure setups. As a result, they are the target of our future research, while this paper focuses on smaller models.

Training costs

We calculated the costs for training SAEs on TPUs in Appendix B, coming to a figure of 1 week of TPU v4-8 usage. It is likely training costs would lessen significantly on the new generation of TPUs and specifically the v5-e, which would enable us to save device memory and time spent on inference by performing matrix multiplication with quantized weights.

Task vector cleanup is a comparatively cheap algorithm, completing cleanup for a single task and value of $L_1$ coefficient on one TPU chip in about a minute for models ranging from Gemma 1 2B to Phi 3 Mini.

$L_1$ regularization sweeps

We have included $L_1$ regularization sweeps across the above setups in the Appendix D.1. These results highlight the optimal $L_1$ coefficient​ values for obtaining meaningful sparse features and show that TVC is able to consistently reduce the number of active task features by at least 50% (and up to 70%), while maintaining task vector performance.

Separation of task-detection and task-execution features

The distinction between task-detection and task-execution features lies both in their activation timing and pattern, and their causal roles within ICL:

• Task-detection features mainly activate on the output tokens in the few-shot example pairs on earlier layers, thus detecting the task.
• Task-execution features activate later in the models, on arrow tokens specifically.

These patterns also persist in their max activating examples from the training data. Executors activate on the tokens just before the something similar to their task is completed, while detectors activate on the tokens that complete this task. That is tokens and and cold in the hot and cold example. We have added max activating examples from the SAE training distribution of those two types of features to make the distinction more clear in Appendix I.

Their relationship is supported by experiments such as those in Figure 6, which show that negative steering with task-detection directions on output tokens disables related task-execution features.

Ablating sparse features

While ablating a few hundred nodes can significantly impair performance, each individual node typically has a low IE, meaning its removal does not entirely disable functionality. The SFC paper was able to disable mechanisms by ablating less than 50 nodes , which is significantly lower than our numbers. This is likely due to the difference in the size of the model (Gemma 1 2B is 30x larger than Pythia 70M) and restoration effects within the model (Explorations of Self-Repair in Language Models by Rushing and Nanda, 2024). The steering experiment in Figure 6 demonstrates that ablating task-detection directions disables related task-execution features, although this is not a node-based analysis.

Functionality of non-task-specific features in task vectors

Many features in task vectors activate on generic patterns commonly associated with ICL, such as arrow tokens or repeated commas. These generic features are less specialized and do not encode the task itself, while still boosting the task vector performance. To illustrate this, we have added full positive steering heatmaps to the appendix (Appendix F), showing that extra features beyond task executors often have broad, unspecific effects.

Thank you for the response and the efforts made in revising the manuscript. Below, I provide my updated comments on the work.

Evaluation on additional models: I appreciate the inclusion of results for Gemma2 and the exploration of SAE width variations, as suggested by reviewer VLpn. However, I disagree with the assertion that evaluating Gemma2-9B requires significant infrastructure setup. Given that v4 TPUs were utilized in this work, the computational resources should be sufficient for running experiments on Gemma2-9B. Since SEA training is not required and the vector clean-up algorithm is computationally inexpensive (as mentioned in the rebuttal), this evaluation should be feasible.

$L_1$ regularization sweep: It would be valuable to assess how performance varies when the SAE is trained under different sparsity constraints. As Gemma Scope provides SAEs trained with varying $L_1$ regularization strengths, this experiment would not require training SAEs from scratch, making it relatively straightforward to conduct.

Separation of task-detection and task-execution features: Thank you for addressing this point and revising Line 407. This clarification highlights an essential distinction between task-detection and task-execution features. But I do think this additional statement is important to distinguish these two features. To further strengthen the argument, empirical evidence should demonstrate that these two feature types exclusively activate for their respective input types.

Ablating sparse features and Functionality of non-task-specific features in task vectors: Thank you for addressing my earlier concerns in the revision. The response provided has adequately resolved my questions on these topics.

Additional consideration: After reviewing the revised manuscript and considering feedback from other reviewers, I believe the paper requires significant restructuring. Several critical details currently relegated to the appendix should be included in the main text to enhance the paper's coherence and readability.

Reviewer JwLJ also raised the important point regarding the roles of MLPs and attention heads. While this has been acknowledged as a limitation in the revised manuscript, I believe it is crucial to address this issue directly in the experiments. Open SAEs like Gemma Scope already include SAEs for both MLPs and attention modules, so conducting such experiments should not be resource-intensive.

Based on the response, the revisions, and feedback from other reviewers, I would like to adjust my rating to 5 and suggest for a complete revision of the work to address these points and improve its structure and presentation

Thank you again for your feedback. We ran additional experiments and updated the main body of the paper for clarity:

Gemma 2 9B: We completed TVC L1 coefficient sweeps with the bigger model by running it with Tensor Parallelism. The canonical Gemma Scope SAEs produced results better than ITO and SAE encoding without major tuning, bu we did need to change the early stopping steps parameter from 50 to 200 to achieve higher sparsity by letting the TVC algorithm run for longer. We have considered SAEs of width 16k and 65k, as for our Gemma 2 2B experiments. The new results are in Figure 15.

Task Detector and Task Executor Feature evidence: To test the extent to which task executor and task detector features activate on their hypothesized token types in distribution, we computed the sum of the activation values of those features on each of the token types and found the percentage of this activation mass corresponding to each of the token types. The results are presented in Tables 1 and 2. They show that over 96% of the activation mass of task detector features corresponds to output tokens, and almost 90% of the activation mass of task executor features is in arrow tokens, which is entirely in line with our expectations. From our results, the evidence is clearer for task detectors being distinct than for task executors, but it is still strong for both types of features.

Less than 1% of the activation mass of task detector features is on arrow tokens. However, around 5% of the mass of task executor features is on output tokens, primarily on translation tasks. This may be due to features that activate weakly on many tokens, even though they have the causal effects of task executor features and activate before task execution, like those seen in Appendix I. Their presence does not detract from the significance of our finding because an overwhelming majority of features follow the expected behavior.

Restructuring the paper: We deeply appreciate the reviewer’s valuable feedback regarding the placement of details within the manuscript. To better align with their suggestions, we have revised the paper by moving certain elements from the appendix to the main body. Specifically, we have incorporated the flowchart illustrating the Task Vector Cleanup algorithm as the new Figure 2 and provided additional explanations within the main text. Furthermore, we have included several maximum activating examples from Appendix I in Figure 4 with activation mass distribution tables for detectors and executors (Tables 1 and 2). These additions clearly demonstrate the distinct activation patterns and timing of these feature types. We sincerely hope these adjustments contribute to a more coherent and readable presentation for the esteemed reviewer.

Regarding SAE training $L_1$ regularization sweeps: To reiterate, we include experiments sweeping across the $L_1$ coefficient used in the TVC algorithm in the Appendix. In the above comment, the reviewer requested sweeps across the $L_1$ coefficient used for training SAEs, which is not a parameter of our algorithm and is a detail of the SAE used which may affect quality or sweep performance.

While we acknowledge the importance of the TVC $L_1$ coefficient sweeps, we maintain that they are supplementary to our paper's primary contribution. As noted by Reviewer VLpn, one of our paper's key strengths lies in "the successful implementation of a relatively complex combination of methods" that demonstrates the potential of SAEs. Therefore, while we have included comprehensive L1 sweep results in the appendix, we believe this supporting material is appropriately placed.

Following the feedback, we re-ran our experiments on our experiments on SAEs with varying L1 coefficients (and, as a result, varying average L0 norm) for Gemma 2 2B, recording our results in Figures 13-15. Our method performs well across L0s ranging from 20 to 300, with the only notable takeaway being that very high L0s seem to produce lower L0 task vectors with smaller TVC L1 coefficients.

We would also like to clarify our previous discussion about approximation errors: these do not affect the validity of our detector-executor relationship findings. Our results demonstrate robust causal relationships between these feature types, supported by our expanded empirical evidence.

These revisions maintain the paper's focus on its core contributions while addressing the broader scope of model evaluation and empirical validation requested by the reviewers. We believe these changes have significantly strengthened the manuscript while preserving its essential contributions to understanding ICL mechanisms in large language models. Given the substantial effort to incorporate the reviewers' feedback and enhance the paper's clarity and rigor, we hope these improvements merit a reconsideration of the evaluation score.

Hello reviewer BufG, we only have 23 hours left to be able to respond to your questions and concerns. We think that we've addressed them as detailed in our rebuttal, so could you let us know ASAP about further worries, or otherwise would you be open to revising your amended judgement?

We appreciate the time and effort the reviewer has taken to engage with our work. Below, we aim to clarify the understanding of our contributions, particularly in light of the points raised regarding their significance and clarity.

Contributions

Contribution 1: Effective Use of Sparse Autoencoders for Large-Scale Mechanistic Interpretability

Our primary contribution lies in demonstrating that Sparse Autoencoders (SAEs) can be effectively used to explain mechanisms behind in-context learning (ICL) tasks in large models, such as Gemma-1 (2B), which is 10–20x larger than models typically analyzed at this depth in mechanistic interpretability research (e.g., Wang et al., 2022; Marks et al., 2024). This scalability is not merely incremental; it is a significant technical achievement because it shows that causal circuit-finding methods, like SFC (sparse feature circuits), can scale to large language models (LLMs), enabling interpretability on tasks where smaller models may not produce the same behaviors. Our work shows a strong lower bound for how useful SAEs are for explaining ICL, and is successful as a proof-of-concept for interpreting ICL with SAEs

Contribution 2: Identification of Core Bottlenecks in ICL Circuits

Our second key contribution is the identification of two core bottlenecks in the ICL circuit: task-detection features and task-execution features, along with an in-depth analysis of their interactions. These features are not only causally important to the ICL circuit but also interpretable. Specifically:

• We demonstrate that task-detection features and their corresponding task-execution features are causally linked via attention mechanisms and MLP layers. Figure 6 shows that by steering with negative decoder directions we can disable corresponding executor features.
• With steering experiments in Figures 3 and 5, we show that both task-detection features and task-execution features can independently trigger task execution.
• We also found that both types of these features are often interpretable. To give more evidence for this, we have added maximum activating examples on the SAE training data of the cleanest features we found as Appendix I.

Contribution 3: Task Vector Cleaning (TVC)

We introduced Task Vector Cleaning (TVC), a novel method that decomposes task vectors into a small set of interpretable and task-relevant features, while keeping their performance intact. This approach addresses the limitations of ITO (inference-time optimization, Smith et al. 2023), which usually degrades the performance and may miss important features. In response to feedback, we have expanded our analysis of TVC to include different models and SAE sizes. We compare its performance to ITO and task vectors in SAEs of two widths for Gemma 2 2B and on our own Phi-3 SAEs. This broader evaluation demonstrates that the method is effective across a range of architectures, affirming its general applicability and robustness. These results can be found in the new Appendix D.1.

We are open to clarifying and discussing these contributions further in the comments, and would appreciate you reconsidering your opinion if we’ve been able to clarify our paper’s findings more clearly.

Stylistic changes

We appreciate the stylistic feedback, and the new revision is adjusted accordingly. Additionally, we inserted citations for causality literature, and, for easier reading, added references to sections describing our contributions when they are mentioned.

We cite papers describing the algorithms we use in detail, such as Sparse Feature Circuits in Section 2.2 and Task Vectors in Section 2.3. We also include pseudocode with concrete hyperparameters. We describe notable changes in Section 4, providing enough detail for a reader to be able to reproduce our version of the algorithm.

Before this phase of the discussion period ends, we wanted to ask the reviewer whether we have addressed your concerns with our work?

We appreciate the feedback regarding our writing and structure, and the lack of clarity in our contributions, and believe we have now addressed these concerns

Hello reviewer LsDx, we only have 23 hours left to be able to respond to your questions and concerns. We believe we have addressed them thoroughly in our two comments, particularly by editing our paper for clarity. Could you please let us know as soon as possible if you have any additional concerns? Alternatively, would you be open to revising your initial judgment?

We thank the reviewer for their thoughtful and constructive feedback. Below, we address the concerns and questions raised:

Limited selection of models for experimentation

While a broad comparison of different SAE architectures and models is beyond the scope of this paper, we appreciate the importance of evaluating robustness across diverse settings. In response, we have added an analysis of the $L_1$ coefficient sweeps for optimal task vectors across three models (Phi-3, Gemma-1 2B, and Gemma-2 2B) and two SAE widths (16k and 65k) for Gemma-2 2B (Appendix D.1, Figures 9-12). These results demonstrate that our method consistently reduces the number of active task features by at least 50% (and up to 70%), while maintaining task vector performance. Additionally, our method almost always beats ITO (a fact that is somewhat obscured by all tasks being plotted on the same charts). This suggests that our method generalizes well across different configurations, though we acknowledge that more comprehensive studies could further substantiate this claim.

Exploration of varying SAE widths

We think that our findings are not greatly impacted by scoping to one SAE width only.

1. Our central findings are related to the interpretation of ICL in terms of SAE features. Using our 32k width SAEs on Gemma 1 2B, we were able to find interpretable decompositions of comparable quality to the task vectors (Figure 2) and find individual features impacting specific tasks (Figure 3). Therefore our work shows a strong lower bound for how useful SAEs are for explaining ICL, and is successful as a proof-of-concept for interpreting ICL with SAEs, which stands without needing to sweep widths.

2. We added positive steering experiments for Gemma-2 2B SAEs to Figure 19 in Appendix F.2. These experiments required a more complex setup compared to Gemma-1, as we could not identify an optimal steering scale and had to optimize it at the feature level. Despite this, we found that the task-specificity level of Gemma-2 SAEs was similar to that of Gemma-1 SAEs. One notable difference is that the wider 65k SAEs appear to have a larger amount of effective executors in the cleaned task vectors. However, this may be attributed to the low $L_1$ regularization coefficient.

Lack of detailed case studies

We agree that detailed case studies are important for understanding the behavior of task features. While an extensive exploration of case studies is beyond the primary scope of this paper, we have included some additional analyses to address this point. Specifically, we provide full unfiltered positive steering plots for Gemma-1 (Appendix F) and Gemma-2 2B SAEs (Appendix F.2), along with max-activating examples of prominent features (Appendix I). Additionally, we include a negative steering chart (Appendix F.1) to demonstrate the limitations of steering-based analysis and emphasize the need for advanced methods like Sparse Feature Circuits (SFC) for a deeper investigation of in-context learning. We hope these additions help to partially address the reviewer’s concerns.

Questions about specific features

1. Feature 11173: From its max-activating examples, we hypothesize that it represents a "provide information about location" feature, as it activates on phrases like <some name>, <city of residence>. Its effect, although very weak, on translation tasks might stem from the fact that geographical names often have non-English origins, and translation task features can be explained as the "meaning of this in another language" features. These connections remain speculative but highlight intriguing areas for further study.

2. Feature 1878: Algorithmic features, like this one, tend to encode less interpretable relationships between objects. This could explain their effects, although weak again, on other tasks, boosting logits of tokens related with the input one.

Future directions

One of our future research directions is studying task feature arithmetic. For example, we aim to investigate how translation task vectors encode target languages by leveraging features from different foreign languages. We also think it is an exciting future direction to conduct more experiments with algorithmic features, such as exploring function vector arithmetics as introduced in the function vectors paper. It is somewhat out of scope for our paper, which focuses on the task-detector to task-executor connection and our algorithms for SAEs.

Thanks for the reply. I revisited the paper and believe that the author should consider making some improvements to the writing in future versions.

Could you please tell us which sections you think require writing improvements?

Hello reviewer VLpn, we only have 23 hours left to be able to respond to your questions and concerns. We think that we've addressed them as detailed in our rebuttal (particularly by improving our writing), so could you let us know ASAP about further worries, or otherwise would you be open to revising your score?