Scaling Sparse Feature Circuits For Studying In-Context Learning

Hello Reviewer fzas. Thank you for your impressively extensive and thoughtful feedback, and for your careful attention to details even in the Appendix.

On our "10-35x more parameters" claim: You note that "[1] has already performed circuit-style analysis [...] using models up to 70B parameters." We should have been more precise in our claim. By "comparable, circuits-style mechanistic interpretability research," we specifically meant work that provides end-to-end descriptions of task performance using SAEs. The Lieberum et al. paper focuses on attention heads' impact on final logits rather than providing a comprehensive explanation of MMLU processing. They state: "Our main focus in this work was on the final parts ... the rest of the circuit is still undiscovered." We will edit this into our manuscript.

Experimental design concerns:

On "lack of quantitative description of experiments": We moved many implementation details to the Appendix due to space constraints. We'll improve figure captions in our revision to better explain used metrics like the Average relative loss change you mentioned. The normalization process for the heatmaps (Fig. 6, 10, 11) follows these steps (Appendix F):

1. Calculate raw metric: metric[task, feature] = steered_loss[task, feature] / loss[task]
2. Clip: metric[task, feature] = clip(metric[task, feature], 1)
3. Normalize per task: metric[task, feature] = (metric[task, feature] - min_f(metric[task, f])) / (max_f(metric[task, f]) - min_f(metric[task, f]))
4. Threshold: metric[task, feature] = 0 if metric[task, feature] < 0.2

On token mass calculation: We use the same parameters as for TVC: a batch of 32 prompts each with 20 randomly sampled ICL examples.

On "missing results though reported": You cite that we mentioned "...we successfully applied ... to Gemma 2 2B and 9B models... it was also successful with the Phi-3 3B model." This statement refers specifically to the TVC algorithm's ability to reduce feature count while preserving task vector effects, not to the full suite of steering experiments. Our Appendix contains TVC results for all mentioned models (Figures 17-18), confirming this capability. Full steering experiments require more computational and human resources than just TVC. We decided to limit this type of analysis to Gemma 2B models, but expect that it will extrapolate to other models, judging by some of the max activating examples we analyzed.

On "mismatched results between heatmap and text description": You noted that "the feature that has the most significant effect on en_fr and en_it is eliminated through the TVC process according to Figure 22."

Feature 26987 is a generic "English to foreign language" translation feature. Since Spanish tokens are prevalent in model training data, the model handles English-to-Spanish translation well, while English-to-French/Italian ICL performance is weaker. Task vectors without cleaning perform poorly for these language pairs.

Task vectors contain multiple mixed-language features (26987, 26594, 6594), with 26987 having the cleanest translation examples. Since en_fr and en_it directions aren't naturally strong in the model, TVC reconstructs them using a different mix of features. In better-trained models and SAEs (Figure 25), these tasks have separate execution features. This shows that TVC finds a decent performing sparse combination of features in the absence of strong task-specific features.

Questions:

1. On task vectors being ood: Yes, we tried identifying common features across individual residual streams, but this highlighted many dense features alongside task-relevant ones. For Gemma-1 2B SAEs, individual residual streams activate ~40 features versus ~10 in averaged task vectors and 3-5 after cleaning.

2. On robustness to sample sizes: We briefly explored different batch sizes and shot counts. Larger samples didn't significantly improve the TVC results. Smaller shot counts risk missing task-specific features whose presence is weaker early in the prompt. Smaller batch sizes may lead to extracting batch-specific features, especially for tasks with weaker vectors. Strong task-executing features are extracted even from small batches.

3. On ablation: Our experiments showed no dramatic differences between zero and mean ablation. Zero ablation also doesn’t require us to store SAE activation statistics.

4. On the apparent contradiction in Section 4.1.3: Figure 12 shows only a small subset of the circuit focusing on core task-related features. The ~500 features needed for 0.6 faithfulness include generic pattern repetition features, task executing features split across multiple features and layers, and features encoding answers in later model layers.

Overall, we appreciate your feedback on the specificity of our claims, and will clearly state the takeaways from our research in our manuscript. We hope you will revise your score in light of this!

Hello Reviewer 1tPd, thank you for your review. We appreciate your time and feedback, but it appears there may be some fundamental misunderstandings about our paper's focus and contributions that we'd like to clarify. You stated: “1. What is the main contribution of this paper? The paper attempts to introduce a new TVC” but this is not the main contribution of our paper. Instead, the main contribution of the paper is the use of SAEs to understand the circuitry behind in-context learning (ICL), as is clear from the title of our paper. This is important because our contributions are considerably more exciting for mechanistic interpretability researchers. TVC is a tool we developed to help identify which existing SAE features are most relevant to ICL tasks. Critically, TVC does not retrain or modify the pretrained SAE in any way - it only identifies which features in the already-trained SAE are causally relevant to ICL. Therefore, the comparison to AxBench (a paper that was released two days before the final submission deadline for ICML 2025 and therefore does not need to be addressed by us since it is contemporary work) is much less relevant to our work because our research is principally about circuits rather than steering.

On the motivation for using SAE to analyze task vectors. Our motivation is to understand the circuit mechanisms behind in-context learning. While previous works like Todd et al. (2024) and Hendel et al. (2023) discovered task vectors, they didn't explain how these vectors are mechanistically implemented in the model. SAEs help us decompose these task vectors into interpretable features that form part of the model's computation.

On separating detection and execution features: This separation was an empirical discovery, not an assumption, supported by multiple lines of evidence:

Different activation patterns: execution features activate on arrow tokens (89.8%), detection features on output tokens (96.76%) - Tables 1 and 2

Layer positioning: detection features work best in earlier layers (11), execution features in later layers (12)

Causal structure: ablating detection features reduces execution feature activation (Figure 11)

Task-specific steering effects: both feature types affect different tasks differently (Figures 6 and 10)

Our answer to reviewer zx2e also contains a more thorough definition of both feature types.

On results with more models and sparsity levels: Our Appendix includes experiments across multiple models (Gemma 1 2B, Gemma 2 2B, Gemma 2 9B, Phi-3 3B) and SAE configurations. Figures 16-19 show TVC performance across these settings, demonstrating that our approach generalizes well. These figures include different sparsities, widths and models, including Gemma Scope SAEs.

Hello reviewer zx2e – thank you for your thoughtful review. We appreciate your recognition that our methods are "solid" and that task vector cleaning is "a simple approach that appears to work well to get strong SAE decompositions."

Regarding feature definitions: we consider task-specific causal influence to be the defining characteristic of both detector and executor features, and apologize for not making this clearer in the main text. We will update the manuscript based on this feedback. To expand specifically on what we mean by task-detection and task-execution features:

Task-detection features are defined as intermediate circuit components between the raw prompt inputs and the executor features. They function as a specialized probing mechanism that identifies and encodes which specific task is being performed. These features primarily activate on output tokens (96.76% as shown in Table 2), consistent with their role in monitoring completed examples to determine the pattern being demonstrated. Importantly, our steering experiments in Section 4.2 demonstrate that these detector features also have direct causal effects on task performance when steered on blank output tokens.

Task-execution features are defined as features that directly impact task completion. They affect whether the specific task happens without requiring additional intermediate processing. Their strong activation on arrow tokens (89.8% as shown in Table 1) demonstrates their positioning at precisely the point where the model must apply the identified transformation.

For greater specificity, we classify these features using clear criteria:

• Detection features: (1) consistent activation on output tokens in examples, (2) minimal activation during generation, (3) causal influence on executor features (quantified in Figure 11), and (4) direct causal effect on task performance when steered on output tokens
• Execution features: (1) peak activation on arrow tokens immediately preceding task completion, (2) strong causal effect on task performance when steered with (Figure 6), and (3) task-specific activation patterns on raw data.

Regarding your suggestion about "constructing task vectors on new tasks" - if you're referring to zero-shot generalization, our steering experiments in Section 3.2 and Figure 6 demonstrate the causal effect of our identified features on model behavior for the tasks studied. While running additional experiments on entirely new tasks is beyond our current rebuttal time frame, we believe the consistent patterns across our current task set suggest potential for generalization.

The circuit analysis in Section 4.2 and Figure 11 provides empirical validation of our proposed abstraction by showing that: (1) ablating detection features reduces execution feature activation, (2) this relationship is consistent across tasks. These quantitative measures establish that detection and execution features form distinct functional components in the ICL circuit.

We appreciate your feedback and will incorporate these clarifications to strengthen the paper.

Hello reviewer K3Cy,

Thank you for your thoughtful and supportive review of our paper. We appreciate your detailed feedback and are grateful for your recommendation to accept the paper.

Regarding your specific question about the weak detection-execution connections for the person_profession and present_simple_gerund tasks:

For person_profession, our analysis revealed the model's general performance of this task is relatively poor. As seen in Figure 27f, the strongest detector feature is a single token feature focused on journalism, indicating that SAE features did not pick up strong task-related directions alongside poor model's accuracy. This both contributes to weak connection of executing and detecting features.

For present_simple_gerund, the situation is more nuanced. One possible explanation is feature splitting. Since gerund forms are common in the training data, they have several corresponding detector and executor features. At the same time, executor and detector features with the strongest steering effect may correspond to different cases of gerund occurrences, and thus be weakly connected.

Thank you again for your positive feedback on our work!