On Mechanistic Circuits for Extractive Question-Answering

We thank the reviewer for their extensive review and also for finding our work encompassing strong mechanistic interpretability insights and strong experiments. Below we address the reviewer's comments:

"Scalability Concerns: circuit extraction for Llama-3-70B is mentioned but not deeply analysed (Appendix J). Moreover, computational cost of hierarchical patching could be very expensive for larger models.":

We thank the reviewer for raising this point. Indeed, the patching operation can be computationally expensive for larger models (e.g., ≥70B parameters). However, we emphasize that it is a one-time cost, essential for uncovering mechanistic insights. As part of the rebuttal, we conducted new experiments using the extracted attention head from LLaMA-3-70B on attribution tasks. The results are as follows:

HotPotQA

Self-attribution: 0.34 | Iterative Prompting: 0.36 | Sentence Similarity: 0.43 | AttnAttribute (Ours): 0.67

NaturalQuestions

Self-attribution: 0.73 | Iterative Prompting: 0.77 | Sentence Similarity: 0.81 | AttnAttribute (Ours): 0.91

These results demonstrate that circuit-derived insights can significantly enhance attribution quality, even in large-scale models. We will include these findings in the camera-ready version.

"Insufficient Circuit Analysis: there are two circuits analysed, but the paper does not explore interactions or edge cases where both memory and context circuits activate (although there are some details in App. D). Moreover, while ablations are persuasive, path-patching may still capture correlated rather than strictly causal components."/ (Q3): Could you provide some explanations on R2?:

The reviewer raises an important and thought-provoking point. In Appendix D, we presented evidence that the context-faithfulness and memory-faithfulness circuits are composed of distinct components, with no significant overlap. Further, in Appendix K, we show that even when the memory-faithfulness circuit is active, the context-faithfulness circuit continues to operate as expected. However, its influence is not reflected in the model's output, likely because each component contributes only weakly. We hypothesize that in extractive QA tasks, both circuits are activated, but only one takes precedence in writing information to the residual stream, thereby determining the final output. We will highlight these points better and in a more clearer way in the final camera-ready version of the paper.

(Q1): How do you ensure token-swapping perturbations (e.g., "Seattle" -> "New York") reflect natural model failures, rather than artificial edge cases? How sensitive are extracted circuits to the specific synthetic replacements in ? Could you release ablations with alternative corruption schemes or natural corpora?

To ensure that token-swapping perturbations reflect natural model failures, we measure the probability of the final answer token (e.g., the probability of "New York" in this case). After applying the perturbation, we retain only those examples where the model still assigns high probability to the (now incorrect) answer token. This indicates that the model is relying solely on contextual cues. Perturbations are selected from a broad pool of candidate replacements to satisfy this condition. In the memory-based setting, we ensure that the model assigns high probability to the correct answer stored in memory, while assigning low probability to the perturbed token. This setup ensures that the model's response is primarily influenced by memory rather than the current context. Moreover, our extensive ablations (e.g., Fig. 3) show that the circuits extracted with this probe dataset design is robust.

We thank the reviewer for finding our work insightful in understanding the inner mechanisms of extractive QA in language models. We are glad that the reviewer find that our experiments have a wide-coverage in terms of models and datasets. Below we answer the questions raised by the reviewer:

"The quality and representativeness of the circuits heavily rely on the designed probe datasets. Although carefully constructed, the generalizability to more diverse, noisy, or longer-context settings could be better discussed" :

The reviewer raises an important point. To obtain robust mechanistic insights, we used a controlled and well-designed probe dataset to ensure the extraction of "clean" circuits. This practice is standard in the mechanistic interpretability community and is followed by several influential works. To assess whether the extracted circuit generalizes to more diverse and noisy datasets, we ablate the circuit—originally extracted from the probe dataset—across multiple popular QA benchmarks, as shown in Fig. 3. Despite being derived from a controlled setup, the circuit remains valid across noisier datasets. To further test generalization to long-context QA scenarios, we designed an additional experiment during the rebuttal period. We constructed long-context inputs by merging documents from HotPotQA and evaluated the circuit’s impact when ablated across 200 examples. The results are as follows:

• Doc Length: 10 — Initial Accuracy: 74%, Ablated Circuit: 3%, Random Circuit: 71%

• Doc Length: 20 — Initial Accuracy: 69%, Ablated Circuit: 2%, Random Circuit: 68%

• Doc Length: 30 — Initial Accuracy: 67%, Ablated Circuit: 2.5%, Random Circuit: 66%

These results demonstrate that the extracted circuit continues to capture core mechanisms even in long-context settings. We will include this new experiment in the camera-ready version of the paper.

"While ATTNATTRIB is efficient at inference time, the initial circuit extraction (causal analysis, patching) appears computationally expensive." :

We agree with the reviewer’s point about the initial circuit extraction being expensive. However, we note that the initial circuit extraction is a one-time cost, which can be especially advantageous when attributions are served alongside answers to many end-users, as it significantly reduces inference-time costs at scale.

We thank the reviewer for the extensive review and acknowledging that our paper contributes to the mechanistic interpretability field of language models with various actionable insights about the extractive QA task.

Below we address the reviewer's comments:

What is the exact difference between the "a" and "m" nodes in Figure 1? : a refers to the attention layers whereas m refers to the MLP layers. We will add a legend corresponding to this in the final camera-ready version of the paper.

While you describe how questions and contexts are "corrupted" in Section 3.1, it remains unclear whether the same corruption method is used in Section 3.2 or other parts of the paper. Are there any differences in the corruption procedures across different experiments? : The corruptions are selected from a large pool of tokens, such that the probability of the answer from the "clean" model drops significantly. In terms of implementations, we implement an iterative function which scans over a set of tokens and selects the token such that the probability drop from the "clean" instance of the model is maximized. Across different experiments, this procedure is used. We will highlight this better in the final camera-ready version of the paper.

"Could you please elaborate on Algorithm 1, particularly the GetMaxSpan function? This function appears to be central to the algorithm, yet it is not mentioned elsewhere in the paper, leaving its implementation unclear." : The GetMaxSpan function utilizes the main attention head from the context-faithfulness circuit and extracts the sentence (amongst many in the context) which contains the token with the highest attention from that attention head. We have explained this in Line 276-277, but will make it clearer in the final camera-ready version based on the reviewer's suggestion.

"The paper relies too heavily on the appendix, which contributes to its lack of clarity and makes it harder to understand the main content." :The reviewer raises a valid concern. However, we note that mechanistic interpretability papers—particularly those focused on circuits—often involve highly detailed analyses that cannot be fully included in the main paper due to space constraints. Notably, prior works on circuits (e.g., 1, 2) also include extensive appendices for this reason. Similarly, our work relies on the Appendix to elaborate on technical details essential for full reproducibility. That said, we have made a concerted effort to present the key findings and takeaways clearly in the main paper.

" The manuscript would benefit from proofreading, as it contains typos. Additionally, the font size in many figures is extremely small, making them very difficult to read." : We will increase the resolution in the figures in the camera-ready so that they are better readable. Thanks for the suggestion!

"From what I understood, the nodes (either MLPs or attention heads) were patched using their activations on "corrupted" questions and contexts. Did you explore using alternative values for patching these nodes? Different patching values may lead to extracting different circuits (see Transformer Circuit Faithfulness Metrics are not Robust by Miller et al.)." : In our experiments, we applied different corruption instances prior to the patching operation and found the extracted circuits to be consistent. This indicates that, for extractive QA tasks, as long as the corrupted run yields a low probability for the correct answer (as compared to the clean instance), the resulting circuits remain similar. We will include this clarification in the camera-ready version. Additionally, we present extended ablations (e.g., Fig. 3) demonstrating that the circuits obtained through our patching procedure are robust—ablating them results in a substantial drop in extractive QA accuracy.

• Attention Layers: [27, 23, 31, 24, 25, 29, 21, 30]

• Attention Heads: [[27, 20], [23, 27], [31, 7], [17, 24], [25, 12], [31, 20], [24, 27], [27, 6], [26, 13], [16, 1], [31, 6], [29, 31], [31, 3], [30, 12]]

• Attention Layers: [27, 23, 31, 25, 24, 29, 21, 30]

• Attention Heads: [[27, 20], [23, 27], [31, 7], [17, 24], [25, 12], [31, 20], [24, 27], [27, 6], [26, 13], [16, 1], [31, 6], [29, 31], [30, 12], [31, 3]]