Thank you for your valuable feedback, we are glad that you appreciate our methodology and mechanistic insights on the large and small models. Below are our responses to your comments and questions.

The propositional logic problem studied is minimal…

While extending to longer problems is beyond the scope of this paper due to our constrained computational resources (longer context lengths coupled with the large models is too demanding for our computing resources), we did conduct two new experiments to understand how much the model reuses the identified subcircuits, under variations of our current setting.

New experiment 1 - Prompt format variation. We conducted the following new experiment for testing the robustness of the circuit across prompt formats. We swapped the fact and rule sections in the prompt area. Now all the logic problems are presented in the form “Facts: … Rules: … Question:...”, for example:

Facts: O is false. Q is true. I is true. Rules: I implies C. Q or O implies H. Question: what is the truth value of H? Answer: Q is true. Q or O implies H; H is true.

We found evidence that on Gemma-2-9B, the circuit for predicting the first answer token stays almost exactly the same! In particular, we find that it is the same heads exhibiting high IE scores in the activation patching sweep, with slightly different scores. Due to the time limit, we have not performed the sufficiency test (as described in Section 3.2 and Appendix B.5, B.8.3) thoroughly yet, so we cannot report DE measurements here, but we think even the necessity-based evidence is quite interesting to present here!

New experiment 2 - Rule-invocation subcircuit reuse. We conduct a further analysis on the second most important place in the model’s generated proof, the first token for invoking the rule – recall that the model needs to invoke the correct fact, then the correct rule to infer the correct truth value of the queried proposition variable. We found that in both Gemma-2-9B and Mistral-7B, the queried-rule locators and decision heads again exhibit strong indirect effects at this proof position, indicating reuse of functional sub-circuits in their proof writing.

We will add the more detailed and rigorous version of these results to our updated paper.

“The contrast with 3-layer task-trained models mentions ‘intermingled reasoning’ but lacks detailed mechanistic comparison…”

Thank you for this excellent point. Due to the page limit we had to delay our mechanistic analysis of the 3-layer 3-head transformer to Appendix D (with its backbone algorithm summarized in Figure 29) – we do see some major differences between how the small models and the LLMs solve the simple propositional problems. We will add further comparisons to our updated paper. For now, we highlight some interesting differences below.

The most pronounced difference is the less interpretable and modular nature of the attention heads (especially their attention patterns) in the small transformer. Since the problem has many token positions which are “redundant” (e.g. “is”, the periods, spaces, etc.), the small transformer’s attention heads tend to make use of these positions in a much less predictable way than the large models.

For example, while we show, through a set of probing and ablation experiments, that when the logical-operator chain is queried, it is only at the final layer of the attention heads that the model arrives at the right token to write down – meaning that they are effectively “decision heads” – their attention pattern statistics are fairly messy, scattered across token positions in the Rules section. This stands in great contrast to the modularity of the attention head families and the interpretability of their attention patterns found in the LLMs. A cautionary remark should be emphasized, however: such modularity is not necessarily observable in all LLMs. For instance, smaller and older LLMs such as Gemma-2-2B and the GPT2 models have very low proof accuracies, they might not even possess “reasoning circuits” as we see in the Gemma-2-($\ge$9B) and Mistral-7B models.

Recommended citation: LogiCoT: Logical Chain-of-Thought Instruction-Tuning

Thank you for this reference, we will add it to our updated paper.

Discussion on future directions. We sincerely appreciate your comments on the possible ways of extending the current analysis. Increasing the complexity of the propositional problems by increasing its length and involving nested operators is indeed a feasible direction. However, even without increasing the problems’ lengths (with heavier compute demands), we believe that there are meaningful directions to pursue.

First, the negation operator is an interesting one to study: its analysis might indicate whether certain “polarized” behaviors arise in the model, and whether the model reuses certain causal pathways when it needs to latently negate a (simple) clause.

First-order logic problems are another possible extension direction of problem setting. We are especially interested in how transformers latently bind quantifiers to variables, and how such binding affects the way they reason about sets – a foundational problem in how LLMs reason.

Thank you for your valuable feedback! We are glad that you find the paper’s writing clear, and appreciate the mechanistic insights on the large and small models. We will address your comments below.

“Given the authors find some evidence that Gemma-2-27b carries out this task somewhat differently than Gemma-2-9b, it would have been nice to see Gemma-2-2b studied as well (to isolate the effect of scale vs. architecture).”, “...did you consider studying a smaller model (e.g. 1b or 2b)?”

We wish to emphasize that the main goal of this paper is to study the latent processes behind how sufficiently capable LLMs write simple but valid (propositional logic) proofs. We found that small LLMs (Gemma-2-2b, the GPT-2 models, etc.)’s proof accuracies are overly low even with a large number of in-context proof examples, showing that they do not appear to recognize the latent reasoning patterns. This is the primary reason for the study of larger LLMs in our paper, compared to the more standard sizes studied in many existing circuit analysis papers (mostly under 1B in size) [1-3].

Focusing uniquely on Gemma-2-9b in the main text makes the narrative clear but makes it harder to get a sense of the generalizability of these findings. Focusing on one model in the text but still including the main numerical results for all models (e.g. Table 2 for Mistral-7b) in the main text would have made for a clearer paper.

Thank you for this point, we will add Table 2 to section 3.2 in the main text, and highlight that we were also able to verify a functionally similar circuit in Mistral-7B.

Can you elaborate a bit more on the significance of the decision head? It appears that it is attending sharply to the atom for which a known fact can be used in one of the rules to derive the conclusion. Calling this a "decision" seems a bit far fetched - it almost seems more like "fact locating" to me?

There are two reasons why we group the decision heads instead of merging them into the family of fact-processing heads.

First, we separate the two families based on their distinct attention patterns. We noted in Appendix B.4.2 and B.8.2 that the fact-processing heads’ attention weights usually do not directly focus on the answer token itself, but are typically spread over the sentence containing the correct fact. In contrast, the decision heads always direct significant attention weight at the precise answer token, for all the LLMs we analyzed. Second, we found that the decision heads tend to have direct effects on the model logits in Gemma-2-9B (the main model analyzed in this paper), as we show in Figure 25 in the Appendix, while the fact-processing heads do not.

We will surface succinct versions of the distinction between the two attention-head families to the main text in the updated version of our paper.

The influence of the decision head is not clear to me, which I believe is because of the focus on the model's state right before outputting the first token. However, I recognize that scaling up this analysis to arbitrary positions in the output sequence would be infeasible.

We found that for rule invocation in the proof, the decision heads again exhibit high importance in the Gemma-2-9B and Mistral-7B’s inference pathway.

More specifically, we performed another set of patching experiments measuring the causal importance of the model components for writing down the first token of the invoked rule – the second most important place in the proof, after the fact-invocation token. We found that the same decision heads and rule-locator heads exhibit strong causal scores (measured via logit difference) in both Gemma-2-9B and Mistral-7B, with similar attention patterns to fact invocation! This suggests that the LLMs reuse functional sub-circuits for writing down the proof. We will add detailed versions of these results to our updated paper.

The results figures (in particular Figure 5 and similar figures in the appendix) could use some polish. Furthermore, the legends do not clarify what the red bars are (95% CI on the mean? +- one standard deviation?).

Thank you for this point, we will make sure to add clarity to the figures in our updated paper.

References.

1. K. Wang et al. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. ICLR 2023.
2. G. Kim et al. A Mechanistic Interpretation of Syllogistic Reasoning in Auto-Regressive Language Models. ArXiv Preprint, 2025.
3. Y. Yao et al. Knowledge Circuits in Pretrained Transformers. NeurIPS 2024.

Thank you for your thoughtful and constructive feedback! Below are our responses to your questions and suggestions.

The analysis for the logit difference by CMA only involves the first answer token. Although predicting the first token is non-trivial and is important for generating the whole output sequence, analyzing the first token only might not be sufficient to understand the underlying mechanism of how LLMs perform propositional logical reasoning.

Thank you for this point. Indeed, it would be nice to study how the models write down all output tokens in the proof, but it is computationally infeasible across the large models we study given our resources.

Nevertheless, we conduct a further analysis on the second most important place in the model’s generated proof, the first token for invoking the correct rule – recall that the model needs to invoke the correct fact, then the correct rule to infer the correct truth value of the queried proposition variable. We found that in both Gemma-2-9B and Mistral-7B, the queried-rule locators and decision heads again exhibit strong indirect effects at this proof position, indicating reuse of functional sub-circuits in their proof writing.

We will add detailed versions of these results to our updated paper.

The problem studied in this paper is somewhat restricted, and the format is not that flexible. To understand how LLMs perform logical reasoning in a more general setting and which heads play the important role, it would be important to analyze the scenarios when the input format changes while keeping the set of facts and rules unaltered (e.g., exchanging the order of the rule area and the fact area, or rules and facts are interleaved)...

Thank you for this nice suggestion!

We conducted the following new experiment for testing the robustness of the circuit across prompt formats. As you suggest, we swapped the fact and rule sections in the prompts. Now all the logic problems are presented in the form “Facts: … Rules: … Question:...”. For example:

Facts: O is false. Q is true. I is true. Rules: I implies C. Q or O implies H. Question: what is the truth value of H? Answer: Q is true. Q or O implies H; H is true.

We found evidence that on Gemma-2-9B, the circuit for predicting the first answer token stays almost exactly the same! In particular, we find that it is the same heads exhibiting high IE scores in the activation patching sweep. Due to the time limit, we have not yet performed the sufficiency test (as described in Section 3.2 and Appendix B.5, B.8.3) thoroughly, so we cannot report DE measurements here, but we think the necessity-based evidence is already quite interesting to present here! We will add the more detailed and rigorous version of these results to our updated paper.

The input contains in-context examples. However, it is unclear how the in-context example will affect the underlying mechanism. For example, when there is no in-context example or when the number of in-context examples varies, will the circuit change?

Thank you for the comment – it is a good question as to if, and how much the circuit changes with respect to the number of in-context examples. It is, however, outside the scope of this paper.

The primary goal of this paper is not about understanding circuit formation in in-context learning, but rather how transformers output tokens that describe simple but valid mathematical proofs: this requires us to work in a regime where the models are sufficiently high in accuracy, in which case the logit difference metric becomes a valid metric for measuring causal effects of model components. If the model is low in accuracy, then it cannot separate the two most likely candidate answer tokens well (the two which we use as normal and counterfactual answers by construction, in our QUERY-flipping experiments). Basically, we need to eliminate possible confounders introduced by having overly limited in-context proof examples, which would make the model too uncertain about the proof structure.

We sincerely thank you for the thoughtful questions and suggestions! We would be happy to continue the discussion on any of our responses below.

CMA experiments, methodology and interpretation.

Before delving into the technical details, we wish to emphasize that the use of activation patching for localizing and interpreting hidden mechanisms in LLMs is popular in mechanistic interpretability, and has been explicitly cast as instances of Pearl’s Causal Mediation Analysis (CMA) in seminal works [1-3]. In particular, it is typically acknowledged that activation patching offers imperfect (and potentially biased) estimates for the indirect and direct effects, due to the LLM’s complexity and limits in computational resources.

With this overall consideration of CMA in interpretability research, we address your concerns below.

the causal graph assumed (X → M → Y) does not reflect the dense interdependencies in a transformer

We employ the simplified causal graph $X \to M \to Y$ as a local abstraction of the LLM: in our circuit analysis, we analyze one hypothesized mediator at a time, treating the rest of the network as background, essentially approximating the indirect effects of the mediator. This approach is common in mechanistic interpretability.

identifiability assumptions (e.g., no hidden confounders, functional invariance) are likely invalid.

While the listed identifiability assumptions are largely satisfied by activation patching, some subtleties exist.

First, please note that we sidestep many identifiability conditions in classical mediation analysis since we can observe the model’s “counterfactual” outcomes. By design of activation patching, no exposure‑induced mediator‑outcome confounder exists. To perform activation patching, we run the model on the normal prompt, and hold the mediator’s output to its corresponding counterfactual activations, that is, we perform a hard intervention $do(M = m_{counterfactual})$. For any potential $C$ in the LLM with $X \to C \to Y$ and $X \to C \to M$, the intervention cuts the link $C \to M$, removing such confounders. Exposure-outcome and exposure-mediator confounding are also ruled out – the LLM’s forward pass is fully deterministic and “self-contained”, as both $Y$ and $M$ are only functions of $X$. Functional invariance can be shown similarly.

The positivity assumption is subtler. Earlier approaches in mechanistic interpretability corrupt a mediator in a “messy” fashion, such as noise injection [4]. In comparison, our counterfactual activations are generated with much care to minimize “off-manifold” effects, as we use activation from altered prompts which differ minimally from the normal one. We emphasize, however, that a balance has to be struck between efficiency of reaching reasonable conclusions and how fine-grained the differences are.

unadjusted upstream activations may confound the estimation of indirect effects…

The upstream activations of the mediator, including those in the lower layers and previous token positions, and parallel activations (e.g. from the attention heads in the same layer, if the mediator is an attention head) are identical with or without intervention. Only the mediator’s output is intervened.

“large indirect effect (IE) does not imply necessity… large direct effect (DE) does not prove sufficiency…”, “...compensatory effects”

To demonstrate “necessity” of (a family of) attention heads in the circuit, we show

1. large IE in the single-component patching sweeps, and
2. when we ablate all non-circuit heads, further ablating any one of the attention families from the circuit causes performance-recovery to drop from >90% to nearly trivial.

This shows that the identified attention head families participate heavily in the LLM’s inference pathway. This is a (stronger) “necessity” result following [2]’s definition.

We believe that the rest of the comment is less about “sufficiency”, but more about the circuit’s “faithfulness” and “completeness” – more advanced criteria introduced in [5]. As shown in Section 3.2 and B.5, the discovered circuit is nearly sufficient in recovering the model’s performance (94% for Gemma-2-9B, 98% for Mistral-7B) – our circuit is faithful. We note that even the faithfulness criterion is not tested for or met in many existing mechanistic interpretability works [6,7].

Now, our circuit is not necessarily complete. Backup heads, albeit typically weak and noisy, have been observed in some LLMs, and is considered an open problem in mechanistic interpretability. Please note that the completeness criterion is rarely tested for in many mechanistic interpretability works, due to its heavy computational demand of having to iteratively ablate the discovered causal mechanisms to track compensating pathways – an aspect which is particularly pronounced in our work due to the large model sizes. We will add a discussion on these more advanced criteria to our updated paper.

“Prompt perturbations… modify multiple features (position encodings, token distance)...”

We first clarify that the sets of patching experiments are independent of each other. For example, when we flip the QUERY token, we do not shuffle the rules: only the QUERY value is changed, no position encoding or other features vary in the prompt.

We treat the rule-shuffling and fact-changing patching experiments as cross-checks for our hypotheses of the sub-circuits in the LLMs. For example, the rule-locators, whose attention weights consistently track the queried rule, have high causal scores in both the QUERY-flipping and rule-shuffling experiments, but trivial scores in the fact-changing experiments. While the correlational and causal evidence do not fully eliminate nuanced factors such as token distances, the proposed circuit is still the more plausible explanation of existing evidence.

In Table 1, how should we interpret negative logit differences?

The scores in Table 1 are normalized. W.r.t. the logit-difference metric, a score of -1.0 shows that the model behaves as if it is solving the counterfactual problem, even though it is run on the normal prompt. It is attained with the null circuit, i.e. freezing all attention head activations to their counterfactual activations. A score of 1.0 is the other extreme: the model behaves as if it is naturally run on the normal prompts. 0.0 is the mid-point.

Writing and conceptual clarity.

Clarify “MLP references…”, “interventions…”, “figure 3…”, “fig 4(b)…”, “table 1…”

Thank you for these suggestions, we will address them in the updated version of our paper!

“definition of ‘causal chain’ …”, “…asserted without sufficient justification”

Thank you for raising this point about the QUERY → Decision chain. We intended it to be the (human-interpretable) hypothesis of how the LLM implements its internal reasoning. We will adjust our phrasing in the updated version of our paper to ensure that this point is clear.

“...rationale behind exactly four types of attention heads remains ambiguous”, “ method for identifying the four attention head types is underspecified”

We primarily distinguish the attention-head families based on their attention patterns. While we do provide detailed explanations of their characteristics in Appendix B.4.2, B.8.2 and B.9.1, we will aim to surface succinct versions of these analyses into the main text for better clarity in our updated paper.

… four attention head types… modularization differ under other prompts or tasks?

We believe the exact types of attention heads for different tasks can change. However, functionality-wise, there have been observations about the usage of locator and mover heads in other mechanistic works, in GPT2 models [5]. An interesting future direction is to study what functional components persist across problems and models, and what emerges with model scale.

“How statistically robust … decision heads…”,

We show their (robust) attention statistics in Fig. 5(b)(iv).

“moving QUERY to ‘:’... (23,3)”

Due to grouped query attention, head (23,6) has the value index (23,3). We primarily classify the mover heads based on their QUERY-invariant attention patterns, trivial query and key casual scores, and high value causal scores.

Pseudocode. We will add the pseudocode for our circuit analysis algorithm to our updated paper. The following is its backbone, in consideration of character limits.

A or B implies C. D implies E. A is true. B is false. D is true. What is the truth value of C?

X  = prompt with Query = C

X* = prompt with Query = E   (counterfactual)

(1) run model(X)    → logits_norm

(2) run model(X*)   → logits_ctfl

(3) run model(X) with head h set to activation from X* → logits_interv

(4) Estimate indirect effects (expression (1) in the Appendix)

We currently manually set the threshold for top-k components to form the circuit (threshold setting is a major open problem in mechanistic interpretability). For more complex propositional problems, we can vary the QUERY token to choose which sub-part of the reasoning chain we wish to probe for.

References.

1. J. Vig et al. Investigating Gender Bias in Language Models Using Causal Mediation Analysis. NeurIPS 2020.
2. F. Zhang et al. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. ICLR 2024.
3. A. Geiger et al. Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability. JMLR 2025.
4. K. Meng et al. Locating and Editing Factual Associations in GPT. NeurIPS 2022.
5. K. Wang et al. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. ICLR 2023.
6. A. Stolfo et al. A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis. EMNLP 2023.
7. B. Wang et al. Grokking of Implicit Reasoning in Transformers: A Mechanistic Journey to the Edge of Generalization. NeurIPS 2024.

the authors say the intervention cuts the $C \rightarrow M$ causal link, but we still compute M from corrupted X. How does M get formed if that link’s severed?

We were explaining how the hard intervention $do(M = m_{counterfactual})$ eliminates (exposure-induced) mediator-outcome confounding. For the "counterfactual version" of $M$'s output, it is still computed naturally on the counterfactual prompt $X^*$.

wonder if this is a standard way of doing activation patching

Our activation patching implementation precisely follows the standard algorithm, as described in [1-3].

Please consider distilling it as a reusable recipe (for other tasks) since it is not open-sourced

We will make sure to add the more complete version of the pseudocode to our paper's appendix.

We plan to open-source our code upon paper acceptance, including a main demo IPython notebook. Our code is built on top of the TransformerLens library (a well-known library in mechanistic interpretability), so it should be easily accessible to other researchers.

I’m not sure how well these findings carry over to, for example, mathematical reasoning.

It is an important future direction to extend our work to more general mathematical reasoning settings, starting with simple, but focused topics such as how LLMs process negation and quantifiers (towards first-order logic problems). They are, however, beyond the scope of this work.

References.

1. F. Zhang et al. Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. ICLR 2024.
2. S. Heimersheim et al. How to use and interpret activation patching. ArXiv Preprint, 2024.
3. A. Geiger et al. Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability. JMLR 2025.
4. K. Wang et al. Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. ICLR 2023.
5. Y. Yang et al. Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Models. ICML 2025.
6. G. Kim et al. A Mechanistic Interpretation of Syllogistic Reasoning in Auto-Regressive Language Models. ArXiv Preprint, 2025.
7. X. Zhao et al. Understanding Synthetic Context Extension via Retrieval Heads. ArXiv Preprint, 2025.

Thank you for your reply, we really appreciate your careful examination of our methodology and the imperfections of activation patching! Below are our responses to your comments.

"When the authors shuffle the rules, token positions will also be changed, which can tweak attention scores. It’d be good to mention this token-distance effect as a limitation", "when we intervene on the mediator via prompt perturbations, the upstream activations for the perturbed prompt will differ from the upstream activations for the original prompt... "

Thank you for emphasizing these points.

In general, this concern goes back to the typically acknowledged imperfection, or caution of using activation patching, which is the imperfect isolation of causal features due to multiple differing features in the altered/counterfactual prompt from the normal prompt, as discussed in [1] and [2]. While our main QUERY-flipping experiments minimize this issue (which our rebuttal's discussions centered around, regarding the identifiability assumptions), the rule-location shuffling experiment indeed introduces token distance as a possible confounding factor, distinct from the focal factor of the experiment, the queried-rule location. More technically speaking, token distance is possibly an exposure-mediator confounder for the rule-shuffling experiment.

Nevertheless, we think it is well-justified to prioritize compute on the rule-shuffling experiments over finer-grained ones for cross-checking the rule locators, by noting that we are already keeping all the major non-focal factors fixed across the normal-counterfactual pairs, which includes QUERY value and position, fact values and positions, proposition variables' choices and relations, choice of logical operator, etc. Moreover, since our problem setting relies on prompts which are short, we do not expect token/clause distance to influence the qualitative conclusions on the rule-locator heads in our setting – i.e. they only surface in the QUERY-flipping and rule-shuffling experiments, but get trivial scores in the fact-changing experiments.

However, it is possible that when the problem becomes very long (e.g. large number of clauses), token/clause distance becomes a factor of importance. Therefore, we will flag this factor in our final manuscript for cautioning the design of the cross-checking experiments in more general settings.

I get that following every token might be too heavy (as Reviewer 9qMC noted), but please flag this as a limitation.

We will flag this limitation in our final manuscript.

Just to ensure that we are on the same page, we did conduct two more experiments to increase the breadth of the experiments: one experiment finding that for writing down the first token of the invoked rule, the rule-locator and decision heads in Gemma-2-9B and Mistral-7B again exhibit strong IE scores; one experiment finding that if we swap the Rules and Facts sections in the prompts (i.e. present Facts before Rules, instead of the original Rules before Facts), the surfaced heads overlap significantly with the circuit found before. Please refer to our responses to Reviewers rGZP and 9qMC for further details (to avoid making this response too cluttered).

the concern about the rationale behind exactly four types of attention heads remains… Is that something that naturally emerged, or did the prompt design force it?

We primarily assign the attention head families based on their distinct attention patterns with detailed discussions in Appendix B.4.2, B.8.2 and B.9.1 (with finer-grained validations from the cross-checking CMA experiments). We are not making any assumption regarding what family of heads must emerge, we are just classifying them based on our experimental observations.

However, in general, we understand your concern regarding how correlated the full circuit is with the prompt distribution we test on, even after we add the results in our previous point. This is a typical and valid concern with circuit-analysis on LLMs: the heavy demand for compute and labor limits analysis to narrow prompt distributions (to be fair, our work already takes a solid step towards more realistic reasoning problem structures compared to prior works [4-6]).

From our perspective, the full circuit in this work likely does not generalize perfectly to substantially different problems – it is possible that finer classes of heads emerge on more complex logic problems. However, we do see early evidence of the generalizability of the functional sub-circuits found in our work: [4-6] observed locator and mover heads on certain simple language problems, [7] found certain “insight” heads which place attention on intermediate steps towards an answer (somewhat similar to our fact-processing heads) in some RAG scenarios, etc.

(Continued in the next part)