We thank the reviewer for the thoughtful feedback and for highlighting the strengths of our work. In the following, we address the reviewer’s questions and concerns by grouping related points.

[W1] Dataset Description:

We thank the reviewer for the suggestion. We acknowledge that the main text currently provides only a brief description due to space limitations, and many details were placed in the appendix. In the revised version, we will:

1. Move key details (e.g., ID/OOD ratios) from the appendix into the main text.
2. Add illustrative examples to help readers better understand how the symbolic dataset mimics realistic multi-hop reasoning scenarios.

We believe these changes will improve clarity.

[Q1] Cross-Query Semantic Patching vs. Standard Causal Patching

We thank the reviewer for the question regarding the originality of our Cross-Query Semantic Patching (CQSP) method. We clarify the key differences below:

1. Same general family, but different purpose.

Our method is indeed inspired by the general causal patching paradigm, but CQSP explicitly targets semantic reusability of intermediate entity representations across queries, which is essential for studying multi-hop reasoning.

Standard causal patching primarily asks “does this activation (in target query itself) affect the target output?”, whereas our method asks “does the representation from a source query carry reusable intermediate-entity information that can correctly drive reasoning in a target query?”.

2. Difference in how source activations are treated.

In standard causal patching, source activations are often treated as perturbations (sometimes even randomized) to measure causal effect on the target prompt.

In contrast, CQSP leverages the source prompt representation as a meaningful semantic candidate, inserted into the target prompt’s context, and we evaluate whether it produces the correct 2-hop answer (see line 191).

3. Stricter criterion

Standard causal patching does not have a well-defined “success” output—it can only evaluate the magnitude of output probability shifts at the target prompt to estimate causal influence at different positions.

In contrast, CQSP has a known expected answer (see line 191), only producing this correct answer counts as success. This makes our metric goal-directed and semantically meaningful.

4. Concrete example vs. standard patching.

In our extended setup of Wang et al. [24], standard causal patching was applied to locate circuits encoding intermediate entities. Their approach compared Phase I and Phase II causal graphs and subtracted them, which produced noisy results (see Figure 9 in their paper).

By contrast, our CQSP method achieves nearly 100% success (Figure 4) in pinpointing the positions that encode the correct intermediate entities, providing a far cleaner and functionally interpretable signal.

In short, CQSP is a cross-query, semantically driven extension of causal patching. We hope this clarification addresses the reviewer's concern about our methodological novelty and supports an improved originality assessment.

[W3] Narrow Cone vs. Entity-Specific Clustering

We thank the reviewers for raising the concern that the observed OOD–ID cosine alignment might be a byproduct of the global “narrow cone” phenomenon (e.g., 2401.12143), where all representations become more similar during training.

Our key distinction is that the alignment we report is entity-specific and functionally predictive:

1. We track hidden states of specific bridge entities, not global embeddings.
2. Their clustering dynamics are tightly linked to functional behavior: ID cohesion rises with Test-II accuracy (Phase II), and OOD alignment rises with Test-OI accuracy (Phase III).
3. To further rule out global collapse, we computed the average pairwise cosine similarity among different intermediate entities over Phases I–III:

As shown, cross-entity similarity consistently decreases, confirming that the clustering is highly entity-specific rather than a byproduct of general compression.

This result directly refutes the “all representations become similar” hypothesis and strengthens our original claim: functional multihop reasoning emerges when bridge-entity representations form tight, entity-specific clusters, not from global embedding collapse.

We will incorporate the findings into the revised version of the paper, and we are grateful to the reviewer for the suggestion that inspired this analysis.

[Q2] Full-Run Probing as Stricter Complement

We appreciate the reviewer’s question regarding why we expect the logit lens to extract the original entity after passing the patched representation through all model layers. Our reasoning is as follows:

1. Empirical observation of early-layer decodability. When we perform cross-query patching at the r1 token position, we observe that:

• Patching at very early layers (1–2) already leads to the final layer decoding the correct intermediate entity with high success.
• In contrast, direct immediate decoding of hidden states (without full-run propagation) only becomes stable from layers 4–5 onward.

2. Role of later layers. This observation indicates that the decodability of the intermediate entity is already established in early layers, and the downstream layers refine and stabilize the decodability.

We therefore introduce full-run probing as a stricter complement to immediate decoding, ensuring that representations which cannot be immediately decoded, but already contain sufficient information for correct decoding, are not overlooked.

[W2, Q3] Contribution Framing and Highlighting Key Findings

1. Adjusting the framing of our analysis methods

We appreciate the reviewer’s comment regarding the novelty of our analysis tools.

Our intention is not to claim standalone methodological novelty. These tools were designed primarily to challenge the prevailing decodability-based assumption in multi-hop reasoning analysis, which assumes that intermediate entities must be explicitly decodable (Section 3.2), and to enable the mechanism discovery that forms the core contribution of our paper.

We will revise the final version to clearly present these tools as supporting analysis, with the core contribution framed as the mechanism discovery.

2. Highlighting the key OOD-ID alignment finding

We thank the reviewer for emphasizing that the alignment of OOD and ID entity representations is a particularly interesting and impactful finding, which another reviewer also noted as a central and impactful contribution.

We fully agree and will revise the abstract and contribution statements to make this finding explicit.

We will clarify this motivation for the full-run probing setup in the revised version.

Thank you for the responses! The author responses address my questions. I am maintaining my currently positive score -- I would also increase my originality rating to 3, but that does not seem to be an option in the review revision. I left a note on the review.

Update: was able to change the originality rating.

We thank the reviewer for the thoughtful feedback and for highlighting the strengths of our work. In the following, we address the reviewer’s questions and concerns by grouping related points.

[W1] Paper Structure

We understand the reviewer’s concern about the organization of the paper, and we clarify our reasoning below.

1. Importance of Section 2.4

We would like to clarify the role of Section 2.4 in our study. Section 2.3 and 2.4 jointly answer the core question of what data signals are truly required for implicit reasoning:

• Section 2.3 demonstrates that ID atomic triples are not required for ID generalization, challenging the prevailing assumption in prior works.
• Section 2.4 goes one step further, identifying what the model actually depends on at the query level, revealing that second-hop generalization requires query-level exposure. This extends the understanding of data dependency from a macro level (type of supervision) to a fine-grained query level, which is essential to answering how LLMs learn.

Therefore, Sec. 2.4 is not redundant but a key part of the data-dependency analysis.

2. Multi-faceted Structure is Intentional

Our study is organized to systematically answer “how LLMs learn implicit reasoning” from multiple complementary perspectives:

1. the role of data (Sec. 2.3–2.4),
2. training dynamics and observed phenomena (Sec. 2.2–2.4),
3. analysis tools (Sec. 3), and
4. internal mechanisms (Sec. 4).

This design is deliberate: phenomena alone cannot explain how learning occurs, and mechanistic insights require the context of observed phenomena.

We also note that different reviewers valued different aspects:

• Both reviewers mwv1 and 4Y73 found the new phenomena observed in Section 2 interesting and informative.
• Reviewer 1ywD highlighted that “The interpretability portion is a highlight.”; Reviewer mwv1 also praised our mechanistic analyses.
• Reviewer 4Y73 expressed interest in the patching tools.

Together, this indicates that the current structure naturally supports a comprehensive answer to the central question.

3. Open to Further Discussion on Presentation

We acknowledge the reviewer’s concern about the placement of certain results between the main text and Appendix.

We are happy to further adjust the presentation, whether by bringing key results into the main text or by moving less critical details to the Appendix, if the reviewer feels this would improve clarity, and we are open to continued discussion to make the structure clearer for the final version.

[W2] Deeper Mechanism Behind Test-OI Generalization

We appreciate the reviewer’s insightful question on the deeper cause of Test‑OI generalization.

Our study is scoped around how LLMs learn implicit reasoning, and we provide an empirical mechanistic explanation—as also noted by the reviewer in the strength section:

"...ID atomic triples...encourage the bridge entity representations under OOD scenarios to converge onto those under ID scenarios.."

We intentionally stopped at this level of analysis for two reasons:

1. Scope alignment with the main research question. Our goal is to characterize how implicit reasoning emerges through data, training dynamics, and internal representations. As Test‑OI generalization constitutes only one component of the broader question, our current explanation is both intuitive and sufficient for supporting this specific behavior, without over-extending the paper into a deeper causal analysis.
2. Ensuring rigor and generality. We believe that a rigorous investigation requires a more general study (e.g., connecting to representational collapse), which cannot be convincingly achieved within the current symbolic setup alone. Therefore, we treat this as an exciting direction for future work rather than something we aim to fully resolve in the current paper.

We will clarify this scope in the revised version and leave a deeper investigation of the timing and underlying dynamics of Test‑OI generalization as a natural direction for future work.

We thank the reviewer again for thoughtful feedback and hope our responses have addressed the concerns raised.

We sincerely thank the reviewer for the thorough reading of our paper and for the detailed and constructive feedback. In the following, we address the reviewer’s questions and concerns by grouping related points.

[W2, Q1] Deliberation tokens

We thank the reviewer for the insightful suggestion regarding the use of deliberation tokens. Our goal in this paper is to investigate the emergence of implicit multi-hop reasoning under the maximum architectural constraint—the transformer’s ability to reason without any explicit space or token dedicated to intermediate computation. We therefore focus on the standard autoregressive setting, where the model must internally encode and reuse intermediate entities.

That said, we agree that introducing deliberation tokens, as proposed by the reviewer, opens up a natural and practical extension to our study. It could allow the model to externalize intermediate representations, which would bridge implicit reasoning and practical CoT-style explicit processes. We will explicitly incorporate this direction into our discussion of future work in the paper.

[Q2] Clustering Generality

We thank the reviewer for pointing out both the visualization and interpretability issues regarding clustering.

We agree that the pale-yellow in Figure 5 is difficult to distinguish. We have already updated the figure with more saturated colors, and slightly increased the marker size for better visibility. These changes will be reflected in the revised version.

Regarding whether the observed clustering is a global effect, we appreciate the reviewer’s insightful question. To address this, we randomly selected a set of different intermediate entities and computed the average pairwise cosine similarity among their representations over training.

We report the average similarity across Phases I–III in the following table:

As shown, the cross-entity similarity consistently decreases. This trend strongly argues against a global representational collapse and confirms that the observed clustering is entity-specific, not a byproduct of general compression or training dynamics. We hypothesize that this decreasing trend results from entity representations being actively pulled toward their own ID-derived cluster centers, which increases intra-entity similarity while decreasing cross-entity overlap.

This additional result provides even stronger support for our original conclusion. We will incorporate the findings into the revised version of the paper, and we are grateful to the reviewer for the suggestion that inspired this analysis.

[W3] Applicability

We thank the reviewer for raising this important point regarding the practical applicability of our findings. While our main experiments are conducted in a controlled symbolic environment, we believe our conclusions offer useful insights into how implicit multi-hop reasoning may operate within pre-trained LLMs as well.

Regarding applicability to LLMs "in the wild":

Although we do not directly evaluate pre-trained models, our findings align well with several recent empirical results on LLMs and provide coherent explanations for their observed behaviors:

1. Our finding in Section 2.3—that models don't rely on atomic triples to learn implicit reasoning—helps explain why single-hop knowledge editing fails to transfer to multi-hop queries (Zhong et al., 2023). Conversely, Section 2.4 shows that such reasoning is acquired from compositional data, which aligns with recent findings that editing LLMs using multi-hop prompts leads to better generalization on multi-hop tests (Yao et al., 2025, Zhang et al., 2024).
2. Our finding in Section 2.4 also sheds light on why exposing LLMs to richer compositions improves multi-hop reasoning (Zhang et al., 2024, Xu et al., 2024).
3. Our Section 4.3 result—that generalization to OOD triples only occurs in the first hop—offers a mechanistic interpretation of the observation in Feng et al., 2024, where second-hop OOD queries are significantly more difficult than first-hop OOD ones.

In short, our findings offer a mechanistic lens through which to understand a range of puzzling behaviors already reported in pre-trained LLMs.

Regarding fine-tuning on pre-trained models with new entities:

We agree that fine-tuning LLMs on new knowledge could be a path to validate our findings. However, multiple recent works have found that knowledge acquired during fine-tuning can behave quite differently from knowledge learned during pretraining, and may even distort pre-existing structure (Gekhman et al., 2024, Zucchet et al., 2024). As our primary goal is to understand the mechanism through which implicit multi-hop reasoning emerges, we chose to train models from scratch in a controlled environment.

Nonetheless, we acknowledge that incorporating experiments with pre-trained LLMs and injected new entities would be a valuable future direction, and we are actively exploring such extensions to test the generality of our conclusions.

[W4] Clarifying Motivation and Connection to Prior Work

We thank the reviewer for pointing out that parts of our study may appear exploratory and not fully integrated with prior research. We acknowledge that this impression likely arises because our motivation was not stated clearly enough in the current version.

An important motivation is the lack of consensus in prior works regarding how LLMs perform implicit multi-hop reasoning and whether they can reliably do so, existing studies report conflicting conclusions on key aspects (see Appendix A). As described in line 34 of the paper, we believe that a key reason for this inconsistency is:

...training data lacks precise experimental control, making it challenging to conclusively determine...

To address this gap, we designed a highly controlled symbolic setup to systematically revisit these questions.

We appreciate the reviewer’s suggestion to better connect our findings to the prior works. In the revised version, we will make this motivation explicit in revised version and clarify how our study aligns with and challenges prior assumptions.

[W1, Q3] Clarity

We appreciate the reviewer’s helpful suggestions on improving the clarity and readability of our paper. We will make the following revisions in response:

• Simplify and break down long sentences. We will break down long sentences to make the language more concise and direct, while preserving the intended technical content.
• Terminology Clarification. We will ensure that all technical terms (especially on page 2) are clearly defined at their first mention.
• Rework the definition of Cross-Query Semantic Patching. We will present the procedure in a clear and sequential manner, consistent with the reviewer’s suggestion.
• We will avoid the “Train-II vs. Train 2” ambiguity by first introducing Train-OI as recommended by reviewer.

Again, we sincerely thank the reviewer for the careful reading and thoughtful suggestions. We hope that our responses have addressed your concerns and clarified the paper. We would be happy to provide any additional clarifications if needed and look forward to further discussion.

We thank the reviewer for the thoughtful feedback and for highlighting the strengths of our work. In the following, we address the reviewer’s questions and concerns by grouping related points.

[Q1] Justifying the Low OOD-Derived Point Ratio in Figure 5

We thank the reviewer for pointing this out. It is correct that the number of OOD-derived points in Figure 5 is smaller than ID-derived points. This stems from our intentional data imbalance. As justified in Appendix B (line 543), this high ID/OOD ratio reflects the realistic distributional skew seen in LLM pretraining corpora:

...they reflect plausible properties of real-world pretraining corpora...most knowledge entities are learned in richly connected contexts (analogous to our ID triples), while only a minority are sparsely anchored or appear in isolation (analogous to our OOD triples)...

This design enables our symbolic environment to approximate realistic structure.

To further address the potential concern about this imbalance, we also conduct an ablation study in Appendix H. We show that only high ID ratios lead to successful cross-distribution generalization.

[W2] Novelty Beyond Wang et al. [24]

We appreciate the reviewer’s assessment regarding the relationship with Wang et al. [24]. While our work indeed builds on their symbolic environment, its novelty lies in substantial extensions in both breadth and depth, as detailed below.

Our contributions cover four key dimensions of multi-hop implicit reasoning:

1. Data Requirements. Our findings in Sections 2.3 and 2.4 challenge the prevailing assumption that LLMs perform implicit reasoning by merely combining known atomic triples (see Appendix A). We identify what data actually supports generalization, and further uncover fine-grained query-level requirements.
2. Training Dynamics. We uncover a three-phase progression of learning and explicitly characterize the boundaries of OOD generalization under diverse training configurations, including the 3-hop setting. In contrast, Wang et al. [24] concluded that OOD generalization does not emerge.
3. Interpretability Tools. We introduce new tools to probe internal representations.
4. Mechanistic Insight and Disproof of Prior Assumptions. As the reviewer noted, our analysis directly challenges the widely held assumption—present in Wang et al. [24] and others—that decodability implies reasoning utility. Instead, we demonstrate that successful reasoning aligns with representational clustering in hidden space.

To be transparent, we do overlap with Wang et al. [24] in two respects:

1. Dataset Construction. We adopt their symbolic setup, but we extend it with new training and test configurations that enable fine-grained ablations.
2. Grokking Observations. Our Phase II aligns with their grokking transition. However, while Wang et al.[24] emphasized that grokked models exhibit implicit reasoning capabilities, we focus on how these capabilities emerge and how they generalize—including to OOD settings.

Together, these overlaps represent only a small part of our contribution. The core of our work centers on fine-grained data requirements, generalization boundaries, and mechanistic interpretability—areas not well-addressed in Wang et al. [24].

[W3] Regarding Single-Trial Experiments

"the experiments contain a single trial per configuration...I would recommend running several trials and including error bars..."

We appreciate this insightful suggestion. Due to the limited time available in the rebuttal period, we are unfortunately unable to rerun full training trials.

That said, we have already conducted experiments under several variant configurations (e.g., with 1000 vs. 2000 entities), which introduce variation in data scale while preserving task structure. These results consistently show the same behavioral trajectory and representational patterns, suggesting that our findings are robust to data-level perturbations.

We will include these results in the appendix of the revised version as additional support for the stability of our observations.

[W1, Q2] Writing and Presentation Improvements

We appreciate the reviewer’s helpful suggestions on improving the clarity and readability of our paper. We will make the following revisions in response:

• Title Revision: We agree that "How do Transformers Learn Implicit Reasoning?" is more natural and grammatically correct. We will adopt this revised title.
• Terminology Clarification: We will ensure that all technical terms (e.g., ID triples) are clearly defined at their first mention.
• Improved Motivation and Task Definition in Section 2: We acknowledge that the current version of Section 2 assumes prior familiarity. In the revision, we will provide a clearer motivation and explicitly define the reasoning task with illustrative examples.
• Model Description Clarity: We will clarify that “GPT-2” refers to the architecture only and specify the model size explicitly.
• Simplifying Complex Phrasings: We will revise selected portions of the text to make the language more concise and direct, while preserving the intended technical content. We agree that some sentences are unnecessarily complex in the current form.

We thank the reviewer again for thoughtful feedback and hope our responses have addressed the concerns raised.