We respectfully disagree with the reviewer’s assessment:

1. The reviewer appears to fundamentally misunderstand our position. They attribute to us Pearl’s position:

   The authors start with Pearl’s Ladder of Causation, arguing that this framework suggests DNNs trained in a “statistical mode” — that is, to predict passive observations — can only capture associations (L1) […]

   We explicitly argue against this position, as stated in the first paragraph of our paper:

   According to Pearl [36], this hierarchy implies that DNNs trained in a “statistical mode” to predict passive observations can only master associations (L1) and are prevented from ever reasoning about actions, experiments, and explanations (L2 and L3). While PCH captures fundamental distinctions among causal concepts—providing an extremely productive theoretical framework with wide-reaching implications—we do not think that Pearl’s claim is one of those implications.

2. The reviewer further asserts that our contributions revisit ideas that have already been extensively discussed in the literature, but provides no citations or specific references to support this claim. In contrast, we have reviewed and cited the relevant body of work in our related work section—including studies of causal capabilities in LLMs and theoretical arguments regarding LLM causal abilities (e.g. Zečević et al., 2023).

3. Our contribution is distinct: we provide a controlled existence proof that next-token prediction on passive data can yield a functioning internal causal model and inference engine. We support this with mechanistic evidence (via decoding and gradient-based interventions), not just behavioral results. This goes beyond existing observational studies and directly challenges the notion that true causal discovery and causal reasoning engines at interventional (L2) and counterfactual (L3) levels of Pearl’s Causal Hierarchy cannot emerge through statistical prediction objectives.

Many in the causal inference community are aware that causality can be learned from statistical data. With sufficient changes/diversity or constraints (e.g., via sparsity) in the data, causal identifiability becomes possible.

Even for LLMs trained solely through next-token prediction, several works have explored related aspects:

[1] E. Marconato, S. Lachapelle, S. Weichwald, and L. Gresele. All or none: Identifiable linear properties of next-token predictors in language modeling. arXiv:2410.23501, 2024.

[2] K. Park, Y. J. Choe, and V. Veitch. The linear representation hypothesis and the geometry of large language models. arXiv:2311.03658, 2023.

[3] Y. Jiang, G. Rajendran, P. Ravikumar, B. Aragam, and V. Veitch. On the origins of linear representations in large language models. arXiv:2403.03867, 2024.

[4] Y. Liu, D. Gong, Y. Cai, E. Gao, Z. Zhang, B. Huang, M. Gong, A. van den Hengel, and J. Q. Shi. I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data? arXiv:2503.08980v3, 2025.

Given these prior works and the existing understanding in the community, could you clarify the specific contributions and novelty of your work?

Many in the causal inference community are aware that causality can be learned from statistical data. With sufficient changes/diversity or constraints (e.g., via sparsity) in the data, causal identifiability becomes possible.

We agree that, under certain assumptions, statistical data may be enough to answer causal questions or identify the causal model. Of course, Pearl would agree with this too.

The controversial foundational question we are interested here is: Can deep neural networks (”statistical systems”) build true causal models of the world by predicting passive streams of data (e.g. passive streams of text tokens)?

Pearl (in “Theoretical impediments…” paper and in “The Book of Why”) and others (Zečević et al., 2023) have argued no. Statistical prediction of passive data can only capture associations (L1—correlational relationships), but not interventions (L2) or counterfactuals (L3) because of Pearl’s Causal Hierarchy (PCH). LLMs can answer some L2/L3 queries because they are “causal parrots” capturing only the “correlation of causal facts” (Zečević et al., 2023).

We hypothesized yes, this should be possible and we should not rule out true causal models in deep neural networks / LLMs a priori. In other words, we do not think that PCH implies the impossibility of building true causal models through statistical prediction of passive data streams. We are motivated by the idea that building causal models and causal inference engines may be useful for prediction purposes (e.g. to predict the next token in text).

Our specific contribution is providing an empirical counterexample to Pearl’s impossibility claim. None of the works cited by the reviewer addresses the question we are interested in. Our contribution is novel in that it uses mechanistic interpretability methods (probing and interventional analyses) to answer a foundational question about deep neural networks and causality and presents a valuable addition to the literature on causal abilities of LLMs.

Our work is complementary to and inspired by many works on emergent (linear) representations in LLMs (e.g. we’re particularly inspired by Othello-GPT work from Li et al. 2022). Works like Liu et al. (2025) [4] are consistent with this general direction—trying to understand the emergent representations within LLMs.

However, many of these studies (including Liu et al. 2025) still work within statistical frameworks (even if couched in “causal” language). LLM causal skeptics may respond by saying that the LLM just captures statistical factors in the data. While it may be possible to extend such work to demonstrate that uncovered latent factors are actually used in internal causal models, this requires going beyond statistical decoding to show genuine causal functionality. This is precisely what our intervention experiments accomplish, demonstrating that these aren't just decodable statistical patterns, but functional causal representations that can be manipulated to predictably alter model behavior.

In the context of such works, we do think we provide a novel piece of the puzzle where we directly empirically show that deep neural networks can learn real causal models and causal inference engines through next-token prediction.

We appreciate the reviewer’s thoughtful and constructive engagement in the discussion, providing the references and their reconsideration of the initial assessment.

We thank the reviewer for a constructive assessment of our work.

Weaknesses

Highly Constrained Experimental Setting: The paper's claims are derived from a very restrictive setting involving linear Gaussian SCMs with only four variables and ternary weights (-1, 0, 1). While this simplicity is necessary for the detailed analysis performed, it raises significant questions about scalability and generalizability. It remains unclear whether the same mechanisms would hold for SCMs with more variables, continuous causal strengths, or non-linear relationships.

• We acknowledge the constrained setting in the discussion. As pointed out by the reviewer, this simplicity was partly dictated by the detailed analysis performed.
• Note that Pearl's claim was categorical and universal: it is theoretically impossible for any deep neural network trained using statistical prediction objectives to ever reach interventional (L2) or counterfactual (L3) levels of the causal hierarchy. For such sweeping theoretical claims, a single well-designed counterexample is sufficient to demonstrate that the claim is false.
• Pearl (interview in Amstat News, September 2023) and Zečević et al. (2023) have implied that L2 and L3 LLM behavioral competence is due to parroting memorized surface statistics, arguing that LLMs cannot possess an actual internal causal model (due to Pearl’s Causal Hierarchy). Our mechanistic analyses (probing and interventions) are particularly valuable to respond to such claims since we show, via a concrete counterexample, how internal causal models may be implemented inside the “black box” LLM.
• While questions about scalability to more complex SCMs are interesting for future work, they do not diminish the theoretical significance of showing that such learning is possible in principle.

Use of a Simplified Artificial Language: The model is trained on a simple, unambiguous artificial language where causal relationships are explicitly described. Natural language, in contrast, is noisy, ambiguous, and context-dependent. The degree to which these findings depend on the clean, structured nature of the input data is a major open question that limits the direct applicability of the conclusions to real-world LLMs.

• We acknowledge that there are important differences between our artificial language and natural language. Natural language is indeed more ambiguous and complex.
• However, we believe the artificial language setup serves our specific research goal: providing a controlled test of Pearl's theoretical claims about neural network limitations. This required a setting where we could have ground truth causal structures for validation, generate unlimited training data with analytical solutions, and perform precise mechanistic analysis of learned representations.
• Most importantly, causal relationships are not explicitly described in our artificial language. The underlying SCMs appear only implicitly through noisy interventional samples and inference examples. The model must discover the hidden causal structure through next-token prediction, as it would need to in natural language.
• While our artificial setup is simplified, we view this as a necessary first step for establishing that such learning is theoretically possible. Future work can explore how these findings extend to more complex, naturalistic settings.

Limited Discussion of Implications for LLMs: The discussion section is brief and does not sufficiently bridge the gap between the findings in this toy model and the behavior of large-scale language models. While the paper serves as an interesting “existence proof”, it offers little speculation on how these mechanisms might manifest, or be identified, in models trained on vast, unstructured web-text. A more thorough discussion of these implications would strengthen the paper's contribution.

• We agree with the reviewer that we should extend the discussion on the implications of our results for LLMs.
• How would these mechanisms manifest in LLMs?
  • Our results suggest that LLMs may be building internal causal models and inference engines to improve next-token prediction. However, we hypothesize that the causal models and engines of pre-trained LLMs are likely to be less formal and more intuitive—supporting everyday causal inferences present in natural language rather than formal causal reasoning (analogous to how intuitive physics differs from formal physics).
• How could causal models and inference engines be identified in LLMs? Several approaches seem promising:
  • Behavioral approaches. Designing evaluation methods and datasets to understand LLM causal reasoning (e.g. by asking LLMs to judge causal vignettes, like in Kıcıman et al. 2023). We cite many studies evaluating LLM causal abilities behaviorally in our related work section. While these studies show mixed results, there is no clear picture when and why LLMs succeed or fail in their causal inferences, and much remains to be done.
  • Mechanistic approaches. To apply similar probing/interventional techniques as in our paper, we would need to train much more sophisticated decoders. One possibility: Train a decoder that maps internal model activations to natural language descriptions (trying to read the “mind” of the model). For instance, the decoded text may say something like “Alice drank the potion…”. We could then intervene on that internal representation using similar gradient based methods as presented in our paper, changing the output of the decoder to something else (”Alice spilled the potion”), and observing functional implications on model output.
• We will expand the discussion on broader implications for causal reasoning in LLMs—and to clarify how our setup lays a foundation for extending mechanistic analysis to real-world models.

Questions

1. The study successfully demonstrates causal discovery in SCMs with four variables. How is the model's performance expected to scale as the number of variables (and thus the complexity of the causal graph) increases? Is there a theoretical or practical limit where the next-token prediction signal becomes insufficient?

• A priori, there is no theoretical limit for the number of variables or complexity of the SCMs.
• However, we expect there to be some practical performance limits when we increase the complexity of the SCM:
  • computational complexity: the number of possible SCMs grows exponentially with the number of variables
  • sample complexity: more interventional samples will be needed to identify the SCM, resulting in longer training runs
  • representational capacity: larger SCMs will require higher dimensional representations
• Nevertheless, multiple factors suggest reasonable scalability: real-world causal structures are often sparse, LLMs show reasonable scaling properties on other tasks, and next-token prediction objective would remain the same, regardless of the complexity of the SCM.

2. The findings rely on a simple, unambiguous artificial language. How robust do the authors expect these results to be if the language included more naturalistic complexities, such as ambiguity, paraphrasing, or implicit causal statements?

• This is an important question for future work. We expect that the findings would be robust: LLMs have demonstrated remarkable robustness to ambiguity and paraphrasing across many domains.
• Furthermore, our artificial language is already implicit in a crucial sense: the model is never explicitly provided the underlying SCM structures and has to uncover them through next-token prediction, similar to how it would need to extract causal relationships from natural language.
• The key insight from our work is that the fundamental capability—learning causal models through statistical prediction—is possible. While natural language complexities would certainly make the task more challenging, they do not seem to pose fundamental theoretical barriers to this type of learning.

3. The probing and intervention techniques are central to the paper's analysis. Do the authors believe these methods, particularly the gradient-based interventions, could be practically adapted to analyze causal representations in much larger, pre-trained foundation models?

• Yes, this is exactly what we would like to explore in future work!
• See above for a description how the mechanistic gradient-based interventions may be applied to a pre-trained LLM.
• We are also interested in applying similar techniques to video generative models (akin to Sora). We hypothesize that video generative models may learn simulators of the world that contain implicit causal structure. To explore this hypothesis, we could train decoders of physical objects and agents and apply similar intervention techniques to the ones presented in our paper. Another path is to train a video generative model ourselves using a game engine (with known ground truth causal structures) and see whether we can decode the causal structures and intervene on them with predictable effects on model’s video generation output.

4. The ternary weight system makes faithfulness violations (e.g., pathways cancelling out) plausible. Could the authors comment on whether the model successfully learns the true SCM structure in such cases, or if it is misled by the resulting statistical independencies in the data?

• It is true that our particular setting presents faithfulness violations. Take the case when paths cancel out: $V_1 \rightarrow_{+1} V_2 \rightarrow_{+1} V_3, V_1 \rightarrow_{-1} V_3$. While in theory the structure could still be identified from the observational covariance structure, it may be difficult to achieve in practice.
• More importantly, our model relies on interventional data to identify the SCM structure, so even when there are faithfulness violations, interventional data would circumvent those issues entirely.

We thank the reviewer for constructive feedback and pointing us to some relevant literature.

Missing related works in uncovering trees or structural models from LMs. There are existing works that come from the more classic NLP interpretability camp (see [1] and its cited works) which are trying to (1) uncover tree structures (i.e., syntactic ones) from LMs; (2) or even induce trees in LMs. I think it would be worth discussing the main differences between this work and them. In particular, focusing on what additional insights can be gained by using SCM-induced datasets from this paper would be interesting. At a high level, both lines of work are trying to interpret very similar things. In [1], a PCFG model is used to generate synthetic data that are then used to train LMs, and interpretability methods are used to assess whether we can uncover syntactic tree structures (i.e., a PCFG graph with edge weights of 1) from the LM.

Needs to be better motivated. Although it is interesting to see some causal structures emerge in the LMs for this very specific synthetic dataset, I think it would be beneficial to say more about the motivation behind this research in a precise way. For instance, what is the additional gain from your setup compared to previous ones (including the line of work I mentioned above)? With your methods, are you finding additional interpretability signals that inform us how transformers work beyond what people already know (e.g., transformers generalize, transformers learn structures of the dataset generation process)?

• The hierarchical generalization test in Murty et al. (2023) has important parallels to our generalization challenge, as both works test whether models learn hierarchical rules rather than surface patterns in their respective domains. This literature is relevant and we will include it in our related work section.
• While Murty et al. focus on syntactic structure discovery, our work addresses a fundamental question in causal inference: whether neural networks can build causal models and perform counterfactual reasoning via statistical prediction objectives, directly challenging Pearl's theoretical claims about the limitations of deep neural networks.
• The key additional gain from our setup is providing an empirical existence proof against a foundational theoretical claim. Pearl and others have argued it should be theoretically impossible for statistical learning systems to reach interventional (L2) and counterfactual (L3) reasoning. Our controlled SCM setup allows us to test this claim with ground truth causal structures and analytical solutions for validation.
• The mechanistic analysis in our paper addresses a further critique from causality researchers.
  • Pearl (Amstat News, Sept 2023) and Zečević et al. (2023) argue that, insofar as LLMs can answer L2/L3 queries, they do so by memorizing and “parroting” causal facts—based on their reading of Pearl’s Causal Hierarchy.
  • Our mechanistic analysis directly refutes the assumption that statistical learning systems cannot build true causal models and reasoning engines through prediction objectives alone. Beyond behavioral generalization, we decode SCM weights from the residual stream and intervene on those representations. Targeted edits reliably flip counterfactual predictions in the same direction as the modified ground-truth SCM—demonstrating that the model has constructed an internal causal model that is interpretable and functionally used in inference.

We thank the reviewer for their feedback. However, the review appears to misunderstand the fundamental goal of our paper. Our aim is not to propose a method that outperforms existing causal discovery algorithms. Rather, we explore a fundamental question: Can deep neural networks trained on passive data streams build causal models and learn causal inference engines?

Pearl and others have argued this should be theoretically impossible: that such networks will be limited to associations (L1) and cannot reach interventional (L2) or counterfactual (L3) reasoning. Through carefully crafted experiments, we provide an empirical existence proof that this is possible.

Weaknesses

1. Lack of Consistency Guarantees

The paper does not provide theoretical guarantees that the model's causal discovery process is consistent (i.e., that it converges to the true SCM as training data increases). Consistency is a fundamental requirement for reliable causal discovery, and its absence limits the method's applicability to real-world problems where correctness is critical.

Without such guarantees, it is unclear whether the model's success is due to the specific simplicity of the linear Gaussian SCM class or a more general capability.

• Our primary goal is an existence proof against Pearl’s strong claim, not an improvement over existing methods. Theoretical convergence guarantees therefore lie beyond the scope of this paper.

2. SCM Index Dependency

The model relies on predefined SCM indices to reference causal structures, which may act as a crutch, allowing the model to "look up" SCMs rather than infer them from raw data.

• SCM indices are random by design and have no inherent meaning at the beginning of training. The model then discovers the underlying SCM associated with a particular SCM index from raw data by predicting text strings. Any model needs some data about an SCM to generalize to it. Without any information, generalization is impossible.

In real-world settings, causal structures are not pre-indexed, so the model's ability to discover causal relationships from scratch remains unproven.

• In real-world settings, causal contexts are identified through variable names (e.g. “smoking”, “cancer”), domain descriptions, etc. Since variable names are the same within every SCM in our setup, our SCM indices serve this same function.

3. Weak Experimental Comparisons

The paper does not compare the model's performance against established causal discovery algorithms (e.g., PC, FGES, LiNGAM) or state-of-the-art neural causal methods (e.g., DECI, DCDI). Such comparisons are necessary to assess whether the transformer-based approach offers any advantages in accuracy, scalability, or robustness.

The lack of benchmarks on standard causal discovery datasets (e.g., synthetic graphs, real-world cause-effect pairs) further limits the ability to evaluate the method's practical utility.

• Our goal is not to introduce another causal‐discovery algorithm, but to answer a deeper question: Can a transformer, trained solely via next‑token prediction, both discover the causal models and build its own L2/L3 causal inference engine? Classical methods like PC and FGES are designed to recover DAG structure from numerical data. Neither employs a text‑prediction loss nor produces counterfactual answers. In fact:
  • structure‑only: PC FGES only output a graph. They have no mechanism for computing “what if?” counterfactuals, so their counterfactual‑query accuracy is effectively 0%
  • orthogonal design: These tools rely on independence tests or score maximization under explicit SCM assumptions, whereas our model learns everything end‑to‑end via language

4. Limited Scope of SCMs

The experiments are restricted to linear Gaussian SCMs with at most four variables. While this simplifies analysis, it leaves open whether the method can handle more complex, non-linear, or high-dimensional causal structures.

• In the discussion, we acknowledge the limited scope of the SCMs tested.
• However, we would like to note that an existence proof requires just one example.
• Moreover, our simplified setting was a strategic choice. Previous studies lacked full control of training data (unclear memorization vs. generalization) and did not probe internal representations (unclear if LLMs build causal models vs. parrot surface statistics). Our controlled setting with mechanistic analysis addresses both limitations.

The reliance on analytical solutions for counterfactual queries (which may not exist for non-linear SCMs) suggests the approach may not easily generalize beyond the tested class.

• The choice of linear SCMs was a methodological choice to use the analytical solution to efficiently generate training data.
• We have already explored creating training data for non-linear SCMs by computing answers to counterfactual queries numerically using probabilistic programming (Pyro package). While it takes longer to generate such training data, it is definitely possible to apply our approach to non-linear SCMs.

5. Ambiguity in Probe Interpretation

While probes can decode SCM weights, the paper acknowledges that decoding accuracy does not necessarily imply the model uses these representations for inference (Section 4.4). This weakens the claim that the model truly "discovers" causal structure, as opposed to merely encoding it incidentally.

• This appears to be a misunderstanding of our analyses. Our successful gradient-based interventions (in sections 4.3 and 4.4) demonstrate that the model actually uses these representations functionally, not just incidentally. When we modify the decoded SCM weights, the model's outputs change predictably—this directly proves the representations are causally relevant to the model's reasoning process.

The intervention experiments, while compelling, are limited to a small set of SCMs and weight modifications. Broader validation is needed to confirm that the model's behavior is systematically controlled by the manipulated representations.

• We respectfully disagree with this assessment. To systematically measure whether we can control the representations, we develop a quantitative “intervention score”. We then measure this score across 200 randomly selected causal queries (with different SCMs from both $D_{train}$ and $D_{test}$) for all SCM weights (10) changing them to two possible other weight values (2) across all layers (12) of the model for both probe types (linear and MLP), resulting in 200 * 10 * 2 * 12 * 2 = 96,000 evaluations. So our interventional analyses are quite extensive, and we show that the model’s behavior is indeed systematically controlled by the manipulated representations in section 4.4.

Questions

1. How does the model's causal discovery accuracy compare to traditional constraint-based (e.g., PC) or score-based (e.g., FGES) methods on the same SCM class?

Our model's near-optimal counterfactual performance on the generalization challenge demonstrates successful SCM discovery. In contrast, PC cannot handle interventional data and would struggle with identifiability in our linear Gaussian setting. FGES might recover the causal DAG but would require explicit priors about noise distributions and SCM structure. FGES also cannot perform counterfactual inference and would achieve 0% on our generalization challenge. Our approach uniquely learns both structure discovery and counterfactual reasoning from text, addressing Pearl's fundamental claims about neural network limitations rather than competing with dedicated causal discovery methods.

2. Can the approach scale to more complex, non-linear SCMs, and how does its performance degrade compared to specialized causal discovery tools?

Our approach should scale to non-linear SCMs since the methodology (learning from text descriptions) does not depend on linearity assumptions. We have already explored generating training data for non-linear SCMs using numerical methods with probabilistic programming. While we focus on linear Gaussian SCMs here for analytical tractability and interpretability, the key advantage is that our approach learns both structure discovery and counterfactual inference without requiring explicit assumptions about functional forms, noise distributions, or causal mechanisms (unlike specialized tools that need such priors).

3. Does the method outperform existing algorithms in settings where only interventional data (no counterfactual examples) are available during training?

While performance comparisons could be interesting, our work addresses a different question: Can transformers emergently learn causal reasoning from text? This is a fundamental capability question rather than a methods comparison.

4. What are the computational and sample complexity trade-offs between this approach and classical causal discovery methods?

These are excellent questions for future work. Here, our focus is on demonstrating that causal discovery and counterfactual inference engines can emerge through statistical prediction objectives, addressing Pearl's fundamental claims about neural network limitations.

Limitations

The requirements of SCM index may restrict the proposed approach to be really useful.

• The SCM index requirement appears to be a misunderstanding. SCM indices are arbitrary labels with no inherent meaning—the model must discover the underlying causal structure associated with each index from the raw data itself.
• More broadly, this criticism misses the flexibility of our approach. In real-world settings, causal contexts are identified through various cues (variable names, domain descriptions, and so on). Our SCM indices serve the same function as these identifiers. The power of next-token prediction is precisely that it can learn from any text format, including natural language descriptions of causal relationships.

I respectfully disagree with the claim that causal context in real-world applications can be reliably identified through variable names, domain descriptions, or other side information.

Questions like “Does smoking cause cancer?” identify/label/distinguish causal contexts.

Such cues are only meaningful when prior knowledge of the underlying causal structure exists. In the absence of this knowledge, these forms of side information merely serve to label or distinguish contexts, without providing genuine causal insight.

Yes, one needs to know the meaning of “smoking”, “cause”, “cancer”, and have prior knowledge about the causal relationship to instantiate the correct causal model.

In this scenario, directly inputting the SCM index as a feature is problematic. If the same index appears in the test set, it may lead to information leakage. Conversely, if the index is unseen during training, the model’s generalization ability becomes questionable.

Let us explain our study design and the generalization challenge once more. $D_{test}$ is the set of SCMs for which the model only sees text describing interventional data samples (DATA strings in our paper). Let’s take one SCM from $D_{test}$ and call it A.

During training, the model learns the causal structure of A by predicting DATA strings that describe interventional data derived from A. It also learns the counterfactual inference engine from INFERENCE strings about other SCMs, but not A. It never sees a counterfactual inference query regarding A.

During testing, we ask the model to generalize to counterfactual inference queries about A. This answers two questions at once:

1. whether the model learnt the causal structure of A from interventional data strings about A
2. whether the model learnt a counterfactual inference engine from other SCMs that generalizes to A

This is our generalization challenge setup. We ask the reviewer to be more specific about their concern about providing SCM index as a feature and “information leakage” in the context of our setup.

Neural networks, being non-parametric models, do not guarantee reliable extrapolation to entirely new indices. Therefore, using SCM indices as inputs is generally not a principled approach.

First, neural networks are parametric models. They have a finite number of parameters and their complexity does not grow with the number of data points, unlike k-nearest neighbors or Gaussian processes. More importantly, it will be impossible for any model to extrapolate to entirely new indices without seeing any data about that index. It’s like asking someone to extrapolate to “X causes Y” context without providing any information about the two variables “X” and “Y”.

Reviewer's concerns do not seem to address the actual experimental design or contributions of our work. We have demonstrated that transformers, through statistical prediction objectives, can learn causal structure from interventional data and apply a counterfactual reasoning engine across different causal contexts. This challenges a fundemantal claim by Pearl about the limitations of deep neural networks and is a meaningful contribution regardless of how the contexts are identified.

There may be some underlying shared structures and insights between SCMs that the network may learn to exploit (like the one the reviewer suggested). We do not think this constitutes memorization in the sense of memorizing the text or capturing just the surface statistics of the text. Rather, this would constitute learning the deep structure of the problem, i.e. that a set of SCMs share some important structural similarities which may help generalize. Also, remember that the actual distributions of the SCM still do depend on the SCM coefficients which are always unique.

Recall also that our goal here was an existence proof against Pearl's strong claim that it is fundamentally impossible for deep neural networks trained using statistical prediction objectives to learn causal models and causal reasoning engines at L2/L3. Through behavioral generalization and mechanistic interpretability methods (probing and interventional analyses), we show that it is possible. Future work could explore more complicated generalization challenges and real-world applications.

We thank the reviewer for the close reading and thoughtful critique! Their feedback should improve the quality of the paper.

Fixed global topological variable order ($V_1 \prec V_2 \prec V_3 \prec V_4$)

Unfortunately, it is a main flaw/limitation of the study design that the causal structure (i.e., the causal order) is fixed across all 59k+ SCMs. As a result, it is sufficient to memorise the correct (partial) causal order globally. That is, the LLM does not have to learn how to perform causal discovery (rendering the title highly misleading), but only needs to solve the simpler, purely statistical problem of learning the partial correlations among variables within each SCM. […] what the study in its current form demonstrates (and does so very well) is the following: given a known/fixed causal order, an LLM trained via next token prediction can learn the path coefficients within each SCM, and, given sufficiently many examples, can learn the abduction-action-prediction algorithm for counterfactual inference.

Model passes generalization challenge without fixed topological order

To directly address the reviewer’s concerns, we demonstrate that our approach works without fixed variable ordering:

• We retrained the model without fixed topological variable order using a smaller setup (due to time constraints):
  • 3 variable linear Gaussian SCMs
  • $w_{ij} \in \\{-2, -1, 0, +1, +2\\}$ (5 possible weight values)
  • $|D_{train}| = 5^6 -|D_{test}|= 15625 - 200$ (counterfactual queries + answers)
  • $|D_{test}| = 200$ (interventional data only)
  • we randomly permute the variable names within each SCM
• The model passes the generalization challenge. It answers counterfactual queries for SCMs in $D_{test}$ near-optimally, despite only seeing interventional data about those SCMs.
• We trained a model with fixed order (no permutation) on the exact same SCM set. It converged faster to near-optimal performance (~200 epoch vs. ~350), indicating that the problem is easier with a fixed variable order. But the fixed order is not essential for the model to discover the SCMs.

Why we chose a fixed topological order

We initially trained models without a fixed topological order, but then pivoted to a fixed order ($V_1 \prec V_2 \prec V_3 \prec V_4$) because of what could be called an “internal representation equivalency problem”:

• Without a fixed topological variable order, for a single ground truth SCM there may be multiple equivalent ways to represent the same SCM internally. For instance, consider a “no edge” SCM (4 variables with biases only) with $4! = 24$ ways to map variable names to internal representations of the variables (e.g. V1 could be mapped to the first variable or the last variable in the internal representation).
• Behaviorally, the model can learn any of these equivalent orderings and produce the correct output. However, it is challenging to set up a probe. The simple residual stream activation map to a fixed weight vector does not cut it because we do not know the internal order of the variables for a given SCM used by the model.
• While it may be possible to train more sophisticated probes without a fixed order of the variables (e.g. by iteratively querying the value of the edge between every two variable names), for the sake of simplicity we decided to fix the order globally, which allowed us to train simple linear and MLP probes without worrying about this internal representation equivalency problem.

Why “causal discovery” in the title is warranted

Even given a fixed topological variable order, the model still has to discover everything from scratch solely through next-token prediction, including:

• The global structure of the SCMs (linear SCMs, Markovian normally distributed noise terms, 4 variables, global order $V_1 \prec V_2 \prec V_3 \prec V_4$)
• Local parameters of each SCM.
• Arguably, the whole problem taken together goes beyond simple “parameter estimation”.

Pearl’s claims and “implicit interventions”

Pearl's claims in the introduction and their relation to the present manuscript appear slightly misleading or inaccurately portrayed. As far as I know, his claims are about a system trained on purely observational (L1) data. While I think the point that natural language often conveys interventional or counterfactual knowledge is a good and valid one, this information is typically implicit in most datasets. In contrast, the interventional nature is made very explicit in the constructed dataset via the DO/OBS distinction.

Regarding Pearl’s claims

The reviewer raises important points about how we characterize Pearl's position. We think it is useful to distinguish between three claims:

1. Pearl’s Causal Hierarchy (PCH) and its uncontroversial implications. Observational (L1) data underdetermines answers about interventions (L2) and counterfactuals (L3), without further assumptions. We agree with that.
2. Erroneous conclusion from PCH about deep neural networks. In the “Theoretical impediments…” (2018) paper and “The Book of Why”, Pearl invokes PCH to argue that deep neural networks trained using statistical prediction objectives will always be limited to L1. We disagree with this position.
3. In so far as LLMs capture L2 and L3, they are “causal parrots”. More recent perspectives like Zečević et al. (2023) try to square recent advances of LLM causal abilities while maintaining (2). They claim that LLMs can make correct L2 and L3 statements, but only because they are “causal parrots” that capture the “correlation of causal facts”. A similar position still seems to be maintained by Pearl (e.g. in an interview with Dana MacKenzie, Amstat News, September 2023), where he says that ChatGPT does not possess a causal model, even if it can answer some L2 and L3 queries correctly.

We hypothesized that LLMs may be building actual causal models and inference engines for the purpose of next-token prediction. Via our artificial language setup, carefully crafted generalization challenge and the mechanistic insights (probing and manipulating model’s learnt representations of the SCMs), we provide an existence proof ruling out (2) and questioning versions of (3) that are based solely on a priori reasons like PCH.

Implicit interventions

Interventions are indeed more implicit in natural language (e.g. “Alice ventured to taste a magical potion…”) compared to our artificial language (”DO V1 2.3…”). The intuition behind our work is that LLMs may learn to detect (implicit) interventions to improve statistical next-token prediction objective (e.g. detecting that Alice drank a magical potion may help the model anticipate that something is about to happen to Alice, improving the ability to predict the next-token).

Regarding observational data and identifiability

For the current design (with fixed causal order), I hypothesise that even observational data should be enough (since no per-SCM causal graph needs to be learnt), provided that counterfactual inference statements are still provided. It could be interesting to try how the model performs when only given counterfactual inference queries (if sufficiently many queries are given, it should be possible to learn the path coefficients) or when additionally given observational instead of interventional data.

• The reviewer appears to be right that, given a fixed causal order and given the particular structure of our SCMs (fixed noise variances, linear Gaussian, etc.), observational data theoretically should be enough to identify the parameters. It could be interesting to see whether our model could uncover the SCM structure from observational data only.
  • However, it may be extremely difficult to identify certain structures in practice. For instance, when paths cancel out ($V_1 \rightarrow_{+1} V_2 \rightarrow_{+1} V_3, V_1 \rightarrow_{-1} V_3$), the only signature in the observational data would be the variance/covariance structure of the variables, which would require precise estimation from finite samples.
  • Moreover, in the rerun version without a fixed topological order (see above), SCMs are no longer identifiable by observational data only.
• Counterfactual inference queries + answers will be enough to learn the coefficients because of Pearl’s Causal Hierarchy (PCH). Since counterfactuals are at the top of the hierarchy (L3), that information determines the other levels: interventional (L2) and observational (L1).
  • While it may be interesting to train on counterfactual inference queries only, we arrived at this particular setup to test the generalization capabilities of the model in the more challenging direction (L2 → L3 of PCH).

Typo in Fig 3.

[minor] in Fig. 3 (top), why does the sample from the interventional distribution contain two different values for ? (V3 -0.9, V3 -1.5) Is this a typo and the first one should be V2? (else, please explain what is going on here)

Thank you for catching this typo!

Cautioning against overconfidence

Readers should not conclude that LLMs outputs necessarily support causal claims, which may lead to false overconfidence in using them for high-stakes causal inference tasks. I think adding a short discussion to clarify this would be a useful addition.

We see our work as an existence proof against Pearl’s strong claim that it is impossible for DNNs trained to predict passive streams of data to build causal models or learn causal reasoning—not that LLMs are good causal reasoners.

We thank the reviewer for raising these points. We will (1) include the motivation for the fixed topological variable order, (2) incorporate the results without a fixed order, (3) clarify our characterization of Pearl's claims and the implicit nature of interventions in natural language, and (4) add discussion cautioning against overconfidence in LLM causal abilities based on our results.

Re: Pearl’s claims and “implicit interventions”. I'm no expert on Pearl's precise claims, but in the generality that they portrayed, they seem slightly absurd. Consider an unconfounded setting in which $X$ causes $Y$. Then, simple supervised learning is enough to recover the causal effect of $X$ on $Y$ since $p(y|do(x))=p(y|x)$. This example highlights that statistical objectives are sometimes sufficient to extract causal knowledge. However, if the causal direction was reversed, the same approach could fail. I thus feel the relevant question is not wheter statistical methods can sometimes work, but whether it can be guaranteed that they will work. I would be curious to hear your take / welcome further clarification. (Based on the other review's and responses, this seems to be a contentious aspect in general...)

• The example provided by the reviewer is valid. Pearl would not disagree with the claim that, given certain assumptions (e.g. that X causes Y), statistical knowledge can be enough to extract causal knowledge. This is relatively uncontroversial.
• The more controversial idea by Pearl is that the (extra-statistical) assumptions must be explicitly provided/implemented if we want to build truly causal AI. More concretely, according to Pearl, training deep neural networks using solely prediction objectives on passive data streams (e.g., next-token prediction on internet text) cannot, in principle, get us to causal models due to the Ladder of Causation/Pearl's Causal Hierarchy.
  • see "Theoretical Impediments to Machine Learning With Seven Sparks from the Causal Revolution" (Pearl 2018) and Chapter 10 in the "Book of Why"
• The causal abilities of large language models (e.g., the ability to answer some L2/L3 queries, as in Kıcıman et al. 2023) do seem to challenge Pearl's strong claim that deep neural networks can never reach L2/L3. However, Pearl (interview in Amstat News, 2023) and others (Zečević et al. 2023 "Causal Parrots" paper) still maintain, without evidence, that LLMs are not truly causal. Zečević et al. (2023) argue that this is a consequence of Pearl's Causal Hierarchy (following the same logic as Pearl in the 2018 paper) and they argue that LLMs are merely "causal parrots" (in the sense that LLMs mostly recite causal facts).
• Our position is simply that we should not rule out causal models in LLMs a priori. So in our work:
  • We first provide a small "intuition pump" in the introduction as to how causal models and causal inference engines may emerge through predicting the next token, given these may be useful for the prediction objective.
  • We then provide an empirical example, where we have both a carefully designed generalization challenge as well as mechanistic interpretability (probing and interventional analyses), demonstrating that actual causal models and inference engines emerge within the activity of model.
• These are complicated and contentious issues indeed. We believe the rebuttal/discussion phase has made these issues clearer and we will incorporate the new insights in the manuscript.

Re: Regarding observational data and identifiability. The claim that the setting without fixed topological ordering is incorrect. Linear Gaussian SCMs with equal error variance are actually identifiable, see: Peters, Jonas, and Peter Bühlmann. "Identifiability of Gaussian structural equation models with equal error variances." Biometrika 101.1 (2014): 219-228.

• The reviewer is absolutely right—we were incorrect about identifiability in the non-fixed order setting. We thank the reviewer for the reference.
• Given this theoretical result, it may indeed be interesting to test whether the network could learn the SCM parameters from observational data only. However, this is not central to our claims in the paper and could be left for future work.

Concrete commitments for revision:

Beyond previous commitments in the original rebuttal, we will make the following specific changes to address the reviewer's concerns:

1. Restructure the paper to lead with the non-fixed topological order results as our primary contribution that demonstrate genuine causal discovery
2. Reframe the fixed-order experiments as a further analysis that enables detailed mechanistic interpretability (probing and intervening)
3. Expand the Pearl discussion, given new insights from the rebuttal/discussion period. We will make it clear that the controversy is not about whether statistical data can be used to answer certain causal queries, but rather about whether deep neural networks trained using prediction objectives on passive streams of data can build true causal models and inference engines.
4. Provide detailed analysis on challenges faced by probing/interventional methods in the non-fixed topological order

These changes should strengthen the paper by emphasizing true causal discovery while maintaining the mechanistic insights from our original approach.

We thank the reviewer for another set of insightful and constructive points that will improve the quality of the manuscript.

Re: "Model passes generalization challenge without fixed topological order". This is interesting and a very valuable and important addition to the experiment from the original manuscript. I strongly encourage the authors to include (a more extensive version of) these experiments in the revised version.

• We totally agree with the reviewer.
• In the rebuttal, we mentioned results on the smaller 3 variable linear Gaussian SCMs. Since then, we also ran a version with 4 variables without a fixed topological order for the exact same setup as in the original manuscript. The model does pass the generalization challenge in this case too (although it takes around 1000 epochs for the model to converge). We will focus on these results in the revised version of paper. We will also perform a supplementary analysis to what extent our naive decoding setup fails without a fixed topological order.

Re: Why we chose a fixed topological order. I understand that randomizing the graph will complicate things, but this is not a good argument against that experiment being one to look at---the analyis should follow the study design and not vice versa.

• We do not think it is generally problematic to revise study designs if one realizes that the tools may be inappropriate. In our case, we realized that the particular probing and interventional tools of mechanistic interpretability we chose may not work in the more general case without a fixed topological order. We wanted to understand internal representations of the model beyond behavioral capabilities, and that is why we simplified the study design.
• However, we agree that we should look at both experiments:
  • The fixed topological order experiment lets us probe the internal workings of the model via relatively simple probing and interventional analyses.
  • The version without a fixed topological order would show that the model generalizes behaviorally in the more challenging and typical causal discovery setting (which is an important result by itself, as the reviewer rightly points out).

Re: Why “causal discovery” in the title is warranted. I disagree that the problem with a fixed topological order can still be called "causal discovery". Any global aspects of the problem only need to be memorised once, reducing the per SCM problem indeed to estimating the parameters for a fixed graph. This task is typically referred to as causal reasoning (or causal inference), and is really only non-trivial in the presence of unobserved confounding. "Causal discovery", on the other hand, has a clear connotation as "structure learning", i.e., inferring the causal diagram/graph. I therefore think that something like "Causal Reasoning through Next-Token Prediction" would be a more fitting choice of title if the main focus remains on fixed topological order.

• We thank the reviewer for these useful points about terminology and the proposed alternative title.
• The concern about the title is valid given our original emphasis on the fixed topological order experiments. However, we will restructure the paper to prominently feature the non-fixed order results as our main contribution, where the model genuinely discovers causal structure (edge directions and strengths) rather than just estimating parameters for known graphs.
• In the revised version, we will:
  • Lead with the 4-variable non-fixed order results as the primary demonstration of causal discovery
  • Present the fixed-order experiments as a further analysis that enables mechanistic interpretability
  • Clearly distinguish between true structure learning (non-fixed order) and parameter estimation (fixed order)
• With this reframing emphasizing genuine causal discovery, we believe the current title remains appropriate and accurately reflects our contribution.

I thank the authors for their response.

Re: "Model passes generalization challenge without fixed topological order"

This is interesting and a very valuable and important addition to the experiment from the original manuscript. I strongly encourage the authors to include (a more extensive version of) these experiments in the revised version.

Re: Why we chose a fixed topological order

I understand that randomizing the graph will complicate things, but this is not a good argument against that experiment being one to look at---the analyis should follow the study design and not vice versa.

Re: Why “causal discovery” in the title is warranted

I disagree that the problem with a fixed topological order can still be called "causal discovery". Any global aspects of the problem only need to be memorised once, reducing the per SCM problem indeed to estimating the parameters for a fixed graph. This task is typically referred to as causal reasoning (or causal inference), and is really only non-trivial in the presence of unobserved confounding. "Causal discovery", on the other hand, has a clear connotation as "structure learning", i.e., inferring the causal diagram/graph. I therefore think that something like "Causal Reasoning through Next-Token Prediction" would be a more fitting choice of title if the main focus remains on fixed topological order.

Re: Pearl’s claims and “implicit interventions”

I'm no expert on Pearl's precise claims, but in the generality that they are portrayed, they seem slightly absurd. Consider an unconfounded setting in which $X$ causes $Y$. Then, simple supervised learning is enough to recover the causal effect of $X$ on $Y$ since $p(y|do(x))=p(y|x)$. This example highlights that statistical objectives are sometimes sufficient to extract causal knowledge. However, if the causal direction was reversed, the same approach could fail. I thus feel the relevant question is not wheter statistical methods can sometimes work, but whether it can be guaranteed that they will work. I would be curious to hear your take / welcome further clarification. (Based on the other review's and responses, this seems to be a contentious aspect in general...)

Re: Regarding observational data and identifiability

The claim that the setting without fixed topological ordering is not identifiable from observational data is incorrect. Linear Gaussian SCMs with equal error variance are actually identifiable, see:

Peters, Jonas, and Peter Bühlmann. "Identifiability of Gaussian structural equation models with equal error variances." Biometrika 101.1 (2014): 219-228.