Compositional Causal Reasoning Evaluation in Language Models

We thank the reviewer for acknowledging the clarity and value of our framework, as well as the importance of validity and consistency. We appreciate the suggestion that this framework has potential utility for AI interpretability and safety, areas where we see increasing crossover with causal theory. We address additional comments below.

1. Comment: Small-scale experiments & presenting results more clearly.
   • Revision 1: To address this comment, we have extended the CandyParty experiments to include results for o1 with and without CoT. See revised Figures 5 and 7 at https://tinyurl.com/bdddu2vx under /figures_and_tables/updated_figures/ and new Fig 9 (CCTs as a visualization tool for error analysis): https://tinyurl.com/yc5cf9ev
   • Revision 2: As further emphasized by our new results and discussion for o1, we evaluated on a task structure that we anticipated to elicit a spectrum of CCR behavior as a means of highlighting the validity and utility of our framework (not as a benchmarking study). The updated discussion emphasizes the implications of this range of behavior, e.g.: “Results for task T revealed three taxonomically distinct error patterns: VC (o1), near-VI (GPT-4o with CoT), and II (all remaining models). Only o1 placed mass in the VC reasoning quadrant (Fig. 5)... However, internal inconsistency made GPT-4o with CoT a near-VI reasoner: 75.1% of all estimates were near-valid, mean external validity on local quantities and compositions exceeded 77% and 95%, respectively, and yet failure to correctly infer $\mathrm{PNS}_{XY}$ revealed internally inconsistent reasoning.” See our response to Reviewer doGN for further discussion of VI reasoning and error analysis for GPT-4o.
   • Revision 3: To facilitate future benchmarking works, we release open source Python code for automated CCR task generation that can be evaluated according to Algorithm 1 for the PNS, where the user can scale the graphical complexity of the causal DAG and choose from 4 themes: candy party, flu vaccination, football games, and flower gardening. The code can be viewed here, and further unit tests will be available at camera ready: https://tinyurl.com/5be5ehyp and https://tinyurl.com/4um56dab
2. Comment: Sensitivity to prompt engineering.
   • Indeed, the potential for sensitivity to prompt engineering was the motivation for comparing CoT and non-CoT prompting in our work. Results from GPT-4o in particular demonstrate such sensitivity. While it is out of scope to consider additional prompting strategies in this work, we agree that this would be an important future direction, especially for large-scale CCR benchmarking.
3. Comment: Taxonomy of reasoners.
   • The reviewer asks whether these reasoning categories define a “single large language model’s overarching behavior, or does it represent the performance of such a model on particular sets of tasks?” In the present work, we limit such labels to task-dependent behavior, though in the future more extensive benchmarking might (cautiously) enable broader claims of overarching behavior. In Section 4 we cite the prior notion of a task-metric-model triple and state that “We tether evaluation to a task-metric-model triple”, clarifying that “success on $\langle \varphi, \mathcal{M}, \mathcal{Q} \rangle$ does not imply success on $\langle \varphi, \mathcal{M}, \mathcal{Q'} \rangle$, $\langle \varphi, \mathcal{M'}, \mathcal{Q'} \rangle$, $\langle \varphi', \mathcal{M}, \mathcal{Q'} \rangle$, etc.” and “success on CCR tasks such as this is necessary but not sufficient for demonstrating that LMs can reason.”
   • Revision 1: To further clarify, we have added the following to Section 6.3 Limitations & Future Directions: “This proof-of-concept limits application of our reasoning taxonomy to task-dependent observations on model behavior. However, extensive benchmarking could potentially reveal model behaviors that persist across categories of tasks. It remains to be seen if and how different LM architectures might display significant tendencies toward certain categorizations (VC, VI, IC, II).”
   • Revision 2: Please see our response to Reviewer doGN for further clarification on the VI reasoning category.

We thank the reviewer for their thorough feedback and for acknowledging the soundness of our approach. We hope that the comments below address all concerns.

1. “Outliers” for GPT-4o (CoT).
   • Results for $PNS_{XY}$ and $PNS_{DY}$ are not extreme outliers, though Fig 7 might imply such (we update to clarify this possible misinterpretation in-text). Please see original Figs F.7, F10: though the model did not achieve external validity according to our pre-defined threshold, predicted PNS distributions were not extremely far from truth.
   • Revision 1: Added to Sec 6.2 (new figures https://tinyurl.com/bdddu2vx under figures_and_tables/new_figures/error_analysis_figures/): "Preliminary analyses indicate that GPT-4o with CoT displayed several kinds of faulty reasoning for both $PNS_{XY}$ (Figs. F.6, F.7) and $PNS_{DY}$ (Figs. F.8, F.9). These include (1) failure to correctly extract causal relations (e.g., true causal parents are missing from the stated parent set; Fig. F.9), (2) incorrect logic despite correct causal relation extraction (e.g., a clause will not logically follow from the preceding clause; Fig. F.8), and (3) a truncated reasoning process, where the model expresses its logic up to a certain point in the underlying causal graph but ignores the remainder of the graph without explanation (e.g., Fig. F.7)."
2. CoT formulation.
   • The reviewer notes that “proper working of the evaluation framework can only be shown within a valid setup.” This is true, and new results for o1 (which show complete VC reasoning) verifies that this setup is valid and achievable by a sufficiently capable model. See response to Xrrd.
   • Potential influence of prompt engineering indeed motivated our comparison of CoT and non-CoT prompting. Our results for GPT-4o do indicate sensitivity to this. Though it is out-of-scope to perform additional prompt engineering experiments, this is an important area for future inquiry.
   • Revision: Added under Limitations: "Future work should explore sensitivity to prompt engineering, including alternative formulations of CoT."
3. Is VI reasoning a valid category?
   • Revision: As the VI reasoner is a counterintuitive yet informative category, we added the following to Sec 6.2: "This framework not only reveals if the reasoner is wrong, but offers insight into how it is wrong. We take VI reasoning as a stark and perhaps counterintuitive example, where external validity alone (as typically measured in reasoning evaluation) provides an incomplete picture. Consider GPT-4o with CoT: the model was near-valid to valid on the majority of quantities, implying some measure of success. Upon closer inspection, it was (near-)valid on most local estimates (e.g., $\widehat{\mathrm{PNS}}\_{CD}$) and their products (e.g., $\widehat{\mathrm{PNS}}\_{XC}\widehat{\mathrm{PNS}}\_{CY}$) but not for $\widehat{\mathrm{PNS}}\_{XY}$. While it might appear reasonable to assume that invalid global estimates stem from poor local reasoning or relation extraction abilities, the potential for alternative explanations is a key insight of our work. Here, GPT-4o displayed compositional inconsistency despite local relation extraction: it was more performant when asked directly about relatively proximal local relationships than when asked about distal cause-effect pairs where the answer relied on correct compositional reasoning over multiple direct and indirect relationships. VI reasoners highlight this potential failure mode: the reasoner does not learn to compose over multiple strands of logic yet can answer local questions when asked directly (akin to a student passing a math quiz simply by memorizing specific answers, instead of synthesizing; see incorrect logic despite relation extraction in new Fig F.8). Uncovering this behavior is precisely the benefit of measuring both external validity and internal consistency."
4. Proof – B.20 $\to$ B.21.
   • We thank the reviewer for reading our proof line-by-line! This is due to monotonicity: $P(y′\_{x}) = 0$ and $P(z′\_{y}) = 0$ (outcome cannot be false when treatment is true). So, $P(y′\_{x}, y′\_{x′} )P(z\_{y′}, z′\_{y′})$ and $P(y\_{x},y\_{x′})P(z_{y},z′\_{y})$ zero out.
   • Revision: Added to line in proof: “By monotonicity.”
5. Potential outcomes vs graphical models.
   • This is not a problem, and going between the two is common (as they each provide their own usefulness). Note: “Formally, the two frameworks are logically equivalent; a theorem in one is a theorem in the other, and every assumption in one can be translated into an equivalent assumption in the other.” — Judea Pearl
6. Complexity analysis. We thank the reviewer for their attention and agree that this analysis can be improved.
   • Revision: Notebook validating the improved complexity analysis: https://tinyurl.com/3tnut7tj
7. Quotation marks in Fig F.2. This was a copy-paste error. Updated figure: https://tinyurl.com/nhctxkat

We thank the reviewer for their attention to the correctness of our proofs, the powerful abstraction provided by CCTs, and the gap in the literature that we attempt to close: the explicit and systematic evaluation of compositional consistency in causal reasoning. We address questions below.

1. Low problem complexity.
   • As further emphasized by our new results and discussion for o1 (see response to Reviewer Xrrd), we chose a relatively low complexity SCM to evaluate a range of models that we anticipated to reflect a spectrum of CCR behavior. The fact that our revised experiments demonstrate partial success (GPT-4o CoT), full success (o1), and failure (all other models) is important, as it validates that our framework is both correctly formulated and capable of disentangling distinct error patterns.
   • We have released automated task design code that enables the user to generate CCR prompts based on casual graphs of scaling graphical complexity. An important future direction will be to benchmark on CCR tasks that simulate real-world problems (e.g., in healthcare), which the "vaccination" setting in our new task generation code is a first step toward.
2. Parametric assumptions / estimands (ATE w. linearity and PNS w. monotonicity). Though we instantiate our framework under select conditions in Sec 5, the framework presented in Sec 4 is highly flexible and is not limited to any estimand nor parametric assumptions. As stated in Sec 4.2, “Sec 5 suggests one possible procedure for assessing compositional consistency in causal reasoning (Alg. 1).” Under Limitations, we state “As this work only considers the ATE and PNS under Theorem 5.1, extensions could consider other estimands and compositional forms.”
   • We provide Example 3.2 as one hint for future work: the decomposition of the TE under linearity. However, Pearl introduced a nonparametric mediation formula similar to Example 3.2 that could just as easily be exploited for CCR. Any nonparametric formula will be particularly suited to “realistic/real-world” task design. The possibilities are only limited by the extent of known compositional formulae in causal inference.
   • Revision: Under Limitations: “Potential extensions could implement CCR tasks based on Example 3.2 or for Pearl’s nonparametric mediation formula, which extends Example 3.2 to nonlinear settings.”
3. Unobserved confounding. Sec 4 does not require that all confounders are observed, rather that valid causal inference can be performed. Sec 5 assumes this only to simplify task design.
   • Revision: "Future work could accommodate latent confounding by exploiting known identifiability results for such cases." (See Pearl for many related works.)
4. Inference hyperparameters. Added to a new table in Appendix F.1.
5. Elaboration on CCTs. We thank the reviewer for acknowledging the utility of CCTs. We direct them to Figure D.1 of the original submission for more examples.
   • Revision 1: Added to discussion w/ link to Fig D.1: "This allows for evaluation on arbitrarily complex DAGs with cutpoints as if they were simply directed chains (Fig. D.1)... As demonstrated in Sec 5.2, CCTs can be leveraged as a design tool for formulating reasoning tasks. As illustrated in Sec 6.2, they can also serve as an interpretable, intuitive visualization tool for graphically representing CCR successes and failures (Fig. 9)."
   • Revision 2: New Fig 9 -- CCTs as a visualization tool for error analysis: https://tinyurl.com/yc5cf9ev
   • Revision 3: We provide open source code to generate CCTs from random DAGs under sufficient conditions, along with our new automated task generation method: https://tinyurl.com/5be5ehyp and https://tinyurl.com/4um56dab.
6. Mechanisms underlying failures / CoT and internal consistency. Mechanistic explanations (e.g., limitations in attention mechanisms), additional prompting strategies, architectural modifications, or novel fine-tuning regimes (e.g., Huyuk et al 2025) are all potentially important areas for future work.
   • Revision: While most of the above is out of scope for this work, we expand our error analysis for further insight. See our response to Reviewer doGN. Examples: https://tinyurl.com/bdddu2vx under /figures_and_tables/new_figures/error_analysis_figures/.
7. Data and code release. All code provided in the repo above will be public, as will code used for simulations in Appendices C/E. Notably, this includes our automated task generation codebase, which enables customized prompt dataset generation. Code and data related to model inference will at minimum be available upon reasonable request after publication.
8. Significance tests: CoT vs non-CoT. See https://tinyurl.com/yck6bd28 for results. All error distributions for CoT were significantly different than non-CoT except for o1 on PNS(X,Y) (Wilcoxon, alpha = 0.05).

We thank the reviewer for their useful feedback. We hope that the following addresses your concerns.

1. Limited results / no dataset.
   • Revision 1: CandyParty experiments now include results for o1 with and without CoT. For o1, CCR reasoning was “complete,” enriching our discussion by (1) validating the correct formulation of the task and (2) empirically demonstrating VC reasoning. Revised Figs 5/7 are at https://tinyurl.com/bdddu2vx under /figures_and_tables/updated_figures/.
   • Revision 2: “Results for task T revealed three taxonomically distinct error patterns: VC (o1), near-VI (GPT-4o with CoT), and II (all remaining models). Only o1 placed mass in the VC reasoning quadrant (Fig. 5)... However, internal inconsistency made GPT-4o with CoT a near-VI reasoner: 75.1% of all estimates were near-valid,... and yet failure to correctly infer $\mathrm{PNS}_{XY}$ revealed internally inconsistent reasoning.” See our response to Reviewer doGN for further discussion and error analysis for GPT-4o. We hope these updates clear confusion on Fig 5, where each quadrant represents a reasoning type.
   • Revision 3: To facilitate future benchmarking, we offer open source Python code for automated task generation for Algorithm 1 / Theorem 1. The user can scale the graphical complexity and choose from 4 themes: candy party, flu vaccination, football, and flower gardening. Further unit tests will be available at camera ready: https://tinyurl.com/5be5ehyp and https://tinyurl.com/4um56dab
2. Figure 7 / Task not challenging for GPT-4o with CoT.
   • As noted, this work is not meant as a benchmarking study but a validation of our framework. As further emphasized by our new results and discussion for o1, we chose a low complexity SCM to evaluate a range of models that we anticipated to reflect a spectrum of CCR behavior. The fact that our experiments demonstrate partial success (GPT-4o CoT), full success (o1), and failure (all other models) is important, as it validates that our framework is both correctly formulated and capable of disentangling distinct error patterns.
   • Revision: Added to concluding remarks: “Open source and smaller models did notably worse on this task than GPT-4o (CoT) and o1, suggesting that even low-complexity CCR is an area on which significant progress can still be made for an important class of LMs (smaller and/or open).”
3. Motivation, usefulness, comparisons.
   • Novelty: (1) compositional consistency, (2) CCR evaluation in LMs, (3) metrics, (4) taxonomy of distinct error patterns, applicable beyond the causal setting, (5) practical algorithm.
   • Revision 1: Added: “Comparison to Prior Methods: See Table A.1 for an extensive comparison. To further clarify our contributions, we compare to the CLADDER benchmark for formal causal reasoning (Jin et al., 2023). Most prior causal reasoning evaluation centers on commonsense knowledge and relation extraction (Table A.1). Like CLADDER, our framework and task generator (1) probe reasoning at the associational, interventional, and counterfactual levels; (2) apply causal inference methods; and (3) evaluate causal relation extraction. Our method notably departs from CLADDER in the following ways: (1) we evaluate compositional reasoning, another dimension of causal reasoning that has received little attention in LMs; (2) while CLADDER uses simple graphs (3-4 nodes), our task generator allows the user to scale graphical complexity; (3) we introduce new metrics for internal compositional consistency of causal reasoning; and (3) we do not require ground truth quantities, as the user can opt to evaluate on internal consistency only.”
   • Revision 2: See Table A.1: https://tinyurl.com/4ctw6bfx.
4. Required access to the true graph.
   • This is standard in evaluation (see Corr2Cause, CLADDER, or arXiv:2402.11821). Evaluation innately requires reference values. The identifiability conditions for valid causal inference are extensively discussed in prior works. In general, these require at least partial structural knowledge (e.g., knowledge of confounders that block backdoor paths; of a valid instrumental variable when latent confounding is present; of mediators for the frontdoor adjustment; etc).
   • This work is explicitly rooted in causal graphical modeling (rather than potential outcomes). We explicitly measure reasoning over graphical components (Theorem 1). Graphical reasoning is an important area itself. Thus, the causal graph is centrally relevant.
   • Revision: Added to Sec 5.2: “When ground truth values are unavailable, Alg.1 can be run for internal consistency only.”
5. “Metrics / quantities should be accompanied by examples.”
   • Each definition is accompanied by concrete examples related to Example 3.2, Fig 2, Fig 3 (e.g. "internally consistent reasoning entails that $\theta($..."). If the reviewer has specific confusions, we are happy to clarify further.