Counterfactual Causal Inference in Natural Language with Large Language Models

Thank you for the detailled review!

Regarding your questions:

1. There indeed exists other studies on causal tasks with LLMs. Thank you for sharing some of them, we will include the missing ones in our related work. We would like to highlight the main differences with our work: (i) On causal discovery, most existing work focuses on recovering causal relationships that exist implicitly in the data (e.g. if a drug has an effect on a disease by reading patient data), this typically corresponds to tabular data, whereas in our work, we use LLMs to find causal relationships explicitly stated, from natural language sources of data. This is a different and less tackled problem. (ii) On counterfactual causal inference, we use a causal graph to divide the reasoning steps into atomic components (abduction, intervention and prediction). This is a different approach from existing work that allows us to study more deeply where the limitations of LLMs are when doing counterfactual causal inference and evaluate LLM predictions from causal parents only.

2. You are correct, we can only evaluate the ground-truth causal graphs with the synthetic data. In most real-world settings, we can only observe one factual world and do not have access to possible counterfactual ground truths (this is the fundamental problem of causal inference [1]). Due to this limitation, there are very few existing datasets that could be used for this task and the ones that exist may be solvable via commonsense reasoning and not true causal reasoning [2]. We will attempt to find or build datasets alleviating these shortcomings. We also look into LLM self-evaluation to study if LLMs can be good causal graph evaluators.

[1] Judea Pearl. Causality. 2nd ed. Cambridge university press, 2009. doi: 10.1017/CBO9780511803161.

[2] Zečević, M., Willig, M., Dhami, D. S., & Kersting, K. Causal Parrots: Large Language Models May Talk Causality But Are Not Causal. Transactions on Machine Learning Research.

Thank you for your detailled review!

Regarding your questions in the weaknesses section:

1. The main contribution of our work is the use of the causal graph to perform counterfactual causal inference. Once we obtain the causal graph, we use it to perform the three steps of counterfactual inference (abduction, intervention and prediction) separately. Compared with direct counterfactual inference, this framework allows us to study more deeply where LLMs fail at this task. Moreover, we can study the ability of LLMs to make predictions on quantities based only on their causal parents.

2. Equation 1 requires the variables X,Y and U to be connected in the causal DAG and U to be the set of exogenous factors. Regarding the connection between the variables in the introduction of Section 3 and Section 3.2: the causal variables considered are subsets of the tokens in the input context $\mathbf{C}$. These subsets correspond to tokens describing the causal parents $\mathbf{pa}(Y)$ of the quantity $Y$ computed. Thank you for pointing this confusion, we will correct it in the next version of the paper.

3. The full set of answers is provided in Figure 4. We did not find any correlations between the failure cases and the type of questions. The failed extractions seem to come from the LLMs' limitations when tasked to output formatted answers and are not linked to specific input examples, so there is no bias introduced in the evaluation.

Regarding your other questions:

1. The query $P(Y_0|C)$ is used as introductory example to show the difference with computing causal quantities. This query predicts a single token from the input context (it corresponds to a single forward pass with an LLM). However, predicting the correct distribution of a sequence of tokens $\mathbf{Y}$ is more challenging as it requires performing multiple predictions autoregressively. Biases and spurious correlations in the input context can alter performance during the prediction. A causal variable is a high-level concept and its description (or its instantiation) typically needs multiple tokens. To improve prediction, we consider selecting the smaller subset $\mathbf{pa}(Y) \in \mathbf{C}$ instead of the entire context. We use the extracted causal knowledge to this end.

2. You are correct! We do not imply that LLMs are good estimators of causal quantities but instead study if they can be made good estimators. We restrict the LLMs' contexts to causal parent variables and observe if they can perform direct causal inference in counterfactual settings. We hope our explanations have cleared up any confusion you may have had about the paper.

Thank you for your review and for the suggestions! We understand that the proposed model does not outperform other baselines. However, our aim with this work is to give insight into the current abilities and limitations of LLMs when performing counterfactual inference. By dividing every step into atomic components, we show that their primary limitation comes from prediction errors and not abduction or intervention errors. We think that these findings and our proposed directions can help drive research in the domain.

Regarding your questions:

1. The graph is generated per paragraph. We task the LLM to read read the text and return the causal relationships explicitly written in the text.

2. Graph merging is indeed a challenging problem, sometimes referred as the structural causal marginal problem [1]. As it is an open problem, we use a simple solution to be efficient. We build node embeddings to find the same variables in different graphs and use a clustering algorithm to merge them. We do no merge nodes if this leads to a cycle in the graph. We provide additional details on graph merging in Appendix A.

3. The literature on causal structure discovery and causal representation learning focuses on recovering causal relationships that exist implicitly in the data (e.g. if a drug has an effect on a disease by reading patient data). This is a very challenging problem that typically involves performing conditional independence tests between the variables [2]. Our work focuses on latter stages in the causal inference pipeline: we use LLMs to find causal relationships explicitly stated in natural language sources and study counterfactual inference from the built causal graphs.

4. We prompt the LLM to propose causal variables not explicitly stated in the data but can plausibly affect the observed variables. This step relies on commonsense causal reasoning, an task LLMs have been shown to be good at [3,4]. These variables do not have an instantiated value as they are unobserved. In the abduction step, we retrieve their values during the abduction step via inference in the anticausal direction (i.e. by computing P(U|children(U)). The found values are then used during intervention and prediction. In cladder, all the variables have an instantiation (even the ones named "confounder"), so there are no hidden variables.

5. This is a good point! We decided to discard wrongly formatted answers in Table 1 to ease interpretation. This table compares LLMs' abilities to perform counterfactual inference. Our method introduces additional sources of errors that are not present with direct prompting of chain-of-thought, including them would have made harder to compare the LLMs on the reasoning parts. We understand that it means that we are not directly comparing our method but we favoured this choice as our aim is to better understand the limitations of LLMs in counterfactual reasoning. We also include the full results with parsing failures in Figure 4. We will work on reducing parsing errors to eliminate this problem.

6. The self-evaluation is used for assessing if the LLMs can detect wrong or unconfident answers. If close to ground-truth, self-evaluation could be used in real-world cases where the ground-truth is not available. We will add an in-depth comparison between accuracy and self-evaluation in the next version of the paper to improve this section.

[1] Gresele, L., Von Kügelgen, J., Kübler, J., Kirschbaum, E., Schölkopf, B., & Janzing, D. (2022, June). Causal inference through the structural causal marginal problem. In International Conference on Machine Learning (pp. 7793-7824). PMLR.

[2] Judea Pearl. Causality. 2nd ed. Cambridge university press, 2009. doi: 10.1017/CBO9780511803161.

[3] Kıcıman, E., Ness, R., Sharma, A., & Tan, C. (2023). Causal reasoning and large language models: Opening a new frontier for causality. arXiv preprint arXiv:2305.00050.

[4] Zhang, J., Zhang, H., Su, W., & Roth, D. (2022, June). Rock: Causal inference principles for reasoning about commonsense causality. In International Conference on Machine Learning (pp. 26750-26771). PMLR.

Thank you for your detailed review and for the advice on how to improve the paper!

Regarding your questions:

1. Yes, this is a direction we are interested in pursuing. We have conducted separate experiments to quantify the quality of the LLM extractions. In this set of experiments, we prioritized getting a fully automated quality assessment. To that end, we considered extracting wiki concepts from the causal variables described in the nodes and the edges of our causal graphs and computed the Jaccard similarity between them. Analyzing nearly 170 causal graphs, we found that the Jaccard similarity was zero for about 55% of the cases, while for only ~10% of the cases, we observed a perfect match. When considering the results, we must note that the Jaccard similarity does not consider the semantic proximity of the wiki concepts: we defer such an evaluation to future work. This evaluation shows that different kind of information is retained in the nodes encoding the causal variables, and the edges informing about the causal relationships. Future work will focus on aligning them better. Furthermore, we will work toward building and releasing a golden dataset against which we could measure the quality of the extracted causal relationships against human-annotated ground truth for causal graphs extracted from media news. To complement this view with a counterfactual perspective, we would like to underscore that conducting counterfactual causal inference on real-world data is challenging. In most settings, we can only observe one factual world and do not have access to possible counterfactual ground truths (this is the fundamental problem of causal inference [1]). Due to this limitation, there are very few existing datasets that could be used for this task and the ones that exist may be solvable via commonsense reasoning and not true causal reasoning [2]. Nevertheless, we are already looking into ways to overcome this challenge. Part of our future research will focus on finding or building datasets to alleviate these shortcomings.

2. You raise an important point: built synthetic causal structures can differ from real-world causal structures. We highlight this difference by comparing the causal graphs in cladder with the ones extracted from real-world news articles. The main differences observed are the size of the causal structures and the account of possible missing variables: real-world causal structures are bigger and more complex than synthetic ones and can contain hidden confounders.

3. We try different prompting strategies to ensure we are not obtaining suboptimal results due to the prompt. Our approach directly aims to reduce the risks of hallucinations during counterfactual inference by reducing the context size to the direct causal parents of the quantity to compute, thereby reducing the number of context tokens generated and the risks of hallucination [3]. Furthermore, we are now exploring additional refinements that would help us to (a) validate and correct the data types assigned to the causal variables and (b) leverage techniques that would allow numerically confirm the causal relationship between them and their direction. This would solve an open research question and a gap that currently could not be solved with the LLMs alone [4].

4. Yes, we provide an example in Section 4.4. In this example, the inference to perform is a logical OR between two parent variables. This information is directly extracted from the input text and included in the prompt. However, GPT-4o interprets it as logical AND (we know this from the chain-of-thought explanation given by the model). The model returns a wrong answer due to an error during the prediction step, while the abduction and intervention steps were correct. Therefore, the error is a failure to process a logical entailment and is not due to a causal reasoning error.

[1] Judea Pearl. Causality. 2nd ed. Cambridge university press, 2009. doi: 10.1017/CBO9780511803161.

[2] Zečević, M., Willig, M., Dhami, D. S., & Kersting, K. Causal Parrots: Large Language Models May Talk Causality But Are Not Causal. Transactions on Machine Learning Research.

[3] Fadeeva, E., Vashurin, R., Tsvigun, A., Vazhentsev, A., Petrakov, S., Fedyanin, K., ... & Shelmanov, A. (2023). LM-polygraph: Uncertainty estimation for language models. arXiv preprint arXiv:2311.07383.

[4] Joshi, N., Saparov, A., Wang, Y., & He, H. (2024). LLMs Are Prone to Fallacies in Causal Inference. arXiv preprint arXiv:2406.12158.