Discovery of the Hidden World with Large Language Models

W1 ‘We will release an anonymous link during the discussion period.’ I will consider raising my score if the code is reasonable.

We send an anonymized link of code to the AC in a separate comment, as we are not allowed to include any links to external pages in the responses this year.

W2 The contribution of LLM in COAT.

LLM is extensively involved in the factor proposal phase as a representation assistant:

• In the factor proposal phase, LLMs are extensively leveraged to identify useful high-level factors from the given samples. The pre-trained knowledge and the reasoning ability of LLMs are heavily involved during this phase. Although COTA reduces the reliance on LLMs, the final results still depend on the capabilities of LLMs, as shown in our theories and experiments.
• Moreover, when an external interface to annotate the factors is not available, LLMs will be leveraged to parse the data. Without LLMs, we can still not obtain tabular data readily to be used for further causal structural learning.
• For the causal feedback module, it is still crucial that LLMs need to properly distinguish the already verified factors and other potential factors that may affect the partitioning of the provided groups.

Although in the current design of COAT, the causal structural learning module can be separated, the causal structural learning module plays an important role in providing various feedback, such as the existence of a hidden confounder, when extending COAT to more challenging scenarios.

W3 COAT will inherit the shortcomings of downstream causal structure learning algorithms.

It is common for all rigorous causal discovery methods to have certain assumptions and requirements on the data, while COAT may relieve the requirements:

• Well-specified variables: Classic causal discovery methods work on tabular data whose columns are meaningful variables. Here COAT extends their scope on unstructure data utilizing the power of LLMs.
• Assumptions on the data and population distribution: Assumptions are inevitable for any causal identifiability guarantee. COAT's factor proposal phase is disentangled with causal discovery, so its identifiability can hold even when one of the chosen causal discovery methods doesn't. For instance, the Apple Gastronome benchmark doesn't satisfy the linear non-Gaussian assumption, yet it still works on the factor proposal task when cooperating with LiNGAM (Appendix F).

Q1 How can we ensure that LLM does not introduce erroneous prior knowledge for reasoning?

Thanks for the good question. This question shares exactly the same spirit as the motivation of the paper. All factors proposed by LLM will be checked on data in the factor proposal phase. LLM can propose irrelevant factors (as witnessed on the META baseline), but they will be filtered out in each COAT iteration (as witnessed on the DATA baseline, the single COAT iteration, and COAT itself).

Dear Reviewer PF9K,

We just received the reply from the Program Chairs:

A possible solution might be sharing an anonymized link with the AC that can be passed further on to the reviewer of interest.

We will communicate and ask the Area Chair iWif to kindly help pass the link to you. Please let us know if you need any further information about the codes. Thank you so much!

Dear Reviewer PF9K,

Thank you for your prompt reply. We need to clarify that, there is a link shared in a comment to Area Chair iWif, since NeurIPS does not allow us to share any links to external pages with Reviewers:

4. All the texts you post (rebuttal, discussion and PDF) should not contain any links to external pages. Furthermore, they should not contain any identifying information that may violate the double-blind reviewing policy. If you were asked by the reviewers to provide code, please send an anonymized link to the AC in a separate comment (make sure the code itself and all related files and file names are also completely anonymized).

If Reviewer PF9K would like to know any details in our code, we are more than happy to provide any details. We could also just copy the codes here if needed. Please kindly understand that it is because of the NeurIPS regulations instead of the authors' intention that we could not provide a link directly in the responses.

Meanwhile, we have sent another email to Program Chairs, to inquire about the case. Please kindly let us know if you feel there is any way that could resolve your concern about the link. Since the code has already been packed up and shared in an anonymous github link, we would like to provide any information for it so long as it is allowed by the NeurIPS regulations!

We give the responses to the other questions in the follow-up comment due to the character limits.

W1.4 discussions on other suggested aspects.

We further improve the paper with the following additional discussions on the impact of other suggested aspects:

• Quality of factor annotations. The annotation could introduce an additional "error term" on the true factor values, as one can observe in Figure 4 (a) (b). The key point here is to ensure the distribution of data from annotation not violating the assumptions of causal discovery methods. For example, the "error terms" should be independent; otherwise, the faithfulness assumption required by FCI will be violated. In practice, LLMs could also use tools like API to acquire data from external resources (we try this way on the Neuropathic Benchmark in section 5.2 and also in the ENSO case study in Appendix J).
• Prompt template. Constraints on the prompt include the LLM's instruction-following ability and the length of the context window. Including more instruction, more data samples, or more background knowledge may improve the $p$ and $C_\Psi$, but would also be more challenging for LLM to handle. In practice, decomposing prompts into multiple simpler sub-tasks could alleviate this issue.

Q1 The paper addresses an important and timely problem and proposes a simple and intuitive solution, leveraging the strengths of LLMs and traditional causal discovery methods. While the experiments demonstrate the effectiveness of the proposed method over two simple baselines, stronger baselines and more ablations on prompts and factor annotations would strengthen this claim. Theoretical analysis is limited to the well-studied causal discovery aspect of the pipeline while making strong assumptions on the powerfulness of the LLMs, diversity, and faithfulness of the raw observational data, and the number of iterations being sufficiently large, seems rather unsurprising.

Here we make an overall clarification:

• This paper considers a novel task to reliably utilize the LLM's ability. In particular, it introduces COAT to propose high-level factors behind unstructured data that form a Markov blanket of target variables.
• Theoretically, we show COAT can provably identify a Markov Blanket with sufficient iterations. Our theoretical results do not assume a sufficiently strong LLM or diverse data, rather, we propose two new metrics to characterize the capability of LLMs in identifying useful high-level factors. Then, we establish the theoretical guarantee including the convergence rate based on the developed metrics. We need to clarify that these results are out of the causal discovery literature.
• For empirical evaluation, we compare the performance of COAT with the state-of-the-art baseline META in the literature, along with two additionally constructed stronger baselines DATA and DATA-CoT. The experimental results demonstrate the superiority of COAT.
• We also conduct extensive ablation studies to verify the robustness of COAT to different prompt templates, LLMs, and causal discovery algorithms.

Thanks for the insightful and constructive comments on our work. We hope our response can sufficiently address your concerns.

W1.1 "strong assumptions" of “sufficiently powerful” LLMs...

Thank you for pointing out these potentially confusing words. We revised the paper to clarify that "sufficiently powerful" and "sufficiently diverse" are not part of assumptions. These words appear in Sec 3.3, which are used to give an intuitive and concrete description of the COAT algorithm.

Our theoretical results in Sec 3.4 do not rely on those assumptions.

• Proposition 3.1 is to show why we expect new factors to satisfy Eq 6. Test on the Eq 6 is a conditional independent test. The assumption behind this is the faithfulness condition, i.e., the conditional independence reflects the d-separation of the causal graph.
• Proposition 3.3 concretely characterizes the impact of LLM's ability on factor identification, and it requires $p>0$ and $C_\Psi>0$ so that Eq 9 is about a finite number.

W1.2 Impacts of modules.

The impacts of each module in COAT can be characterized by our theoretical framework. Let $Y$ be the target variable. Before entering one certain COAT iteration, $k$ factors (denoted by $h_{[k]}(X)$ as defined on page 6 line 202) have been proposed and verified. In this iteration, the LLM receives a feedback constructed with those factors, and then we expect the LLM to propose a new factor $w_{k+1}$ such that $Y \perp w_{k+1}(X)\mid h_{[k]}(X)$ doesn't hold. In definition 3.2 (page 6 line 209), we define two measures to quantify the capabilities of LLMs in the COAT framework:

• Perception Score $p$: the probability that the LLM proposes a new factor. This can be seen as a measure of the LLM's responsiveness to the given prompts and the feedback.
• Capacity Score $C_\Psi$: the decreasing ratio of the conditional mutual information, as described in Eq 8 on page 6. This can be seen as a measure of the quality of the factors proposed by the LLM.

At each iteration, we prompt LLM to propose factors multiple times so the two measures can be estimated, as shown in Table 6 (page 25, appendix). With the help of these two metrics, we then theoretically analyze the impact of the LLM-involved modules on the number of COAT rounds, as shown by Eq 9 in proposition 3.3 (page 6).

W1.3 Convergence rate of COAT.

As suggested by the reviewer, we are happy to further improve proposition 3.3 with the result about rate of convergence (the proof is similar): with probability at least $1-\delta$, the following inequality holds:

$$\frac{I(Y;X\mid h_{\le t}(X))}{I(Y;X)} \le \left(\frac{1}{1-C_\Psi}\right)^{-tp-z_{\delta}\sqrt{tp(1-p)}}$$

That is, under the setting in proposition 3.3, with both $C_\Psi$ and $p$ being positive, COAT would converge exponentially with its number of rounds.

W2.1 Ablation of the prompt templates.

We conduct an ablation study with a different prompt template following [1]:

• We put the task description (also the format instructions) in the beginning [System] part; and we put samples in the last [Data] part of the prompt.
• The markdown grammar is replaced by blankets to represent headings, like [System], [Data], and [Groups with Y=1] ...
• 3 COAT iterations are performed, which is aligned with the original experimental setup.

COAT with changed prompt template:

We observe that COAT is robust to the choice of templates, rejects unexpected factors (zero NMB and OT), and keeps a high precision.

[1] Judging llm-as-a-judge with mt-bench and chatbot arena, NeurIPS'23.

W2.2 Baselines are too simple.

We need to clarify that, to the best of our knowledge, the META baseline is already a strong baseline, as the extensive empirical evidence about LLM-based methods in causality-related tasks [2]. If you happen to know a stronger baseline, please let us know.

Meanwhile, we additionally construct a stronger baseline with CoT based on DATA, where the LLM is prompted to "Think step by step to consider factors", and to output these factors in the same format as other methods.

It shows that LLMs with CoT can:

• be aware of high-level factors behind data (lower OT than META);
• still struggle to distinguish the desired factors in Markov Blanket (higher NMB than COAT).

[2] Causal reasoning and large language models: Opening a new frontier for causality, arXiv'23.

Dear 1cJQ, thank you so much for reading the paper and sharing your opinion. I went through your review and I think that it would be very beneficial for the authors and for their paper whether you could suggest how to tackle the issue emerging from the following sentences of yours.

"Finally, the chosen baselines are far too simple to make any strong claims on the effectiveness of COAT. Comparing against some of the methods covered in the related work section would help bolster this claim."

In particular, I kindly ask you to share which more appropriate baselines could be taken into account to improve the quality of the evaluation , and which methods covered in the related work are the most appropriate ones .

I think the authors will greatly benefit from your insights.

All the best

Thanks for your support and constructive comments on our work. We hope our response can sufficiently address your concerns.

W1 More comparisons with advanced prompting techniques such as CoT.

We construct a CoT baseline based on DATA, where the LLM is prompted to "Think step by step to consider factors", and to output the desired factors in the same format as other methods.

It shows that LLMs with CoT can:

• be aware of high-level factors behind data (lower OT than META);
• still struggle to distinguish the desired factors in Markov Blanket (higher NMB than COAT).

W2 More discussions should be provided on the hyperparameters used in COAT, such as the group size in feedback.

Thanks for the suggestion. We have revised the manuscript to include a discussion on hyperparameters:

• group size in prompt: In the COAT prompt, several samples are given and grouped by the values of the target variable. The samples in each group are randomly selected and are kept in a fixed number (like 3 samples per group). Empirically, we keep it to be 3 throughout all experiments (sometimes smaller than 3 if samples are not enough). In practice, it is mainly constrained by the LLM's context length.
• the number of clusters: When constructing feedback, we first use clustering to separate the dataset and then find the cluster where the target variable is not explained well by current factors (This is a heuristic for the problem in line 191). Empirically, we set the number of clusters to be one plus the number of current factors.

Furthermore, we conduct ablation studies of COAT with GPT-4 using different hyperparameters:

One can observe that COAT is not sensitive to these hyperparameters, and performs robustly well than the baselines under different hyperparameter setups.

W3 Model names are inconsistent. The name in Fig 4(c) is not the same as other names.

We have fixed them in the revised manuscript.

W4 GPT-4 reasoning in Fig 7(c) is unclear in meaning.

Fig 7(c) shows the result based on directly prompting LLM to reason for the causal relations among given factors. We add explanations in the figure caption now.

We give the responses to the other questions in the follow-up comment due to the character limits.

Q4 Illustrative examples for Sec 3.2 and Fig. 2.

We revised Sec 3.2 and Fig. 2 with more illustrative examples:

• Factor proposal. After observing customers' comments on apples in different ratings, LLM may propose that sweetness is a possible factor with its criterion to assign values, like more sweet is 1, more sour is -1, and 0 if not mentioned in the text. LLM can also propose other factors, like the color of the apple.
• Factor parsing. Then, another LLM goes through all comments to assign values for each factor. Therefore we get tabular data with rows for comments and columns for sweetness and color factors.
• Verification. We use the tabular data to check these new factors. We find the color is conditional independent with the rating given existing factors in the current representation (line 202, empty now), and sweetness is not. So we add sweetness to the current representation.
• Feedback. Following Sec 3.3, we find a subset of comments where the current representation cannot explain Y well. We pass these comments to step 1 to continue the next iteration.

The revised Fig.2 is included in the pdf.

Q7 Definition of MB, NMB, OT.

Formal definitions are now given in section 4 as follows:

• MB means the desired factor forming the Markov Blanket of Y.
• NMB means the undesired factor relevant to data but not in MB.
• OT means the unexpected factors irrelevant to data.

Q8 Are Table 3 and Fig. 5 in the Appendix? If so, then mention that.

Table 3 is in the appendix, which is now mentioned in the main paper. Fig.5 is on page 8.

Q9 I’m not convinced that the “realistic” benchmarks are realistic. It’s too bad I can’t gauge this from the main text.

Thanks for the important suggestion. These details in the appendix will be also seen in the main paper, as we responded in the comment about weakness.

Q11 Minor comments.

Thanks for reminding the typos. They are fixed now.

shared notations means: the notation (like $p$ and $C_\Psi$) used here are defined in Definition 3.2.

Thanks for the detailed and insightful comments on our work. We hope our response can sufficiently address your concerns.

W1. Reality of benchmarks

The choice of synthetic and realistic benchmarks is because of the evaluation purpose. Since we usually do not have access to the ground truth causal graph of the realistic benchmarks, we need to synthesize ones to effectively evaluate the performances of COAT.

Meanwhile, we do evaluate COAT in realistic data, for which we have revised our manuscript to make it clearer:

• Brain Tumor: This is an open-sourced dataset containing MRI images for brain tumor classification.
• Stock News: This dataset contains the close price of a company from 2006 to 2009 and its 804 news summary (from the New York Times).
• ENSO: This dataset contains high-dimensional information about Earth’s atmosphere with fine-grained time and space coverage from the 19th century to the early 21st century. This is a popular real-world dataset in climate science.

W2 Evaluation details.

Thank you for the suggestion! We have revised the paper to include more details of the evaluation in Sec 4.1, and please kindly let us know if you feel any additional revision could further improve the clarity:

• Purpose: We evaluate whether COAT can propose and identify a set of high-level factors belonging to the Markov Blanket of the target variable Y.
• Construction: We prepare different high-level factors: 3 parents of Y, 1 child of Y, and 1 spouse of Y. These factors form a Markov blanket of Y. In addition, we also prepare one "disturbing" factor related to Y but not a part of this blanket.
• Expectation: A good Method is expected to propose the 5 high-level factors (up to semantical meanings) and exclude the "disturbing" factor.

The key findings from the experiments are summarized below:

• COAT is more resistant to the "disturbing" factor, which is supported by the lower NMB column (number of factors out of the Markov blanket) in both table 1 (page 8) and table 5 (page 24).
• COAT filters out irrelevant factors from LLMs' prior knowledge that are not reflected by the data, which is supported by the lower OT column (number of other irrelevant factors).
• COAT robustly encourages LLM to find more expected factors through the feedback, which is supported by the lower MB column (number of factors in the Markov blanket).

W3 Lack of limitation discussion.

We indeed provided a discussion on limitations in the future work section on line 629, page 17. We revised the section title to Limitations and Future Directions to make it clearer.

Q1 Scope of the work.

The problem setting in Sec 3.1 establishes the objective of this work that reliably leverages an LLM to identify the underlying high-level factors in the Markov Blanket of a given target variable.

The inputs are merely the target variable and the unstructured data, which can be either text or images. After identifying a set of candidate high-level factors, the values of those factors can be either obtained by LLM annotations or by some external tools.

Meanwhile, identifiable causal discovery also requires certain assumptions about the data:

• Faithfuness. The empirical distribution of the data reflects the actual data-generating process.
• No selection bias. Otherwise, the faithfulness condition would be violated.
• Enough sample size. Our method involves statistical tests, the higher the better.

Q2 Use of identifiability mentioned on page 2.

The references here are classic books and survey papers about causal discovery methods with rigorous theoretical guarantees of identifiability. We use them to show the important role of identifiability, which is lacking in the current literature on using LLMs for causality-related tasks. The discussion in line 33 to 38 gives rise to our main research question: How can LLMs reliably assist in revealing the causal mechanisms behind the real world?

Q3 Description of problem setting.

The meanings of the sentence are:

• Why Y serves as a guider: Only a subset of hidden high-level factors belongs to a Markov blanket of Y, that correlates with Y. Therefore, one may use Y to guide the identification of the desired high-level factors.
• The relation actually means causal relations. In a Markov Blanket, there are three types of causal relations: Parents, Children, and Spouses. In the second type, some elements in z could be functions of Y.

We have revised the sentence to "Note that the target variable Y serves as a guider, and no prior assumption between x and Y is assumed" to avoid any potential misunderstandings.

Q5 The meaning of C and p, and the significance of Proposition 3.3?

We define the two concepts to formalize the LLMs' ability to propose useful factors:

• Perception Score $p$: the probability that the LLM proposes such a new factor. This can be seen as a measure of the LLM's responsiveness to feedback.
• Capacity Score $C_\Psi$: the decreasing ratio of the conditional mutual information, as described in Eq 8. This can be seen as a measure of the quality of the proposed factors.

The significance of Proposition 3.3:

• It shows that COAT can provably find a set of high-level factors in the Markov blanket of Y with sufficient rounds if $p>0$ and $C_\Psi >0$.
• It also characterizes the influence of LLM's ability ($p$ and $C_\Psi$) on the efficiency of COAT by Eq 9.

Q6 The details about benchmarks.

Thanks for the suggestion. These details (initially in appendix G) are added to the main paper now. A related discussion can be found in our response to W1.

Q10 & L1 Limitations.

Now the limitation part (initially a part of appendix B) is a single section. We also add discussions about faithfulness, selection bias , and sample size, as we responded. Note that COAT is relying on LLM for factor proposal, whose risk can be controlled given the faithfulness condition, as implied by our theoretical results.