A Versatile Causal Discovery Framework to Allow Causally-Related Hidden Variables

We appreciate your time dedicated to reviewing our submission and your valuable feedback. Please see our responses to your questions point-by-point below.

Q1: Are there any opportunities for the proposed method to be extended to nonlinear cases?

A1: Thank you for your insightful question. Intuitively speaking, the rank of the covariance matrix serves as a certain kind of "information bottleneck", indicating the presence and dimensionality of mediators or common causes between two sets of variables. From this perspective, the proposed method has the potential to be extended to non-linear scenarios. Such an extension would require a non-linear version of 'rank' information, possibly through the utilization of kernel representations, and we will leave it for future exploration.

Q2: If $\mathbf{A},\mathbf{B}$ are not t-separable, what does Theorem 3 imply?

A2: By the definition of t-separation, a pair of sets of variables $\mathbf{A},\mathbf{B}$ are always t-separable: at least $\mathbf{A},\mathbf{B}$ can be t-separated by $(\mathbf{A},\emptyset)$ or $(\emptyset,\mathbf{B})$. Combining this with Theorem 3, we will arrive at $\text{rank}(\Sigma_{\mathbf{A},\mathbf{B}})\leq \min(|\mathbf{A}|,|\mathbf{B}|)$ which is consistent with the rank of a general matrix. We thank the reviewer for pointing out the confusion and will add this discussion to the revision.

Q3: Definition of $\Sigma_{\mathbf{A},\mathbf{B}}$ within Theorem 3 and runtime analysis compared to methods such as FCI and LiNGAM.

A3: We appreciate the reviewer's suggestion on writing and have made the corresponding revisions. Regarding the runtime analysis, it can be found in Appendix B.16. Specifically, the proposed method requires approximately 3 hours to complete all experiments (across three random seeds and three different sample sizes). FCI takes approximately 10 minutes, whereas RCD and a LiNGAM-based method each take around two days. In comparison to FCI, as we explore a much larger hypothesis space aiming to uncover the entire underlying causal structure (while FCI only indicates the existence of latent variables), RLCD is slower than FCI, but considerably faster than LiNGAM-based methods.

We hope that your concerns/questions are well addressed and we would appreciate it if you had further feedback.

We thank the reviewer for carefully reading the submission and providing valuable comments and suggestions. Please see our point-by-point responses below.

Q1: Detailed description of steps after Phases 1-3.

A1: We thank the reviewer for the valuable suggestion. Accordingly, we have added a more detailed description of these steps in Appendix D.5.

Q2: Illustrative outputs of different algorithms to see why the proposed method is better.

A2: We appreciate your helpful suggestion. We have added illustrative examples to Appendix D.4 for comparing the outputs of various methods, together with a discussion about why the proposed method performs better. Please see the resubmission for details, and below is a summary: RLCD correctly recovers the structure asymptotically, while FCI cannot recover edges involving latent variables, and Hier. rank and GIN cannot recover edges between two observed variables and the edges from observed variables to latent variables.

Q3: Does FCI rely on the independence of latent variables?

A3: Thanks for your insightful comment. Indeed, the correctness of the FCI's result, which does not indicate whether potential latent confounders are causally related, does not rely on that assumption. (With that sentence, we intended to say that the result by FCI is not informative about the potential causal relations among latent confounders.) We thank the reviewer for your thoughtful comment and have revised the first paragraph accordingly.

Q4: F1 score and FCI's performance.

A4: An F1 score of zero indicates that none of the edges are correctly identified. While, in theory, FCI can identify edges between observed variables up to a certain type of equivalence class, our experimental results with a finite sample size reveal that it does not perform well, as you noticed. There might be two main reasons: (i) the graphs we tested have relatively few direct edges between observed variables, and (ii) FCI relies on conditional independence (CI) tests that require conditioning on multiple variables. However, statistical tests for independence conditioned on a large number of variables have been reported to have low power [1], especially with relatively small sample sizes.

[1] Spirtes, Peter. "An anytime algorithm for causal inference." International Workshop on Artificial Intelligence and Statistics. PMLR, 2001.

Q5: In Theorem 7, what is the distinction between $\mathbf{X}$ and $\mathbf{X}_{\mathcal{G}}$?

A5: As in Definition 1, $\mathbf{X_{\mathcal{G}}}$ refers to the set of all observed variables in the underlying graph $\mathcal{G}$, while $\mathbf{X}$ is a set of observed variables such that $\mathbf{X}\subseteq \mathbf{X_{\mathcal{G}}}$.

Q6: By "latent ones" do you mean latent variables?

A6: Yes. We thank the reviewer for pointing out this potential confusion and we have revised accordingly.

Q7: Why is it so important to have a unified causal discovery framework, in the sense that rank constraints are used for finding the CI skeleton? Do rank constraints provide more accurate d-separation statements than CI tests? Could it also be better to mix different types of tests?

A7: We appreciate your insightful question. Theoretically, using rank constraints in specific ways following Lemma 10 and using CI tests would arrive at the same CI skeleton (for linear causal models). Also, empirically, we found that using CI tests and rank tests for Phase 1 would arrive at almost the same results (in terms of precision and recall of edges). We further looked into the type-2 errors of these two tests and found that the type-2 errors are also almost the same. For instance, in 1000 random trials each time using 1000 random samples, where H0 assumes independence and it is not true, when alpha=0.005 both the rank test and CI test have 70 type-2 errors. Given the substantial similarity in outcomes, we opted for the use of rank constraints to enhance the overall unity and coherence of the framework.

Q8: What extra information can be ascertained relative to methods like FCI? Does the difference lie solely in that some latent variables can be identified? Will RLCD always provide more structural information than CI-based methods?

A8: We thank the reviewer for the insightful question. It does not seem straightforward for extra information to be ascertained relative to methods like FCI, because CI relations among the observed variables are just a particular feature of the join distribution. On the other hand, rank constraints, if used in the way advocated in the paper, naturally capture more information about the distribution relevant to the underlying causal structure. Specifically, based on Lemma 10, we can see that all d-separations can be captured by rank constraints as special cases, that is, rank constraints are strictly more informative than CI information regarding the underlying structure.

As for the comparison between algorithms, CI-based methods such as FCI can only infer edges between observed variables up to a partial ancestral graph, which may contain many uncertainties. In contrast, the proposed RLCD offers more comprehensive information of the underlying structure: RLCD can identify all latent variables given the assumed conditions; at the same time, it can discover all causal edges among all observed or latent variables, without introducing extra edges (please note that some of the edges with double circles by FCI may not be causal edges, but just represent the existence of possible latent confounders).

Q9: Miscellaneous comments (about the format, location of figures, etc.)

A9: We are grateful for the helpful suggestions and have already revised the paper accordingly (we changed the location of the figures, corrected the typos, removed duplicated references, etc.).

We genuinely appreciate the reviewer's effort and hope that your concerns/questions are well addressed.

Thank you for your positive feedback and we are happy that your concerns have been addressed. Please kindly note that we have made those changes and uploaded the revised submission. It would be highly appreciated if you could have a look and see whether to update your recommendation at your earliest convenience.

We are grateful to the reviewer for the insightful comments and valuable feedback. Please see our point-by-point responses to your comments and questions below.

Q1: Explanations and examples of the minimal-graph operator and the skeleton operator will be helpful.

A1: Thanks for the suggestion. As stated in the first paragraph of Section 5, an illustrative example with related explanations can be found in Appendix B.12. In light of your suggestions, we have added a brief description of the two operators in the first paragraph of Section 5 of the updated paper as well.

Q2: Corollary 1 says it degenerates to PC when there is no latent variable. I suppose that is under the assumption that the causal model is linear as assumed throughout this paper.

A2: Yes, you are totally correct, and we have made this statement explicit in the updated paper.

Let us add that while we assume linear causal models throughout this paper, the framework has the potential to handle nonlinear relations. Intuitively, the rank of the covariance matrix serves as a certain kind of "information bottleneck", indicating the presence and dimensionality of mediators or common causes between two sets of variables. From this perspective, the proposed method has the potential to be extended to non-linear scenarios. This extension might involve incorporating a non-linear version of 'rank' information, possibly through the utilization of kernel representations. We thank the reviewer for the insightful comment and we will leave it for future exploration.

We genuinely appreciate the reviewer's effort and hope that your concerns/questions are addressed.

We thank the reviewer for the time dedicated to reviewing the paper and the insightful comments. Please see our point-by-point responses below.

Q1: Using metrics such as SID and SHD.

A1: We thank the reviewer for the valuable comment. We have added new experimental results using SHD in Appendix D.1, and we find that the proposed RLCD continues to outperform all baselines across all tested settings. For example, given 10K samples generated from latent measurement models, the average SHD of RLCD is 2.9, while the runner-up is 7.4. As for SID we are still running experiments and will include the results once they are available.

Q2: Can graphical assumptions be verified before the procedure? (might be for the field in general)

A2: We appreciate your insightful comment. In practical applications, certain strategies can be employed to gain a preliminary understanding of whether the assumptions hold. One effective approach is to utilize domain knowledge of the data. For instance, if we know in advance that we have enough measurements of a latent variable that serves as pure children, then the atomic cover condition is satisfied. Nevertheless, we agree that a rigorous verification of graphical assumptions before the procedure serves as a general challenge to the field and will be a focus for future research. At the same time, as illustrated below, in some cases one can verify the conditions after the procedure.

Instead, traditional wisdom often focuses on another critical question: what implications arise if conditions are violated? We thank the reviewer for the valuable question and have provided a detailed discussion in Appendix D.2 with illustrative examples, which is summarized as follows. If a latent variable does not have enough pure children and neighbors, then Condition 1 is not satisfied. In this case, the result is still consistent with ground truth in the sense that they are rank-equivalent to each other, and at times, it can still provide valuable insights into the underlying structure. Moreover, if Condition 2 is not satisfied, then we will arrive at a self-contradictory situation, and thus we can infer that the condition is violated in hindsight.

Q3: How is the condition weaker than previous method? More detailed discussion about the difference with Huang et al. 2022. For example, what specifically allows for the identification of children of latents and mediatior latents etc as opposed to this work.

A3: A detailed discussion about how is our condition weaker than previous methods can be found in Appendix B.1 with illustrations in Figure 11.

Compared to the most related work Hier. Rank (Huang et al. 2022), our graphical conditions are not only strictly weaker but also significantly weaker. Specifically, our conditions are strictly weaker in the sense that Hier. Rank can be taken as a restrictive, special case of the proposed method by prohibiting direct edges between observed variables, which is formally stated by our Corollary 1, and prohibiting direct causal influences from measured variables to latent ones. This is illustrated in Figure 11 (b) compared to Figure 11 (d).

In practice, some measured variables may directly influence each other and they (e.g., environmental factors) may cause latent causal variables (which can cause other latent or measured variables). So our conditions are much weaker and hopefully open the gate to a much wider range of applications.

Intuitively, our ability to identify latent structures that Hier. Rank cannot lies in the more flexible and comprehensive use of rank constraints. Specifically, Hier. Rank utilizes only a portion of the rank information, denoted as $\text{rank}(\Sigma_{\mathbf{A},\mathbf{B}})$, where $\mathbf{A}\cap\mathbf{B}=\emptyset.$ In contrast, our proposed method utilizes $\text{rank}(\Sigma_{\mathbf{A},\mathbf{B}})$ with arbitrary $\mathbf{A}$ and $\mathbf{B}$, which may share some variables in common, enabling the inference of more types of t-separations. This additional graphical information empowers us, e.g., to leverage Lemma 10 to identify edges between observed variables and Thm 8 to identify partially hidden and partially observed atomic covers.

We thank the reviewer for the insightful question and have added the above discussion to the paper, at the end of Related Work.

Q4: Are Lemma 10 and Theorem 4 novel contributions?

A4: Yes, to be best of our knowledge, both of them are novel. Specifically, Theorem 4 provides a sufficient condition to determine nonadjacency, which generalizes PC’s condition to scenarios where latent variables may exist and Lemma 10 captures how partial correlation tests (as specific conditional independence tests) can be taken as a special case of rank constraints. As illustrated in the paper, they enable the proposed procedure to discover the underlying causal graph under much weaker assumptions.

We genuinely appreciate the reviewer's effort and hope that your concerns/questions are addressed.

Thank you very much for your interest! We have recently optimized our code with multi-processing, making it ten times faster. The code and data will soon be available at https://github.com/dongxinshuai/scm-identify. We appreciate your patience, hope you find it useful and look forward to your feedback!