Identifiable Exchangeable Mechanisms for Causal Structure and Representation Learning

We thank Reviewer anYP for praising our “significant contribution bridging the fields of causal inference and representation learning”, our “straightforward and intuitively convincing approach”, and the “expected essential role” our contribution might play in CRL. We also thank you for pointing out the inconsistencies and typos, which we corrected in the new version.

Concrete examples

Thank you for pointing out the merit of concrete examples to provide intuition about IEM. We now provide intuition and real-world examples in Appendix E to demonstrate how the assumptions for the IEM framework, particularly for cause and mechanism variability, can model realistic scenarios. The intuition behind these conditions relies on the Sparse Mechanism Shift hypothesis from Perry et al., 2022, which posits that only a subset of causal mechanisms changes between environments.

Our examples show how only a subset of the causal mechanisms changes realistically across environments, including simple climate models, natural experiments in economics, and medical examples. For example, natural experiments in economics are based on selecting two sets of populations with (approximately) the same environmental and social factors, but with a difference in the relevant economic policies (e.g., a different local tax policy applies). This example belongs to mechanism variability, as the populations are the same.

We thank Reviewer Ruk1 for praising our contribution's elegance and its valuable perspective, and deeming its impact highly significant! We also appreciate the constructive and detailed feedback, which we will address below. We also addressed the notational issues (a-e) in our new submission.

Causal structure and causal variables

We thank the Reviewer’s insightful question about the relation between identifying causal structure and causal variables. As Wendong et al., 2023 noted in their Causal Component Analysis papers, the two notions are related, and required for causal representation learning. However, identifying the causal variables does not necessarily imply the identification of the causal structure. One might see the motivation for CauCA as admitting this nontrivial relationship - as an example consider that even if we learned all causal variables X, Y, Z correctly, this does not imply a single causal structure, not even the topological ordering can be determined from knowing only X, Y, and Z. However, by leveraging, e.g., the learned functional relationships, one could potentially apply the partial derivative-based approach from Reizinger et al., 2023 to extract the causal structure. In short, knowing the causal variables and the causal structure is related, though not equivalent.

Real-world insights from the duality of cause and mechanism variability for TCL

At the end of Section 3.3, we provide a medical example to illustrate under what circumstances the duality could prove useful - please also refer to the real-world examples of cause and mechanism variability we provide in Appendix E. Thus, we think that our proposed duality could open up new identifiability results, yielding to novel representation learning methods to so far not addressed realistic scenarios.

Insights for the edge-based conditions of CauCA and Liu et al., 2024

We thank for the insightful question! We compare the two assumptions in Appendix A.7, concluding that the condition of Liu et al., 2024 can be seen as “emulating” perfect interventions. That is, it is stronger than the assumption of CauCA. As the Reviewer noted, the identifiability guarantee of Liu et al., 2024 is also stronger as it additionally identifies the exogenous variables. Thus, it is unsurprising that they require a stronger assumption. The additional high-level insight IEM provides for these assumptions is that edge-based conditions can be approached from a passive (distribution shift) and an active (interventional) perspective.

We thank Reviewer pAxk for providing feedback on our manuscript and praising our submission’s contribution towards a deeper understanding of how causal discovery, Independent Component Analysis, and causal representation learning methods relate to each other. In the following, we address the Reviewer’s concerns.

Sections 3 & 4

Reviewers szvU, anYP, and Ruk1 praised our clear writing and the paper’s clear structure: “The paper is well-structured and clearly written. I appreciate a broad Discussion and future directions section.” (Reviewer szVU) “straightforward and intuitively convincing approach” (Reviewer anYP), “TCL is a very elegant and simple framework for nonlinear ICA. The analysis connecting identifiable exchangeable mechanisms to temporal contrastive learning (TCL) is insightful.” (Reviewer Ruk1). These statements respectfully disagree with the Reviewer’s criticism of the parts Section 3 being loosely connected. We devoted significant time to providing a unified structure not just to our theoretical framework but also to our writing. Notably, each subsection starts with a probabilistic model, then provides a case study by describing a well-known algorithm in that field, and ends with our new, more general results.

The first paragraph of Section 4 starts with explicitly answering the Reviewer’s question about what do the fields of causal discovery, ICA, and causal representation learning gain from our IEM framework.

• We mention the active-passive view (i.e., whether data is collected or considered as given);
• We end Section 4 with a discussion of how IEM can enable identifying components of causal systems, which might lead to more data-efficient algorithms (Sec 4.3);
• We also pinpoint potential gaps, uncovered by the duality of cause and mechanism variability (Sec 4.4);
• Furthermore, the last paragraph in Section 3 also shows how the field can benefit from IEM.

Section 4 indeed discusses OOD generalization and interventional distributions, as those are the instantiations which constitute exchangeability in causal discovery, ICA, and causal representation learning.

Specific examples

Thank you for pointing out the merit of concrete examples to provide intuition about IEM. We added intuition and real-world examples in Appendix E to demonstrate how the assumptions for the IEM framework, particularly cause and mechanism variability, can model realistic scenarios.

Causal variables

Moreover, the variable Z is referred to multiple times as a causal variable. Is there a rigorous definition of this variable, and how can a variable contain cause-effect relationships?

Causal variables are defined as the right-hand side of structural equations in Structural Equation Models (SEMs), for example, as described in Eq (12) in (Schölkopf et al., 2021). The causal variable can only contain the topological order, which, by convention (Pearl, 2009), means that for the vector variable $\mathbf{Z}$ for each $i<j$, only $Z_i$ can be a parent of $Z_j$ - i.e., only $Z_i$ can be present on the left-hand side in the structural equation expressing $Z_j$.
That is, the causal variables on their own are not enough to provide all causal information. To express the cause-effect relationships, a DAG is used, whereas the functional relationships are described by the SEM equations (the partial derivatives of the equations also give back the DAG as for non-existing edges, the partial derivatives will be zero).

We thank Reviewer B5R6 for praising our contribution’s novelty and our contribution to relax idealized assumptions in causality research. We also thank you for the detailed remarks and constructive feedback. We hope that our answers below can sufficiently address your concerns.

Empirical evaluation

Based on the reviewer’s feedback, we implemented a synthetic evaluation protocol, based on the publicly available repository of the Causal de Finetti paper (to be found at https://github.com/syguo96/Causal-de-Finetti). We focus on the continuous case, as our proofs hold for that scenario, and implement cause and mechanism variability. The continuous experiments used in the original CdF paper consider the bivariate case, which we follow to be comparable. The only change in the evaluation protocol is not evaluating the CD-NOD method, as we do not have access to a MatLab license.

We include the results of causal structure identification in our revised manuscript in Appendix F in Fig. 6. Our evaluation corroborates our Thm. 2, demonstrating that if the data is generated with a cause (Fig. 6b) or mechanism variability assumption (Fig. 6c), then causal structure identification is possible. Furthermore, the accuracy of the Causal de Finetti method does not significantly diminish compared to the original setting (Fig. 6a) and is still substantially better than the other methods (NOTEARS, DirectLinGAM, FCI, GES), showing that cause or mechanism variability is sufficient for causal discovery.

Real-world examples for cause and mechanism variability and intuition

Thank you for pointing out the merit of concrete examples to provide intuition about IEM. Beyond the synthetic experiments, we also provide real-world examples in Appendix E to demonstrate how the assumptions for the IEM framework, particularly cause and mechanism variability, can model realistic scenarios. Our examples include ones from economics, healthcare, and simple environmental models. We also discuss the intuition behind cause and mechanism variability. which relies on the Sparse Mechanism Shift hypothesis (Perry et al., 2022).

Our examples show how only a subset of the causal mechanisms changes realistically across environments, including the simple climate models, natural experiments in economics, and medical examples.

Applications to real-world data sets

As our real-world examples from the previous paragraph show, we believe that the IEM framework can model realistic medical and socioeconomic data. Thus, we desire to transfer our insights to applications for which we would first need access to such data sets.

Extensions to structured but non-exchangeable data

That is a great question! As IEM encompasses TCL, it can already model a specific type of temporal data. Based on this, we would conjecture that further extensions to other data structures is possible.

Direct comparative analysis

In our new experiments, we compare IEM to other methods from the causality literature. Theoretically, we would like to clarify that the goal of proposing IEM was to provide a unifying framework. That is, many causal discovery, traditional ICA, and causal representation learning methods are encompassed by IEM; IEM is not a particular method we can compare to. Where IEM provides new results - for example, with the relaxation of the Causal de Finetti theorem -, we provided an experimental comparison (please refer to Appendix F). We kindly ask the reviewer to clarify their point if we misunderstood it.

Dear Reviewer B5R6,

Thank you again for your constructive feedback! We have updated our submission with experimental evaluation, and described real-world examples to illustrate our framework's assumptions (particularly regarding cause and mechanism variability). Please let us know if you have any further questions.

We thank Reviewer vkuV for praising our “novel, theoretically grounded perspective that could be influential for identifying latent structures and causal mechanisms” and our new identification results.

Experimental evaluation

Based on the reviewer’s feedback, we implemented a synthetic evaluation protocol, based on the publicly available repository of the Causal de Finetti paper (to be found at https://github.com/syguo96/Causal-de-Finetti). We focus on the continuous case, as our proofs hold for that scenario, and implement cause and mechanism variability. The continuous experiments used in the original CdF paper consider the bivariate case, which we follow to be comparable. The only change in the evaluation protocol is not evaluating the CD-NOD method, as we do not have access to a MatLab license.

We include the results of causal structure identification in our revised manuscript in Appendix F in Fig. 6. Our evaluation corroborates our Thm. 2, demonstrating that if the data is generated with a cause (Fig. 6b) or mechanism variability assumption (Fig. 6c), then causal structure identification is possible. Furthermore, the accuracy of the Causal de Finetti method does not significantly diminish compared to the original setting (Fig. 6a) and is still substantially better than the other methods (NOTEARS, DirectLinGAM, FCI, GES), showing that cause or mechanism variability is sufficient for causal discovery.

Dear Reviewer vkuV,

Thank you again for your constructive feedback! We have updated our submission with experimental evaluation to illustrate our framework's practical applicability. Please let us know if you have any further questions.

We thank Reviewer szvU for acknowledging our submission's clear writing, good structure, and the extended Discussion. We address the reviewer’s questions and criticisms in the following.

Empirical evaluation

Based on the reviewer’s feedback, we implemented a synthetic evaluation protocol, based on the publicly available repository of the Causal de Finetti paper (to be found at https://github.com/syguo96/Causal-de-Finetti). We focus on the continuous case, as our proofs hold for that scenario, and implement cause and mechanism variability. The continuous experiments used in the original CdF paper consider the bivariate case, which we follow to be comparable. The only change in the evaluation protocol is not evaluating the CD-NOD method, as we do not have access to a MatLab license.

We include the results of causal structure identification in our revised manuscript in Appendix F in Fig. 6. Our evaluation corroborates our Thm. 2, demonstrating that if the data is generated with a cause (Fig. 6b) or mechanism variability assumption (Fig. 6c), then causal structure identification is possible. Furthermore, the accuracy of the Causal de Finetti method does not significantly diminish compared to the original setting (Fig. 6a) and is still substantially better than the other methods (NOTEARS, DirectLinGAM, FCI, GES), showing that cause or mechanism variability is sufficient for causal discovery.

Insights from real-world examples

Beyond the synthetic experiments, we also provide real-world examples in Appendix E to demonstrate how the assumptions for the IEM framework, particularly cause and mechanism variability, can model realistic scenarios. Our examples include ones from economics, healthcare, and simple environmental models.

IEM for hidden confounders

Does the proposed framework generalize to a setting where we allow for hidden confounds between causal variables?

This is a great question! In some settings, the answer is yes. Namely, the work by Jin & Syrgkanis, 2023 proves three theoretical results (two for different linear SEMs, and one for nonparametric SEMs) without assuming faithfulness, i.e., latent confounders can be present.

Eq 6.

Thank you for pointing out the discrepancy, the equation was indeed missing the conditioning variables from the Markov factorization, the correct first term is $p(y^i|x^i, \psi)$. We corrected this in the new version of our submission.

Dear Reviewer szvU,

Thank you again for your constructive feedback! We have updated our submission with experimental evaluation and described real-world examples to illustrate our framework's assumptions and applicability. Please let us know if you have any further questions.