Unifying Causal Representation Learning with the Invariance Principle

Thank you for the positive feedback, and we will answer your questions as follows.

 

Reply to weaknesses

Apply the invariance framework to diverse data types

• The primary motivation of our work is indeed theoretical; however, prior works (shown to be special cases of our framework) have been applied to diverse data types, such as video $`7, 12, 13, 14`$ , image $`3, 4, 5, 6, 8`$ and text $`4, 5`$ . We did not replicate these results because our work is a strict generalization of individual special cases.
• Instead, our experiments focus on new capabilities enabled by our framework. Most notably, we have shown the first application of causal representation learning on a real-world ecological problem (and data – Section 5.1), which was posed as an open challenge for CRL in $`15`$ as existing approaches are not directly applicable. The Nintervention experiment on numerical data (Section 5.2) successfully removed the causal ingredient in existing interventional approaches and demonstrated that the data symmetry employed by these methods does not have to be causal (e.g., the result of an intervention).
• In general, the proposed invariance framework is not limited to any certain data types, and we agree with the reviewer that applying the invariance approach to various data types would be exciting for future research. We believe that the flexibility of our framework will be broadly beneficial for applications in CRL, which are a known major challenge in the field, and we have already demonstrated this with our case study in ecology (Section 5.1).

 

Reply to questions

How to choose the invariance regularizer $\\lambda_{\text{INV}}$?

• We suggest selecting the invariance regularizer $\\lambda_{\text{INV}}$ by minimizing the treatment effect bias on a small (ideally heterogeneous) validation set, as elaborated in Lines 476-177.
• In Figure 2, we highlighted three models among all trained models (varying seed and $\\lambda\_{\\text{INV}}$): the one minimizing the Empirical Risk (yellow star), the Invariance Loss (violet star), and the Treatment Effect (green star) on the validation set.
• Among these criteria, minimizing the Treatment Effect on the validation set better approximates the true Treatment Effect for the entire population (i.e., the minimum of the blue function).
• Investigating more principled selection criteria and incorporating uncertainty quantification could be interesting directions for future work.

Thank you for your thoughtful comments and constructive feedback. We hope that our responses below adequately address your concerns.

 

Reply to weaknesses

Explicit form of the invariance property $\\iota$ for special cases of our framework

• Thank you for your suggestion. The new Table 1 has provided a list of concrete examples and explicit forms of the underlying invariance principle of individual special cases.

• As shown by Table 1, multiple kinds of invariance are exhibited in multi-environment CRL. For example, when performing single-node intervention (row 3 in Table 1), the marginal distribution of non-descendants remains invariant.

• Here, the invariance property $\iota$ is a function that maps the invariant partition $\mathbf{z}_A$ to its probability density function $p_{\mathbf{z}_A}$, i.e., $\mathbf{z}_A \mapsto p_{\mathbf{z}_A}$.

• Correspondingly, the codomain $\\mathcal{M}$ is the set of valid probability density functions, formalized as: \ $\\{ p\_{\\mathbf{z}\_A} : \\mathcal{Z}\_A \\to \\mathbb{R}\_{\>0} \\mid p\_{\\mathbf{z}\_A} \\text{ is a valid pdf} \\}.\\$

• In most cases, the codomain $\mathcal{M}$ of $\iota$ is implicitly determined by the choice of the equivalence relation $\sim_{\iota}$, as it is inherently defined as the quotient space corresponding to this relation (see Definition 2.1), and therefore does not require explicit specification.

Space issue and a better main-appendix split

• We indeed faced challenges in deciding what content to include in the main text due to the extensive volume of the paper.
• The main reasons for submitting to ICLR are: (1) this paper focuses on learning representations, making it a strong fit for ICLR, and (2) we hope that the research idea will reach a broader audience highlighted through the conference, as many works we unified have been published at ICLR $`4, 5, 12, 13, 14`$ . (3) While our contributions are primarily theoretical, they have immediate practical implications, which we have demonstrated with our case study in experimental ecology.
• We have revised the split between the main text and the appendix, adding more concrete mathematical explanations to the main text (Table 1) to make the paper more convincing. Please refer to “Modifications in the manuscript”(under the general response) and the updated manuscript for further details. If you have other feedback on improving the presentation, we will happily take them into account.

Thank you for your valuable feedback; we hope our responses provide the necessary clarification.

 

Reply to weaknesses

Clarifications on contributions 2 and 3:

• It is possible that these two contributions may have been overstated, and we agree that this is not one of the most important points of the paper in any case. We are not aware of existing literature that extensively examines the transition between different levels of identification granularity (contribution 2), or the distinction between assumptions for variable identifiability and graph identifiability (contribution 3).
• We would be very grateful if you could suggest relevant theoretical results or similar works that we could cite to provide a more complete context in our paper. What we want to say is that any CRL algorithm is capable of achieving any of the identifiability definitions depending on the assumptions we highlight. For example, single fat-hand intervention (which was shown to provide block-identifiability under non-parametric mixing $`3, 4, 5`$ ) and polynomial mixing lead to block-affine identifiability (implied by Proposition C.2); or paired perfect intervention per node, leading to element-wise identifiability under non-parametric mixing $`2`$ , can be refined to affine-identifiability by employing polynomial mixing (again, implied by Proposition C.2). All these individual cases do not need to be proven one-by-one and to the best of our knowledge were not included in neither the original nor follow-up papers, with our result we can tell exactly which identifiability result can be achieved.

Explicit examples of CRL categories:

• The new Table 1 provides concrete examples of special cases of our theory and summarizes their underlying invariance (see “Concrete examples of our theory” under the general response for more details). We have included this table in the revised version, and hopefully, this will help with understanding both Definition 2.1 and 2.2. If you have further suggestions regarding this aspect, we will be happy to incorporate them as well.

Why are previous CRL approaches not directly applicable to ISTAnt?

• We would like to clarify that previous assumptions in CRL lack the flexibility needed for ISTAnt. In this setting, we consider a concrete task to identify the factual outcome $Y$, e.g., whether ants are grooming or not.
• Since interventions on the behavior of live ants cannot be performed, multi-environment approaches (interventional CRL) are not applicable. Additionally, contrastive multi-view approaches lack negative samples (we need pairs of samples with the same or different $Y$, while we can assume that nearby frames will have similar $Y$, we cannot assume that far away frames will have different $Y$); we cannot take actions over time as required by temporal CRL; and we only have one task which is misaligned with the assumption in multi-task CRL.

Why does V-REx $`1`$ (the method used for the ISTAnt case study) fit under the invariance framework?

• For ISTAnt, we exploit an invariance we can assume on the causal mechanism among different experiment settings. Although the training set is biased, called out as a challenge in $`15`$ , we can naturally incorporate this constraint to improve the estimate. The proof in the appendix (see Appendix D.6, (f)) derives how the domain generalization approach used in the ISTAnt experiment (i.e., V-REx $`1`$ ) falls under our invariance framework. This proof is also referenced in the main text (Page 8, Line 463).

 

Reply to questions

Notations in Constraint 3.2

• Thank you for your question. Here, $ H(\cdot) $ denotes the differential entropy of the ground truth latent distribution $ p_{\mathbf{z}_{A_i}} $, where $ A_i \subset $N$ $ is the index subset for which the invariance property $ \iota_i $ holds. We have updated the manuscript accordingly to provide additional clarity.

Explanations for “data pockets”

• We apologize for the confusion. By "data pocket," we refer to a set of observables generated from latent variables that share certain invariance properties, such as the observables $\\mathbf{x}^1, \\mathbf{x}^2$ in the examples from Table 1. This can be formally defined using $\\mathbf{x}\_{V\_i}$ as introduced in Assumption 2.1. We have incorporated this clarification in the revised manuscript; see Assumption 2.1.

We thank the reviewer for the positive feedback and will address the concerns individually.

 

Reply to weaknesses

More mathematical explanations in Section 4:

Thank you very much for suggesting that we include the explicit mathematical explanations from Appendix D in the main text.

• In the current manuscript, we opted for more plain text primarily due to space constraints. Additionally, we believed that a simpler, plain-text style might be easier for readers to digest, potentially reaching a broader audience.
• However, we fully agree that presenting the related works in a more mathematical manner would enhance the comprehensiveness and rigor of the paper. Accordingly, we have added Table 1 to provide more mathematical details and to serve as concrete examples for the related works in Section 4.

What new insights does the invariance framework lead to?

Yes, there are many new insights that this framework enables, some of which have already been explored in the paper, and we can further highlight them if the reviewer suggests we do.

• Theoretical derivatives explored in the paper

  • Directly following the invariance framework, we have proven identifiability with single-node imperfect intervention per node under the nonparametric setting (Corollary D.1), which was an open conjecture from [2].

• Methodological similarities across areas

  • The proposed invariance framework sheds light on connecting causal representation learning with various settings, including more surprising ones like domain generalization and multi-task learning. For example, for the ISTAnt experiment (Section 5.1), we have derived the V-REx[1] loss from our framework. See Appendix D.6, (f). The lack of applicability of CRL methods on ISTAnt was called out explicitly in [15].
  • From the causality perspective, CRL methods are often arranged in Pearl’s causal hierarchy(e.g., “counterfactual” $3, 4, 5$ or “interventional”[2, 10, 11]). Finding a methodological equivalence goes against the common folklore, and we think is very surprising.

• Other possible links we did not explicitly explore:

  • Our theory does not assume that the type of equivalence relation is the same for all latents. While we believe this may be useful in applications, we did not have a real-world problem in mind, and so we did not explore this empirically.

• Potential for new CRL approaches and future work

  • We strongly believe that this paper will inspire new work in causal representation learning. However, we hope that it will also shift the focus away from proving custom identification results based on assumption classes (e.g., one or more interventions per node or N views with overlapping latents), which implicitly describe equivalence relations. Instead, we hope our work enables us to work backward from concrete invariances that can be assumed in applications, as we did in our ISTAnt case study (Section 5.1). Overall, we hope our paper will enable a flurry of new results that are specific to and inspired by real-world problems, which current CRL research largely lacks.

[1] David Krueger, Ethan Caballero, Joern-Henrik Jacobsen, Amy Zhang, Jonathan Binas, Dinghuai Zhang, Remi Le Priol, and Aaron Courville. Out-of-distribution generalization via risk extrapola- tion (rex). In International conference on machine learning, pages 5815–5826. PMLR, 2021

[2] Julius von Kügelgen, Michel Besserve, Liang Wendong, Luigi Gresele, Armin Keki´c, Elias Barein- boim, David Blei, and Bernhard Schölkopf. Nonparametric identifiability of causal represen- tations from unknown interventions. Advances in Neural Information Processing Systems, 36, 2024.

[3] Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34:16451–16467, 2021.

[4] Dingling Yao, Danru Xu, Sebastien Lachapelle, Sara Magliacane, Perouz Taslakian, Georg Martius, Julius von Kügelgen, and Francesco Locatello. Multi-view causal representation learning with partial observability. In The Twelfth International Conference on Learning Representations, 2023.

[5] Imant Daunhawer, Alice Bizeul, Emanuele Palumbo, Alexander Marx, and Julia E Vogt. Identifia- bility results for multimodal contrastive learning. In The Eleventh International Conference on Learning Representations, 2023.

[6] Kartik Ahuja, Jason S Hartford, and Yoshua Bengio. Weakly supervised representation learning with sparse perturbations. Advances in Neural Information Processing Systems, 35:15516–15528, 2022b.

[7] Sébastien Lachapelle, Pau Rodríguez López, Yash Sharma, Katie Everett, Rémi Le Priol, Alexan- dre Lacoste, and Simon Lacoste-Julien. Nonparametric partial disentanglement via mecha- nism sparsity: Sparse actions, interventions and sparse temporal dependencies.

[8] Francesco Locatello, Ben Poole, Gunnar Raetsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learn- ing, volume 119 of Proceedings of Machine Learning Research, pages 6348–6359. PMLR, 13–18 Jul 2020.

[9] Kartik Ahuja, Amin Mansouri, and Yixin Wang. Multi-domain causal representation learning via weak distributional invariances. In International Conference on Artificial Intelligence and Statis- tics, pages 865–873. PMLR, 2024.

[10] Jiaqi Zhang, Kristjan Greenewald, Chandler Squires, Akash Srivastava, Karthikeyan Shanmugam, and Caroline Uhler. Identifiability guarantees for causal disentanglement from soft interventions. Advances in Neural Information Processing Systems, 36, 2024a.

[11] Chandler Squires, Anna Seigal, Salil S. Bhate, and Caroline Uhler. Linear causal disentanglement via interventions. In International Conference on Machine Learning, volume 202, pages 32540– 32560. PMLR, 2023.

[12] Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves. Causal representation learning for instantaneous and temporal effects in interactive systems. In The Eleventh International Conference on Learning Representations, 2022a.

[13] Weiran Yao, Yuewen Sun, Alex Ho, Changyin Sun, and Kun Zhang. Learning temporally causal latent processes from general temporal data. International Conference on Learning Representations, 2022a.

[14] David A Klindt, Lukas Schott, Yash Sharma, Ivan Ustyuzhaninov, Wieland Brendel, Matthias Bethge, and Dylan Paiton. Towards nonlinear disentanglement in natural data with temporal sparse coding. In International Conference on Learning Representations, 2021.

[15] Riccardo Cadei, Lukas Lindorfer, Sylvia Cremer, Cordelia Schmid, and Francesco Locatello. Smoke and mirrors in causal downstream tasks. Advances in Neural Information Processing Systems, 37, 2024.