Turning Challenges into Opportunities: How Distribution Shifts Enhance Identifiability in Causal Representation Learning

We thank Reviewer FhZC for taking the time to review our submission and for their positive feedback, including a meaningful theoretical contribution, as a bridge is an interesting direction. Below, we address the reviewer’s concerns on presentation.

Q1: The notion of “soft intervention” seems to vaguely refer to "intervention different from the standard definition"... The concept seems unnecessary for the paper,....If “soft intervention” is central, a formal definition would help..

R1: We greatly appreciate the reviewer for pointing out this concern. We fully acknowledge the issue raised, as without a clear definition, confusion can easily arise. To address this, we have added a new Section K in the Appendix. In this section, we clarify that u is a surrogate variable introduced into a causal system to characterize changing causal mechanisms, and emphasize that u should not be interpreted as a variable within the causal system itself. We also highlight the distinction between soft and hard interventions, not only in the general understanding of these terms but also by citing a paper that provides a formal definition. Additionally, we have modified the position of u in Eq. (2) from being an input to a superscript, aligning it more closely with the representation of soft interventions. Furthermore, we clarify that when we refer to interventions, we are generally discussing interventions acting on latent causal variables, e.g., $z_i$. More details on these points can be found in the newly added Section K in the Appendix.

Q2: Regarding assumption (iv), I see it does not require data specific to a particular value, but I’m unclear why..... It seems that whether values are necessary depends on how assumptions are used in proofs...

R2: Indeed, whether the values of u are necessary depends on how the assumptions are applied in the proofs. We have updated this section for greater clarity, thank you once again for your careful review. Essentially, the necessity of u depends on the specific role the assumptions play in the proof. The logic follows that different proof techniques lead to different conditions for identifiability. In this work, we show that a limited function class, such as the one constrained by assumption (iv), can contribute to identifiability. Interestingly, assumption (iv) requires the existence of a mechanism for removing incoming edges from parent nodes, which is closely related to the concept of hard interventions used for identifiability in existing work. Further exploration of assumption (iv) could provide a new bridge connecting this work with existing studies, presenting an exciting avenue.

Q3: Comparison with Liu et al. (2022; 2024) in Generative Models and Inference Methods.

R3: We have added a new Section L in the Appendix of the updated version. In this section, we detail the differences between our work and previous studies in generative models, as well as provide a comparison of inference methods. For more details, please refer to Section L.

Q4: Component-wise identifiability isn’t achieved for general exponential family distributions....

R4: Thank you for your rigorous review. We have revised the relevant parts accordingly and added a footnote to highlight that we use the two parameters from Sørenson et al. (2020) in a limited manner.

Q5: In Remark 3.2, it would help to clarify that an unidentifiable and identifiable example are being considered from the outset....A clearer illustration would be examples that satisfy (iii) but only one satisfies (iv).

R5 We have revised the relevant part in Remark 3.2. Additionally, we have provided another example for stronger understanding in Section I of the Appendix.

Q6: In Remark 3.4, ....In Thm. 3.5(a), L307: I think identifying...Both Remark 4.2 and Intuition regarding Corollary 4.1 could be removed,..

R6: We have revised these parts accordingly. Thank you for your suggestion!

Q7: It would be informative to see if this method outperforms "iVAE to recover and then existing causal discovery methods to build causal graphs among,.... if time permits, additional synthetic data

R7: We have added the results from iVAE + CNNOD [1] as shown in Figure 1. We have also added a new Section J to present results with a larger number of latent causal variables.

[1] Huang, Biwei, et al. "Causal discovery from heterogeneous/nonstationary data." Journal of Machine Learning Research 21.89 (2020): 1-53.

We thank Reviewer Rg2d for taking the time to review our submission and for their positive feedback, including recognition of our work as well-structured and fluently written, with excellent intuition and insights, as well as meaningful and impactful results for real-world applications. Below, we address the reviewer’s concerns in detail.

Q1: The paper relies on [Liu et al., 2024], extending ..to nonlinear functions, and modifying condition (iv).. Although the relaxation of these assumptions significantly enhances the model's applicability to real-world scenarios, the core motivation..overlaps substantially with [Liu et al., 2024]. Thus, as a follow-up work, I tend to view this paper as an incremental improvement.

R1: Thank you for the detailed comparison!

First, we would like to emphasize that using distribution shifts for identifiability is a well-established research direction, with many existing works as mentioned introduction and related work. The key difference among these works lies in considering which types of distributions apply under which function class.

Second, in additive noise models, the nonlinear component is not solely based on changes in the coefficients. Since the nonlinear part is entirely nonparametric, the changes encompass the entire nonlinear mapping. For example, the changes could involve $\lambda$ in $z_2=\lambda_1 z_1 + \lambda_1 z^2_1 +n_2$, or they could involve $\beta$, $z_2= g(z_1,\beta) +n_2$ where the changes occur within the unknown function $g$, beyond the coefficients.

Third, one significant reason for the success of modern machine learning is its reliance on sophisticated structural units, such as transformers. These complex structures often correspond to intricate function relationships and are typically non-parametric, as seen in models like MLPs. The nonparametric components in additive noise models allow for complex structures that go beyond the limitations of polynomial forms, which are restricted to terms such as quadratic or cubic.

Furthermore, we extend our results to post-nonlinear causal models, which are more powerful than additive noise models, i.e., they accommodate multiplicative noise.

Finally, we emphasize that progressing toward a general result is a long-standing research endeavor, and obtaining such results is typically not a one-step process. In this long process, this work serves as a foundational step, generalizing some previous results and providing a potential cornerstone toward a general result.

Q2: What is the essential difference between the nonlinear assumption in condition (iv) and the scenario where the coefficient of is zero?

R2: Note that here function $g_i$ is non-parametric. Therefore, condition (iv) is specifically designed for such non-parametric functions to define the boundary of identifiability. Unlike parameter-specific conditions that are restricted to particular cases, such as polynomials with zero coefficients or the nonlinear case $sin(z_1)$ you mentioned, our condition transcends these limitations. It offers a comprehensive and general framework applicable to all function classes, including special cases like polynomials where the coefficient equals zero. This implies that if changes in causal influences satisfy condition (iv), identifiability is guaranteed; otherwise, it is not, irrespective of the underlying function class. The strength of this non-parametric approach lies in its broad applicability and robustness, greatly expanding the scope and practical importance of our identifiability analysis.

Dear Reviewer Rg2d,

As the submission deadline approaches, we wanted to gently remind you of the clarification we need regarding the concerns in your review. We've invested significant time and effort into polishing the draft, carefully addressing your suggestions.

We understand you have a busy schedule, but your response is essential for us to proceed with the final version. Your input is invaluable, and we would greatly appreciate your insights at your earliest convenience.

Thank you so much for your time and assistance.

Best regards,

Authors

We thank Reviewer srq7 for taking the time to review our submission and for acknowledging that our results are easier to follow, more general, and offer hope for practical applications. Below, we address the reviewer’s concerns in detail.

Q1: This paper touches upon a CRL setting which is quite similar to the mutli-environment setting in [1]. One main difference that I can see is the assumed intervention type. However, the authors do not contrast their results to [1]. How is this work a generalization of that?

R1: Thank you for your careful review. We have incorporated reference [1] in the updated version. As noted in [1], it includes Assumption 2.9 (Perfect Interventions), which corresponds to hard interventions, the primary focus of most existing works. This stands in contrast to the soft interventions investigated in our study. In the introduction, we emphasized a research direction that leverages distribution shifts for identifiability analysis. A key distinction among these works lies in determining which types of distributions are suitable for specific function classes, such as those involving hard or soft interventions. Soft interventions, in particular, offer greater flexibility by accommodating a wider range of self-initiated behaviors compared to hard interventions. Advancing the development of soft interventions for identifiability is thus crucial for the causal representation learning community and represents a significant step toward a more practical and comprehensive theory.

Q2: The experimental evaluation is lacking many important details. E.g. for the synthetic data, the used mixing function is not stated......Furthermore, no CRL method, such as e.g. [1] and others, is compared against.

**R2:**We have updated the details of the mixing function in the synthetic data experiments. Regarding comparisons, a key challenge lies in the lack of alignment in assumptions across studies employing interventions. For instance, some studies assume hard interventions (e.g., [1]), while others, like ours, focus on soft interventions. Model choices further differ, with some assuming linear latent spaces and others adopting nonlinear models. Additionally, while some works rely on paired interventional data, our study addresses unpaired data. These inconsistencies make fair comparisons difficult, posing significant challenges for most studies, including ours, to provide comprehensive and meaningful analyses. In a comparable setting involving soft interventions, such as the polynomial models in [Liu et al., 2024], the primary distinction lies in the assumed function class for latent variables. In our experiments, we use this work as a baseline for comparison, as highlighted in the results.

Q3: I'm confused about Assumption (iv): The setting is introduced as using soft interventions, however, as far as I understand, Ass. (iv) constrains the interventions to behave almost like a hard intervention. I.e. there exists an exogenous setting for each of the parents of a given variable such that the parents have no influence on the child. How is this assumption more general than a hard intervention?

R3: Please note that soft interventions allow for the assignment of potentially functional transformations to a variable, which can, in general, encompass the specific changes induced by hard interventions, such as through the application of Dirac delta functions. Assumption (iv) imposes constraints on the function class, requiring it to include a special point u' where the partial derivative is zero. Importantly, this function class is not limited to such special points u', it also includes other points that differ from u'. As highlighted in Remark 3.3, prior works based on hard intervention rely solely on the special point u' for identifiability. In contrast, our results demonstrate that any point within the constrained function class—whether the special point u' or any other differing points—can be utilized for identifiability. This makes our assumption strictly more general than hard interventions, extending the applicability of the identifiability analysis.

Q4: Why do you only consider two parameter exponential family distributions? Is this a simplifying assumption, or is there some fundamental reason other distributions wouldn't work?

R4: As we mentioned, the two-parameter exponential family eliminates the need for assumptions about the dimensionality of latent causal or noise variables. More details can be found in "Sorrenson, Peter, Carsten Rother, and Ullrich Köthe. "Disentanglement by nonlinear ica with general incompressible-flow networks (gin)." arXiv preprint arXiv:2001.04872 (2020)."

We thank Reviewer y9GW for taking the time to review our submission and for recognizing well structured, relatively easy to follow. We address the reviewer’s conerns below.

Regarding to Proof

Q1: Some "hatted" function 'having to belong to the same function class' to transfer properties of a non-hatted function to a hatted one. I do not see, where this assumption(?) comes from and what it means on a technical level? This appears to be a crucial element in various proofs, e.g., lines 888-889 in the orignial version, which corresponds to lines 943-944.

R1: The underlying logic is that if we constrain the function class in generative models, the same function class can be enforced in inference models to maintain consistency with the generative model, without introducing any additional assumptions. Specifically, the statement in lines 888-889 in the orignial version, we have a point $\mathbf{u} _ {i'}$, so that $\frac{\partial {h _ 3(n _ 1,n _ 2,n _ 3,\mathbf{u}=\mathbf{u} _ {i'})}}{\partial n _ {2}}=0$, is an alternative expression of assumption (iv) regarding function class constraints in generative models, supported by Lemma A.3. Consequently, the same function class constraints can be applied in inference models, such as ensuring $\frac{\partial {\hat h _ 3(n _ 1,...,n _ 3,\mathbf{u}=\mathbf{u} _ {i'})}}{\partial n _ {2}}=0$. This idea is analogous to assuming an exponential family distribution for noise in the generative model, which allows us to enforce the same distribution in the inference model, or assuming that the mapping from latent to observed space in the generative model is invertible, enabling the enforcement of an invertible mapping in the inference model as well.

Q2: I am not yet 100 convinced by the sufficiency proof (Thm 3.5). In particular, the authors state that line 944 (e.g., 998 in the new version) that the required equality for matching marginals can be true even when don't see why such a situation must always exist within the problem setting.

R2: As stated in lines 944 in the original version 'This implies that $J _ {{\mathbf{\Phi}} _ {2,1}}$ can have a non-zero value.' The rationale here is that this step is intended to demonstrate unidentifiability. In this context, showing the existence of a scenario where $J _ {{\mathbf{\Phi}} _ {2,1}}$ can take on a non-zero value is sufficient to establish unidentifiability. It does not require $J _ {{\mathbf{\Phi}} _ {2,1}}$ to always have a non-zero value.

Regarding to assumptions

Q3: In this regard, the authors mention in line 214, that 'these assumptions have been verified to be practicable in [...] application scenarios'. First, I'm not sure what it means for assumptions to be practicable, I suppose they're defendable or justified? However, when looking at the cited works, I do not see how these works justify the made assumptions in practice?

R3: Assumptions (i)-(iii), as discussed, were originally proposed in the context of nonlinear ICA and have since been widely utilized in a variety of real-world applications. Although fully justifying these assumptions in practical scenarios remains challenging, their adoption has enabled the development of theories and methods that consistently achieve state-of-the-art performance in the cited works. This success provides strong empirical evidence that these assumptions are not only theoretically meaningful but also practically viable, at least within the scope of the settings and applications explored to date.

Q4: additive noise models are another (possibly restrictive) assumption.

R4: Thank you for your insightful feedback. We acknowledge that additive noise models indeed represent a specific and constrained class of models. Our original aim was to emphasize that, in contrast to the polynomial models in [Liu 2024], which depend on fixed terms like $z^2_i$ or $z^3_i$ to represent nonlinear relationships, the nonlinear component in additive noise models is often more versatile. For example, it can be modeled using complex structures such as multilayer perceptrons (MLPs) or other transformations commonly employed in modern deep learning frameworks. We have revised this section in the updated version to better articulate this distinction. We appreciate your careful review and constructive comments.

Q5: I believe (Forbes Krueger, 2019) somehow snuck in there accidentially (not related to PPI or small molecules at all)?

Q5: Thank you for pointing out the citation typo. The correct citation should indeed be: Scott, Duncan E., et al. "Small Molecules, Big Targets: Drug Discovery Faces the Protein–Protein Interaction Challenge." Nature Reviews Drug Discovery, vol. 15, no. 8, 2016, pp. 533–550.