Identifiable Latent Polynomial Causal Models through the Lens of Change

Q1: the difficult cases: e.g. inverse Gamma etc, could have been included ..

R1: Thank you for your suggestion. Given the constraints of the rebuttal time frame, we must prioritize addressing more complex reviews. We will add these experiments in the final verision. Your understanding is greatly appreciated.

Q2: How to obtain the number of latent variables?

R2: That is an excellent question. It is worth noting that some existing works may require knowledge of the number of latent variables. However, this is not a theoretical hurdle in our work. It is essential to highlight that a crucial step in our results involves leveraging the identifiability findings from [1] to identify latent noise variables. The key insight in [1] demonstrates that the dimension of the generating latent space can be recovered if latent noises are sampled from the two-parameter exponential family members (consistent with our assumption on latent noise). This is, if the estimated latent space has a higher dimension than the generating latent space, some estimated latent variables are not be related to any generating latent variables and therefore must encode only noise. More details please check the work [1]. We do not highlight this problem in our original version, considering limited space and that it is not our contribution. We add a discussion about it in new version, see Section A. 11.

[1] Sorrenson, Peter, Carsten Rother, and Ullrich Köthe. "Disentanglement by nonlinear ica with general incompressible-flow networks (gin)." arXiv preprint arXiv:2001.04872 (2020).

Q2: Condition (iii) has u{2l+1} although u’s exist only until 2l. Possible typo?

R2: Please note that the indexing of u starts from 0, not 1.

Q3: Additional intuition for conditions (iii) and (iv)?

R3: Condition (iii) stems from nonlinear ICA, it is basically saying that the auxiliary variable must have a sufficiently strong and diverse effect on the distributions of the latent noises, to facilitate their identifiability. Condition (iv) is derived from the work by Liu et al. (2022), aiming to constrain the function class of lambda to prevent a specific case, e.g., lambda(u) = lambda(u) + b, where b is a constant. In this scenario, the invariant part, e.g., b * Pa (parent node), leads to unidentifiability, as discussed in Corollary 3.3. We have added Section A. 10 in the revised verison for understanding our assumptions. Thanks for your suggestion.

Q4: . additive noise models are identifiable.

R4: It should be "arises from the fact that nonlinear models with additive noise models are identifiable". Thank you for your careful review. In fact, linear Gaussian models in multiple environments are generally identifiable, thanks to the assumption of independent causal mechanisms [1] [2]

[1] Ghassami, AmirEmad, et al. "Multi-domain causal structure learning in linear systems." Advances in neural information processing systems 31 (2018).

[2] Huang, Biwei, et al. "Causal discovery from heterogeneous/nonstationary data." The Journal of Machine Learning Research 21.1 (2020): 3482-3534.

Q5: About Corollary 3.2..What prevented such a condition in Theorem 3.1.

R5: Theorem 3.1 claim a complete indeitifbiality result, requiring all all coefficients to change, as modelled by Eq. (4). However, this assumption may appear strong. To assess its necessity, we introduce such a condition in Corollary 3.2 to reinforce the assumption, highlighting its indispensability for identifiability in the absence of any supplementary conditions. Moreover, as the primary motivation in this work is to capture the changes in causal influences, it is crucial to note that, in practice, these changes could be arbitrary. This implies the possibility that some causal influences remain unchanged.

R5: Assumption (iv) imposes a constraint on the function class, ensuring that $\lambda_{i,j}(\mathbf{u})$ is not non-zero constant. Corollaries 3.2 and 3.3, "Under the condition that the assumption (i)-(iv)" should be "Under the condition that the assumption (i)-(iii)". Thank you for your thorough review.

Q6: .Where is z hidden on the RHS..

R6: Again, that is an excellent question. As non-Gaussian noise lacks an analytically tractable solution in general, we model the prior distribution $p(z|u)$ using $p(\lambda, n|u)$, this is because z can be entirely represented by $\lambda$, $n$, and $u$, as further elaborated in Eq. (14). More deeply, from the perspective of independent causal mechanisms, the latent noise variables $n$ need not to be independent of the coefficient $\lambda$, given $u$, since they both correspond to the mechanism generating the effect on latent causal variables. However, considering the challenges in modeling the complicated dependence between $n$ and $\lambda$, and the challenges in optimization, we employ an approximate implementation, as shown in the second line of Eq. (6), which assumes their independence.

Q1: a definition is not given and it remains unclear what you mean precisely by "identifiability" and by "unidentifiability", rendering the claims of the corollaries unclear.

R1: Please note that the definition of identifiability has already been established in our Theorem 3.1 and in Corollary 3.3, as stated "the true latent causal variables $\mathbf{z}$ are related to the estimated latent causal variables ${ \mathbf{\hat z}}$, which are learned by matching the true marginal data distribution $p(\mathbf{x}|\mathbf{u})$, by the following relationship: $\mathbf{z} = \mathbf{P} \mathbf{\hat z} + \mathbf{c},$ where $\mathbf{P}$ denotes the permutation matrix with scaling, $\mathbf{c}$ denotes a constant vector". Naturally, unidentifiability emerges as the complementary concept to identifiability. That is, z can not be recovered up to $\mathbf{z} = \mathbf{P} \mathbf{\hat z} + \mathbf{c}$.

Q2: Changes across environments, it difficult to determine from the text what the precise meaning is...

R2: Please be aware that we have formulated the changes using the function $\lambda_{i,j}(u)$ for the coefficients as defined in Eq. (4). This implies that the changes are entirely dependent on the function class of $\lambda_{i,j}$, as long as the function satisfies assumption (iv). Additionally, it is crucial to note that we do not make any claims regarding the pair of environments in our results. Specifically, we do not require paired environments, distinguishing our approach from some prior works that do necessitate such pairing. We have also clarified this aspect in our introduction.

Q3: measure zero (i.e., has at the most countable number of elements)...

R3: Thank you for such carefully check. In our previous version, we included the phrase 'has at most a countable number of elements' merely for the purpose of easier reader understanding. However, we recognize that this constitutes a stronger assumption than measure zero. In our updated version, we have removed this phrase. Essentially, we here aims to emphasize that $\varphi_{\varepsilon}(\mathbf{x}) \neq 0$ almost everywhere. We appreciate your careful review.

Q4: What do you mean by "takes up"?...

R4: Thank you for seeking clarification. It's important to note that our assertion does not involve multiple random variables. Our emphasis is on the existence of a single random variable, typically represented by a one-hot vector denoted as $\mathbf{u}$, signifying the environment. Each point or value $\mathbf{u}_i$ represents one environment. This assumption, which is commonplace in nonlinear Independent Component Analysis (ICA) literature, essentially states that the random variable $\mathbf{u}$ must have a sufficiently strong and diverse influence on the distributions of the latent noise variables.

Q5: DAG constraint, notation, and typos

R5: Thank you for such carefully check. We have carefully modified these problems in new version.

Dear Reviewer 22xe,

Your thoughtful reconsideration of our work is highly valued, and we sincerely hope that our clarification provides valuable information, addressing your concerns and enhancing the overall understanding of the contributions and merits of this work. Your thorough re-evaluation is crucial to this work, and we appreciate the opportunity to further engage with any aspects that may contribute to a more comprehensive assessment. Thank you for your time and consideration.

Dear Reviewer 22Xe,

Thank you for such a quickly replay.

For Q1, we have provided an explicit definition in the latest version.

For condition (iii), we have also revised it in the latest version, following a common style found in the majority of existing works that leverage results from nonlinear ICA and incorporate assumption (iii). That is,

There exist $2\ell+1$ values of $\mathbf{u}$, i.e., $\mathbf{u}\_{0},\mathbf{u}\_{1},...,\mathbf{u}_{2\ell}$, such that the matrix $\mathbf{L} = (\boldsymbol{\eta}(\mathbf{u} = \mathbf{u}\_1)-\boldsymbol{\eta}(\mathbf{u} = \mathbf{u}\_0),..., \boldsymbol{\eta}(\mathbf{u} = \mathbf{u}\_{2\ell})-\boldsymbol{\eta}(\mathbf{u}= \mathbf{u}\_0))$ of size $2\ell \times 2\ell$ is invertible.

For Corollary 3.2, please note that we do not impose any limitations on function $\lambda_{i,j}$ in equations (1)-(4). Therefore, even if equations (1)-(4) hold, $\lambda_{i,j}(\mathbf{u})$ could be a constant. In fact, our assumption (iv) in Theorem 3.1 imposes that $\lambda_{i,j}(\mathbf{u})$ can not be a nonzero constant.

Dear Reviewer 22Xe,

Your feedback is of immense importance to us. Could you kindly verify if the provided clarification effectively addresses your concerns, especially regarding the core concepts?

Should any uncertainties persist or if additional clarification is warranted, please donot hesitate to inform us. We recognize the demands on your time and sincerely appreciate your thoughtful consideration. Your reassessment plays a pivotal role in advancing our work, and we stand ready to provide any further clarification you may require.

Best regards, Authors

Dear Reviewer 22Xe,

We are writing to express our sincere appreciation for the time and effort you have dedicated to reviewing my manuscript. Your valuable feedback has been instrumental in refining the quality of the work.

We are currently in the process of finalizing the revisions based on your insightful comments. As the deadline for rebuttal is approaching, We wanted to reach out and ensure that we have adequately addressed any concerns you may have had.

In response to your feedback, We have focused on:

1. We have extracted the definition of identifiability from Theorem 3.1 and formalized it as Definition 3.1.
2. We have reexpressed the assumption (iii) in Theorem 3.1.
3. We have taken special care to clarify that even if equations (1)-(4) are satisfied, $\lambda_{i,j}(\mathbf{u})$ could be a constant.

We believe these revisions contribute to the overall clarity and robustness of the manuscript.

We understand that your schedule may be busy, and we greatly appreciate the time you have already invested in the review process. However, as the deadline is imminent, we wanted to kindly remind you of the approaching date and express my eagerness to incorporate any additional suggestions or address any lingering concerns you may have.

We are more than willing to provide further clarification or discuss any aspects of the manuscript in more detail. Your feedback is invaluable, and we want to ensure that the latest version of the manuscript meets the high standards.

Thank you once again for your time and consideration.

Best,

Authors

Q1: Latent structure identifiability

R1: Thank you for your high-level and insightful suggestions. Our primary finding, as presented in Theorem 3.1, demonstrates that latent causal variables can be identified up to permutation and scaling. Importantly, permutations do not affect the underlying latent graph structure, as they are solely related to namespace. Regarding scaling, as mentioned in our paper, the identifiability of nonlinear models with additive noise remains unaffected by scaling, as we claimed in the paragraph following the proof sketch for theorem 3.1. Consequently, the equivalence class, encompassing permutation and scaling, corresponds to the same graph structure. In terms of partial identifiability, we think that our current results only allow for the identifiability of the values of a subset of latent variables. In such cases, asserting the recovery of a portion of the true causal graph is challenging, given our belief that the unidentifiable part may introduce various possible effects on the graph structure within the identifiable portion. We argue that understanding the underlying causal structure is often more crucial than identifying individual variables. However, for certain applications, such as domain adaptation, we may be interested in splitting the latent space into an invariant space and a variant space, as domain adaptation can leverage the invariant part to predict labels, regardless of the graph structure in latent space. We have incorporated these discussions in the new version. Please see Section A.8 in the revised version for more details.

Q2: Functional assumption

R2: Thank you once again for your insightful suggestions. In general, we believe that model assumptions can be extended to a more general case than polynomial, even for non-parametric models, as long as the changes in causal influences are sufficient. However, defining what sufficient changes poses a challenge. In this work, we opted for polynomial models, considering their approximation ability and simple expression, which is beneficial for analysis and formulating the notion of sufficient changes. Simultaneously, we acknowledge the limitations of this choice, particularly with the presence of high-order terms in polynomials, making optimization highly challenging. Please see Section A.9 in the revised version for more details.

Q3: Undestanding the assumptions

R3: Thank you once again. We also recognize the significance of facilitating a better understanding of these assumptions for our readers. Consequently, we have augmented our presentation by including a dedicated in Section A. 10 in the revised version to provide detailed insights into these assumptions. For example, two of the main assumptions (iii) and (iv) in Theorem 3.1. Assumption (iii) stems from nonlinear ICA, essentially requiring sufficient changes in latent noise variables to facilitate their identification. Assumption (iv) is derived from the work by Liu et al. (2022), aiming to constrain the function class of lambda to prevent a specific case, e.g., lambda(u) = lambda(u) + b, where b is a constant. In this scenario, the invariant b leads to unidentifiability, as discussed in Corollary 3.3.

Q1: an incremental improvement, a lack of how alleged extension was achieved

Response: We respectfully disagree with that. Our contributions, highlighted throughout the abstract, introduction, and theoretical results, encompass the following advancements:

1. The dependencies among latent variables, previously assumed to be linear in Liu's work, are demonstrated to be polynomial in our research.
2. Contrary to the Gaussian assumption in Liu's work, we establish that the noise associated with latent variables follows an exponential distribution.
3. In comparison to Liu's prior work, our proof requires a significantly reduced number of environments (2l+1 environments), a substantial improvement from the previous quadratic requirement in Liu's work.
4. We also provide proofs for partial (un)identifiability, specifically addressing situations where the assumption of a complete change in causal influences is violated.
5. Introducing an estimation method, we validate its efficacy on both synthetic and real-world datasets.

All theoretical contributions are rigorous justified by the Theorem 3.1, Corollary 3.2, Corollary 3.3, and their respective proofs. Further, experimental results serve to affirm not only the soundness of our theoretical contributions but also the practical efficacy of the proposed estimation method. Finally, to understand the motivation behind these theoretical advancements, our Introduction sheds light on the fact that LIU's work primarily focuses on identifying latent causal variables from the perspective of observed data. In contrast, our research is propelled by the realization that we can significantly narrow down the solution space of latent causal variables by considering the perspective of latent noise variables. This new insight motivates us to propose more general models than LIU's work.

Q2: Kronecker product definition is not clear, not the usual definition.

Response: We deviate from the conventional definition by employing the notation ${\bar \otimes}$ in place of the usual ${\otimes}$ (usually denote the Kronecker product) to denote the operator. Additionally, we explicitly elucidate that the operator, denoted as ${\bar \otimes}$, signifies the Kronecker product with all distinct entries.

Q3: Reviewer kugU, Q3: Unusual, and IMHO absurd, number of assumptions

Response: We respectfully disagree with this point. Many of the assumptions, encompassing i to iii, stem from the realm of nonlinear ICA. It is important to acknowledge that concerns related to objectivity and the existence of $\mathbf{L}$ pertain to assumptions ii and iii. Over recent years, nonlinear ICA has found applications in diverse real-world scenarios, such as domain adaptation [1] and image generation and translation [2], domain generalization [3]. Consequently, we assert that these three assumptions have been substantiated through practical applications.

The remaining assumptions, including model assumptions and assumption (iv), revolve around the changes of causal influences. These changes are firmly grounded in the prevalent occurrence of distribution shifts in real-world applications, as elucidated in the first paragraph, far from being speculative. In contrast to many prior works that necessitate hard interventions to model the potential changes above, our approach employs soft interventions, enabling the modeling of a broader spectrum of conceivable alterations.

[1] Kong, Lingjing, et al. "Partial Identifiability for Domain Adaptation." ICML 2022.

[2] Xie, Shaoan, et al. "Multi-domain image generation and translation with identifiability guarantees." ICLR 2022.

[3] Wang, Xinyi, et al. "Causal balancing for domain generalization." arXiv preprint arXiv:2206.05263 (2022)

Q4: all shades of exogenous/latent terms (z, u, n) are not being explored thoroughly from first principles.

Response: Again, I respectfully dissent from this point. Within the causal representation community, there is a widely accepted understanding that observed data is generated by latent causal variables z. It is also well-established that each latent variable zi corresponds to a specific exogenous factor ni in causal systems. Furthermore, we assert that the variable (u) serves as a surrogate, capturing changes within the system. These conceptualizations align with established conventions in the community, and we contend that they are readily embraced by the community members.

Dear Reviewer kugU,

We greatly appreciate your feedback. Could you kindly verify if the provided clarification addresses your concerns, particularly regarding contributions and assumptions?

We wish to highlight our contributions once again. The challenge of identifying causal representations in learning is particularly formidable. Current approaches, which mainly focus on hard interventions through changes in causal influences (distribution shifts), are limited. We contend that implementing hard interventions in the latent space is nearly impractical due to the unobservable nature of latent causal variables.As a consequence, hard interventions should be interpreted as modeling changes generated by self-initiated behavior within causal systems across diverse environments. However, self-initiated changes in causal systems can be arbitrary. In this context, hard interventions only permit model-specific changes, whereas our approach, involving soft interventions, has the potential to model a broader range of possible changes. To our knowledge, the work of Liu et al. 2022 is the sole instance utilizing soft interventions, albeit confined to special linear Gaussian models. In contrast, our work advances this by considering nonlinear models with exponential family noise. This transition from linear to nonlinear, from Gaussian noise to exponential family noise, is a significant step forward. Importantly, it also mitigates the requirement for a high number of environments.

If there are lingering uncertainties or if additional clarification is necessary, please don't hesitate to notify us. We understand the time constraints you face and truly appreciate your thoughtful consideration. Your reassessment plays a crucial role in advancing our work, and we stand prepared to offer any further clarification needed.

Best regards, Authors

Q1: The proof uses common ideas of exponential family distribution [1] to obtain identification guarantees up to permutation and scaling for the noise variables.

R1: It appears there might be confusion. We main result, Theorem 3.1, is using the result from [2], not [1], to obtain identifiability up to permutation and scaling for the noise variables.

[1] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables and generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 859–868. PMLR, 2019.

[2] Sorrenson, Peter, Carsten Rother, and Ullrich Kothe. "Disentanglement by nonlinear ica with general incompressible flow networks." arXiv preprint arXiv: 2001.04872 (2020).

Please be aware of a substantial disparity between the identifiability of latent noise variables n and latent causal variables z. Notably, assuming an absence of all edges among z makes it trivial to apply the results of nonlinear ICA. However, in real-world applications, latent graph structures can be arbitrary. Consequently, the solution space for identifying z is notably more expansive than that for identifying n. This disparity is critical, especially given the complexity introduced by arbitrary graph structures. Essentially, many existing works focusing on the changes of causal influences, e.g., hard intervention or soft intervention, are dedicated to addressing this pronounced gap.

Q2: Further, the idea of using the changing distribution of latent variables identified up to polynomial mixing can be used to obtain permutation and scaling identification guarantees has been explored in prior works [2].

R2: It is essential to highlight that [2] achieves the recovery of latent causal variables identified up to polynomial mixing by assuming a polynomial mapping from latent causal variables z to observed x. In contrast, our approach employs a more general mapping f, as defined in Eq. (3). Consequently, in our setting, the task of recovering latent causal variables identified up to polynomial mixing becomes more challenging due to the increased flexibility and complexity introduced by the general mapping function f.

Most importantly, the distinction between this work and [2] is substantial. [2] utilizes hard interventions to identify latent causal variables, while this work employs soft interventions for the same purpose. To elucidate this significant difference, let us consider a fundamental question: how can hard interventions be practically applied to latent causal variables?

We posit that conducting hard interventions on latent space is almost impractical due to the unobservable nature of latent causal variables. Consequently, hard interventions should be understood as modeling changes generated by self-initiated behavior within causal systems across environments. However, the self-initiated changes in causal systems across environments can be arbitrary. In this context, hard interventions in [2] only allow model specific types of changes, while this work, involving soft interventions, can model a broader range of possible changes. This versatility may prove more attainable for latent variables than the constrained nature of hard interventions.

Q3: ...In terms of only the latent identification (not recovering causal relationships)...

R3: Please take note that our identifiability results, elucidated in Theorem 3.1, inherently signify the identifiability of the causal graph structure, as explicitly stated in the paragraph following the proof sketch for Theorem 3.1. This assertion is grounded in the inherent identifiability of nonlinear models with additive noise. Please see the detailed clarification provided in the corresponding paragraph for a more thorough and comprehensive understanding of this crucial aspect. Furthermore, it is important to highlight that our results are not contingent upon any assumptions regarding latent structures. This underscores the flexibility of our framework, allowing for the consideration of arbitrary structures.

We will add experiments in the final version! We sincerely appreciate the time and effort you dedicated to reviewing our work.