On the Identifiability of Nonlinear Representation Learning with General Noise

Q5: It would have been nice if there were experiments for the other specific settings.

A5: We sincerely appreciate your constructive feedback. In light of it, we have conducted additional experiments and included the results in Appx. D.2. As shown in Figure 11, latent variables are identifiable in the presence of noise and distortion in the considered settings. Additionally, we can recover the structure of the Jacobian; however, since we have not proposed a practical causal discovery algorithm, we focus here on the identification of latent variables.

Q6: I found the writing quite verbose in many places. The authors seem to give examples without giving citations.

A6: Thanks a lot for your suggestions on the writing. In light of your feedback, we believe that removing some examples would provide more space for including proof sketches in the main paper, which will highlight the unique technical contributions in the proof as well as making the discussion more concise. We have made the changes in the updated manuscript accordingly (e.g., L168-175). We sincerely appreciate your time devoted to improving the manuscript, and looking forward to any further feedback.

References

[1] Von Kügelgen et al., Self-supervised learning with data augmentations provably isolates content from style, NeurIPS 2021

[2] Kong et al., Partial identifiability for domain adaptation, ICML 2022

[3] Lachapelle et al., Nonparametric partial disentanglement via mechanism sparsity: Sparse actions, interventions and sparse temporal dependencies. arXiv 2024

[4] Zheng et al., On the identifiability of nonlinear ICA: Sparsity and beyond, NeurIPS 2022

[5] Zheng and Zhang, Generalizing nonlinear ICA beyond structural sparsity, NeurIPS 2023

[6] Khemakhem et al., Variational autoencoders and nonlinear ICA: A unifying framework, AISTATS 2020

We deeply value your detailed review and the insightful feedback you have shared. All of these constructive suggestions have further improved our manuscript. In light of these, we have incorporated several new discussions and a new section in the updated manuscript, particularly focusing on clarifying theoretical results within the context of broader literature. Moreover, we have also conducted additional experiments as you suggested. Please kindly find our detailed response to each of your comments below. If you have any additional feedback, we would be truly grateful to hear it and would be delighted to provide further clarifications.

Q1: Difference on the noise modeling and the separation of content and style.

A1: This is an excellent point, which inspired us to add a new section for detailed discussion and clarification. These works are highly interesting and contribute significant theoretical advances in identifying latent variable models. Regarding the two considered settings, we believe the key distinction lies in the nature of the noise and content variables.

For content, they are usually semantically meaningful and are often explicitly compared with style variables. This makes it natural for existing techniques to disentangle content from style. For instance, in contrastive learning (Von Kügelgen et al., 2021), one can find pairs of observations differing only in their styles and use contrastive learning to disentangle the content. Similarly, in multi-domain settings with $O(n)$ domains (Kong et al., 2022), the content remains invariant while there are $O(n)$ (e.g., $2n+1$) distinct domains characterized by different styles. Similarly, in scenarios involving actions or interventions (Lachapelle et al., 2024), agents/domains/environments act as auxiliary variables, inducing multiple changes in the conditional distribution by intervening/impacting on latent variables.

However, for noise, they often lack semantic meaning and cannot be explicitly changed across multiple domains like actions or styles, nor can they be paired in observations as in contrastive learning. Thus, it is challenging to assume the existence of $O(n)$ conditional distributions based on $O(n)$ auxiliary variables or to define contrastive objectives for noise modeling. This is a specific challenge that we face and cannot be directly addressed by previous frameworks. Moreover, for learning SEMs with measurement error, we have more noises compared to observed ones, which cannot be solved by previous methods.

Therefore, for the modeling of general noise, we propose assuming only the existence of variability in the noise distribution, specifically requiring two distinct distributions as a minimal degree of change. This relaxation, from $O(n)$ to two distributions, not only quantitatively weakens the assumption but also qualitatively broadens its applicability. It shifts from explicitly controlling different types of variables to achieve a required degree of change or transition, to accommodating any scenarios where the distribution exhibits variability and is not completely invariant. This is clearly a significant technical contribution that addresses the unique challenges in modeling general noise.

In light of your great question, we have also added a new section for detailed discussion, with a particular focus on the difference with previous work in content-style separation:

“In this section, we discuss the challenge of modeling general noise, emphasizing the distinctions between noise and content variables as explored in previous works.”

“Content variables are typically semantically meaningful and are often explicitly contrasted with style variables. This enables existing techniques to disentangle content from style through structured variability. For example, contrastive learning frameworks (Kugelgen et al., 2021) use paired observations differing only in style (e.g., images with the same object but different backgrounds) to disentangle content. In multi-domain settings (Kong et al., 2022), content remains invariant across $O(n)$ distinct domains characterized by different styles. Similarly, in intervention-based settings (Lachapelle et al., 2024)}, agents or environments serve as auxiliary variables that induce changes in the conditional distributions of latent variables. These structured variations provide the foundation for effective disentanglement.”

“In contrast, noise variables often lack semantic meaning and cannot be explicitly manipulated across multiple domains or paired in observations. This makes it infeasible to assume the existence of $O(n)$ conditional distributions or define contrastive objectives. As a result, existing frameworks designed for content-style disentanglement cannot be directly applied to general noise modeling.”

We are very grateful for the time and effort you dedicated to thoroughly reviewing our manuscript and providing constructive feedback, which has been invaluable in improving its quality. In particular, your thoughtful suggestions have prompted us to include several clarifications and new discussions to enhance the clarity of our messages. New experiments have been also been conducted to further support our theoretical findings. We kindly invite you to review our detailed responses below and would sincerely appreciate any additional feedback you may have.

Q1: While notation is thoroughly defined and overall there is a lot of discussion, still there are many key places where details are either missing or difficult to ascertain.

A1: Thank you very much for taking the time to thoroughly review our manuscript. We sincerely appreciate your comments. Here we provide point-to-point responses to the mentioned details as follows:

• Q1.1: Is it assumed that the $2$ domains are labeled?

• A1.1: Thank you for the question. Yes, as mentioned in L469 and L475, the two domains are labeled. It might be worth noting that, most previous works (see e.g., a recent survey (Hyvärinen et al., 2024)) require $2n+1$ labeled domains, where $n$ is the number of latent variables; while we only necessitate $2$ domains for arbitrary number of latent variables, which is essential a minimal degree of variability. To avoid potential confusion, in addition to the current notes, we have further highlighted it earlier in the updated manuscript:

  “These two domains are realizations of a domain variable $\mathbf{u}$, which are observed and labeled.”

• Q1.2: What is the sparsity constraint and where is it listed in the assumptions?

• A1.2: Thanks for your question. The sparsity constraint refers to an $\ell_0$ regularization on \hat{\mathcal{F}}_\hat{z} during estimation, approximated using $\ell_1$ for gradient-based optimization (L468). Since the identifiability studies what conditions make the data generating process identifiable, and the sparsity constraint is a regularization used during estimation, we previously did not list it in the assumptions. However, we agree with you that introducing it in the theorem could help avoid potential confusion. Therefore, we have updated the manuscript and added the following clarification to the theorems:

  “Together with a $\ell_0$ regularization on $\hat{\mathcal{F}}\_\hat{z}$ during estimation (${\\|\hat{\mathcal{F}}\_{\hat{z}}\\|}_0 \leq {\\|\mathcal{F}_z\\|}_0$), suppose the following assumptions …”

  Thank you again for your constructive feedback. If you have any additional suggestions, please feel free to share them, and we would be more than happy to incorporate further changes to improve clarity.

• Q1.3: Why is Eq. 35 (now Eq. 54) without loss of generality?

• A1.3: This is because $j_1 \neq j_2$, and thus it is either $j_3 \neq j_1$ or $j_3 \neq j_2$. Since $j_1$ and $j_2$ are symmetric in the proof, we can assume $j_3 \neq j_1$ without loss of generality, making Eq. 35 (now Eq. 54) valid.

• Q1.4: Can more details be provided on this split between latent factors and noise factors in the synthetic data? What is the “base” experiment in this case?

• A1.4: Thanks so much for your suggestions. As mentioned in the experimental setup, the main difference between latent and noise variables in generating synthetic dataset is that, for latent variables, we sample them from two distinct multivariate Gaussian distributions conditioning on the corresponding domain index, while the noise is sampled from a single multivariate Gaussian. The corresponding details of the distributions are included in the appendix, which we now moved to the main paper in the updated manuscript to make details more noticeable:

  “During training, latent variables are drawn from two multivariate Gaussian distributions to satisfy the variability condition, while noise is also sampled from a separate multivariate Gaussian, with means sampled uniformly from the range $[-5, 5]$ and variances sampled uniformly from $[0.5, 2.5]$.”

  For the baseline model, we use a fully connected structure to violate the structural sparsity condition, and use a single domain to violate the variability condition. We have modified the current description (L479-481) as follows:

  “In contrast, the baseline model (Base) violates key assumptions, particularly those related to structural sparsity (by a fully connected structure) and variability (by sampling from a single domain).”

We sincerely appreciate your insightful feedback and valuable comments, which have been immensely helpful in improving our manuscript. In response, we have included additional clarifications, expanded discussions, and a new section in the updated manuscript. Moreover, we have conducted additional experiments to further validate our theoretical results in more diverse settings. Please find our detailed, point-by-point responses below. If you have any further feedback, please do not hesitate to share, and we would be more than happy to address it.

Q1: It is unclear how Definition 1 (element-wise identifiability) is commonly used in the field of latent variable identification, as only two papers (by the same author) are cited below this definition.

A1: Thanks for raising this point. Element-wise identifiability is the most common objective in the related field, and arguably the best we can achieve without additional information. In the original submission, we only listed two papers since (Hyvärinen and Pajunen, 1999) was one of the first to explore the problem of identifying nonlinear latent variable models with a focus on existence and uniqueness, while (Hyvärinen et al., 2024) provides a comprehensive survey on the development in the last two decades.

At the same time, we agree that introducing more works would further highlight the importance of Definition 1 in the literature, and we have updated the manuscript as follows:

“Element-wise identifiability guarantees that the estimated factors correspond to the true generating factors without any mixture or entanglement. Standard ambiguities such as permutations and rescaling may remain after identification, which are fundamental indeterminacies commonly noted in the literature (Hyvärinen & Pajunen, 1999; Khemakhem et al., 2020a; Sorrenson et al., 2020; Hälvä et al., 2021; Yao et al., 2021; Lachapelle et al., 2022; Buchholz et al., 2022; Zheng et al., 2022; Lachapelle et al., 2024; Hyvärinen et al., 2024) and represent the best achievable outcome without imposing further restrictive assumptions.”

We hope that our explanation and the update addresses your concern and clarifies the relevance of Definition 1 within the context of latent variable identification. Thanks again for your comment.

Q2: In all the theorems, the author notes that the dataset should be sufficiently large. However, it is unclear what qualifies as 'large' and how this requirement is applied in the proofs.

A2: Thank you for your great question. Following standard practice in the identifiability literature (see e.g., a recent survey (Hyvärinen et al., 2024)), our theoretical results are derived under the asymptotic setting, which is why we note ‘sufficiently large’ in the theorem. This ensures that finite sample errors do not interfere with the results, allowing the proofs to focus on the asymptotic case.

To clarify and avoid potential confusion, we have removed this phrasing from the theorems and instead highlighted the asymptotic setting in the preliminaries as follows:

“Following the previous works (see e.g., a recent survey (Hyvärinen et al., 2024)), all of our results are in the asymptotic setting.”

We hope this could help to avoid potential misunderstanding, and we sincerely appreciate your constructive feedback.

We deeply appreciate the time and effort you invested and your insightful comments, which have greatly enhanced the clarity of the manuscript. Accordingly, we have added several new discussions, full details of previous results, and additional experiments in the updated manuscript, with a special focus on the relation with previous works and clarification on conditions. Please kindly find our response to each point detailed below. If you would like to offer any further suggestions, please feel free to let us know; we would be more than happy to make further adjustments as needed.

Q1: The paper is missing the citation of some important theoretical results from the field, e.g., causal discovery with continuous additive noise models.

A1: Thank you for raising this point. Upon carefully reviewing the mentioned results—which are indeed fundamental to causal discovery—we believe there may be some misunderstanding regarding the distinction between our work and existing causal discovery methods, such as those based on additive noise models (e.g., Peters et al., 2014). The key differences are as follows:

• In existing works, they only consider the general SEM with additive noise, without any measurement error. That is, $\mathbf{x}_i = f\_{1,i}(**Pa**(\mathbf{x}_i)) + \boldsymbol{\xi}_i$.

• In our work, we consider the general SEM with general noise, with nonlinear measurement error. That is, $\mathbf{z}_i = f\_{1,i}(**Pa**(\mathbf{z}_i), \boldsymbol{\xi}_i)$ and $\mathbf{x}_i = f\_{2,i}(\mathbf{z}_i) + \boldsymbol{\eta}_i$.

In summary, the "additive noise" in existing works refers to noise strictly within the SEM, whereas we allow for more general, nonlinear, and non-additive noise in the SEM while also addressing scenarios with additional nonlinear measurement error. Thus, we are actually dealing with a more general setting compared to existing works.

Q2: Lack of sufficient explanatory details of the theoretical constructs and manipulations.

A2: Thank you very much for your time and effort in reviewing our manuscript. We greatly appreciate your thoughtful feedback. After carefully considering your questions, we believe there may be potential misunderstandings, some of which likely stem from a typo that we can immediately clarify. Your constructive comment has been invaluable in helping us identify and correct this typo to improve the clarity of our work. We have addressed these concerns in the revised manuscript to ensure better understanding and avoid further confusion. Please find our detailed response below:

• Q2.1: The assumption of “Variability exists” seems to be much stronger than the assumption of “Domain Variability” (Kong et al. 2022), and it is not clear how restrictive it is.

• A2.1: Thanks a lot for raising this point. We sincerely appreciate you bringing this to our attention. We conjecture that the confusion stems from a typo in our original submission. Our intention was for the assumption of “Variability exists” to be equivalent to the “Domain Variability” assumption as defined in Kong et al. (2022). To address and rectify this, we have implemented the following revisions in our manuscript:

  • Notation:

    • Our domain variables are exactly the same as these in (Kong et al., 2022). To avoid potential misunderstanding, we have changed the notation from $\mathbf{d}$ to $\mathbf{u}$.

  • Corrected Typo:

    • Identical to (Kong et al., 2022), the assumption requires that for any set $A$ there exists two domains, and these domains can change across different sets of $A$. The original manuscript contained a typo in the assumption statement by switching ‘for any’ and ‘there exists’, which we have corrected in the updated manuscript:

    “ Suppose for any set $A \subseteq \mathcal{Z} \times \mathcal{E}$ …, there exist two domains $u_1$ and $u_2$ that are independent of $\boldsymbol{\epsilon}$ s.t….”

  • Enhanced Explanation:

    • To further prevent confusion, we have included the following statement next to the assumption:

    “The same as in (Kong et al., 2022), these two domains can differ for different values of $A$, providing great flexibility.”

    As a result, it is clearer that the assumption holds as long as the conditional probabilities do not cancel with each other over all pairs of domains. We have also highlighted it in the example, hoping that could also address your concerns:

    “... Let us consider the two domains where the integrals do not cancel with each other (the domains can change across different sets $A$):...”

  These updates enhance the overall clarity of our presentation and make it straightforward that the assumption of variability is natural, which is consistent with the conclusions in (Kong et al. 2022). We sincerely apologize for the oversight and any confusion it may have caused. Your feedback was instrumental, and we are grateful for your diligence in improving our manuscript.

Dear Reviewer 88C8,

Thanks so much for your prompt and insightful feedback. These new questions, again, helped us to further improve the manuscript, especially in presentation and clarification. We sincerely appreciate your time and effort. Please kindly find our point-by-point responses:

Q5: In the manuscript (Kong et al. 2022) there is an assumption that each domain $\mathbf{u}$ is the component-wise monotonic transformation of the latent variables $\mathbf{z}$. In (Kong et al. 2022) this is an important assumption, so they conclude some specific properties for the estimations, which makes sense. There is no such assumption in this manuscript that feels concerning.

A5: Thanks for raising this point. By “each domain $\mathbf{u}$ is the component-wise monotonic transformation of the latent variables $\mathbf{z}$”, if we understand correctly, perhaps you mean style variables $\mathbf{z}_s$ are generated from a high-level invariant part $\tilde{\mathbf{z}}_s$ by a component-wise monotonic function $f\_\mathbf{u}$, i.e., $\mathbf{z}\_s = f\_{\mathbf{u}} (\tilde{\mathbf{z}}\_s)$. The assumption is only used for the application in domain adaptation but not in the proof of identifiability in (Kong et al. 2022). Since our study does not address domain adaptation, this assumption is unnecessary for our theoretical results.

Please let us give a bit more details to also describe our understanding of the role of assumption in (Kong et al. 2022). In that work, for the purpose of domain adaptation, they need to find a high-level invariant representation across domains to learn an optimal classifier over domains. Thus, although $p(\mathbf{z}_s)$ change but $p(\tilde{\mathbf{z}}_s)$, $p(y|\tilde{\mathbf{z}}_s)$, and $p(y|\tilde{\mathbf{z}}_c)$ stay the same ($\mathbf{z}_s$ denotes changing variables while $\mathbf{z}_c$ denotes the invariant part). Specifically, if they assume that transformation is monotonic, then they can directly find the information of $\mathbf{z}_s$ in each domain, and thus domain adaptation is achieved. In our work, since the focus is not on domain adaptation, this treatment is not relevant to our setting or results.

Q6: I could not find in this manuscript how the function is defined except that it is an estimation of function f. How one can obtain such an estimation and what properties this estimation has?

A6: Thanks so much for your question. In light of it, in addition to the details in the experimental setup (L465-476), we have also added the following sentence earlier in the preliminary to avoid any potential confusion (L128-130):

“The estimated model $(\hat{f}, \hat{\mathbf{z}}, \hat{\boldsymbol{\epsilon}})$ follows the data-generating process and matches the observed distributions, i.e., $p(\hat{\mathbf{x}}) = p(\mathbf{x})$ ($p(\hat{\mathbf{x}}|\mathbf{u}) = p(\mathbf{x}|\mathbf{u})$ if there exists a domain variable $\mathbf{u}$).”

Please kindly let us know if you have any further suggestions. Thanks again for your valuable feedback.

Q7: Certain aspects of the discussion regarding the (Variability exists) assumption are unclear or potentially misleading.

A7: We sincerely appreciate your detailed reading and constructive suggestion, which helped us to further improve the manuscript to avoid potential confusion. You are totally right, that sentence could be misleading. Therefore, in light of your insightful suggestion, we have changed it as follows (L219-222) to highlight the specific condition that needs to be satisfied (note that we changed the name of the assumption):

“Differently, our theory does not put a hard constraint on requiring $O(n)$ domains, as long as the specific assumption of domain variability holds. However, since the conditions are different, the assumption of domain variability is not strictly weaker than the previous assumptions.”

We hope this clarification could address your concern and ensure the discussion is more precise. If you have further additional feedback, please feel free to let us know, and we would be more than happy to make corresponding adjustments.

Q8: If the assumption (Variability exists) is equivalent to the assumption (Domain Variability) (Kong et al. 2022), why do they have different names? It may be confusing.

A8: Thank you for your excellent suggestion. Accordingly, we have updated the terminology throughout the paper to use "Domain Variability" for consistency. While we initially chose the previous name to emphasize that the problem is not domain adaptation, we fully agree that having different names could be confusing. We sincerely appreciate your constructive feedback, which has helped us improve the clarity of our manuscript.

We greatly respect the depth of your review and the meaningful perspectives you offered. All of these constructive comments have not only improved the manuscript but also inspired us to think more deeply about the future challenges of the field. In light of these, we have incorporated new experiments, new discussions, and additional clarification in the updated manuscript. Our detailed responses are outlined below for your review. Please feel free to share any additional suggestions; we would be more than happy to make further adjustments as needed.

Q1: Explanation of several details.

A1: We sincerely appreciate all these insightful questions, which have helped us to further improve the presentation. Please kindly find our point-by-point responses below.

• Q1.1: Where in the proof the assumption that variability exists is used?

• A1.1: Thank you for your question. The variability assumption is utilized to demonstrate that the bottom-left block of the Jacobian, $\dfrac{\partial \boldsymbol{\epsilon}}{\partial \hat{\mathbf{z}}}$, is zero. As noted in the manuscript, this portion of the proof directly follows steps 1, 2, and 3 of the proof of Theorem 4.2 in (Kong et al., 2022), where the variability assumption is applied. In light of your great suggestion, we have now included the relevant result from (Kong et al., 2022) as Lemma 1 in Appendix B.1, along with the complete proof, adapted to align with our notation.

• Q1.2: How does the approach avoid the requirement of at least $O(n)$ environments? What are you leveraging to avoid that? Is it that you only require two environments to separate the noise from the latents or something else?

• A1.2: Exactly, we only require two environments to disentangle the noise and latent variables, where the dimensions of both can be significantly larger than two. As you pointed out, this approach is fundamentally more general than prior work that assumes at least $O(n)$ environments. Requiring only two environments essentially reduces to ensuring the presence of variability. This is not only practically significant but also theoretically intriguing, as it avoids introducing any additional assumptions. The relevant part of the proof can be found in L888-1057. Essentially, we found that the variability assumption, previously used to disentangle invariant variables from changing ones, can also imply the disentanglement of the changing part $\mathbf{z}$ from the invariant part $\boldsymbol{\epsilon}$ by leveraging the generative process where $\mathbf{z}$ is independent of $\boldsymbol{\epsilon}$, and thus establish block-wise identifiability.

• Q1.3: I think giving a proof sketch in the main paper is a far better use of space than some of the examples that were given.

• A1.3: Thanks so much for your great suggestion. In light of this, we have removed some real-world examples and added a proof sketch for the main result as follows:

  “Proof Sketch. We leverage distributional variability across two domains of the latent variables $\mathbf{z}$ to disentangle $\mathbf{z}$ and $\boldsymbol{\epsilon}$ into independent subspaces. To separate general noise from latent variables, we use the independence between $\mathbf{z}$ and $\boldsymbol{\epsilon}$ alongside the variability within $\mathbf{z}$. The structural sparsity condition is then employed to identify individual components of $\mathbf{z}$ in the nonlinear setting. Specifically, for each latent variable, the intersection of parental sets from a subset of observed variables uniquely specifies it. Since we only achieve relations among supports due to the nonparametric nature of the problem, an unresolved element-wise transformation remains. Consequently, we achieve element-wise identifiability for the latent variables $\mathbf{z}$ (Defn. 1).”