Revealing Hidden Causal Variables and Latent Factors from Multiple Distributions

We sincerely thank the reviewer for the time dedicated to reviewing our paper and the constructive suggestions. Our responses to these comments are given below.

Q1: "While the bulk of the paper establishes fundamental results for causal inference, this seems to have little impact on the design or guarantees of machine learning algorithms."

A1: Thanks for this insightful comment. The theoretical results have two major impacts on the design of machine learning algorithms: (1) learning $(\hat{g}, \hat{f},p_{\hat{Z}})$ to achieve Eq. (2), and (2) applying sparsity constraint on the edges of Markov network over $\hat{Z}$. We will make this clear in the revised manuscript.

Q2: "it is not clear to me how the data is generated, and the results shown in Figures 2 and 3 have little explanation. Hence it is quite challenging to understand what exactly the authors are trying to demonstrate, and how well their procedure performs."

A2: Thanks for pointing this out. We assume that the influences from domain and its parents are location and scaling transformations. Specificically, for each component $z_i$ in each domain, suppose the parents are $PA(i)$, the $z_i=\sum_{j\in PA(i)} s_{j,i}z_j+b_i$, where $s_{j,i}$ are scalars sampled from $Unif[0.5, 2]$ and $b_i\in Unif[-2,2]$. We use 100 domains such that the number of environments are sufficient. As for the nonlinear mixing function, we follow the standard nonlinear ICA literature and construct a MLP whose output dimension is the same as input dimension. Each layer of MLP consists of a orthogonal matrix weight and a leaky-relu function. Since the weights are orthogonal and the activation is invertible, the MLP is invertible too. For the Y-structure, we have $z_1\rightarrow z_3 \leftarrow z_2$ and $z_3 \rightarrow z_4$. As shown in the Fig. 3(a)(b), the estimated $\hat{z}_1$ recovers $z_2$ and $\hat{z}_1$ is independent from $z_1$; the estimated $\hat{z}_2$ recovers $z_1$ and $\hat{z}_2$ is independent from $z_2$. Therefore, our estimated $\hat{z}_1, \hat{z}_2$ recovers the $z_2, z_1$ and their independence relationship. As for $\hat{z}_3$, it is dependent on $\hat{z}_1$ and $\hat{z}_2$. Therefore, we have $\hat{z}_1\rightarrow \hat{z}_3 \leftarrow \hat{z}_2 \Leftrightarrow z_2 \rightarrow z_3 \leftarrow z_1$. As for $\hat{z}_4$, it is dependent on all previous variables, i.e., $\hat{z}_1, \hat{z}_2, \hat{z}_3$. Since we have the sparisty regularization, we can remove the redundant link from $\hat{z}_1$ and $\hat{z}_2$ to $\hat{z}_4$. Therefore, we are able to recover the true latent cause structure. In addition, in eq 6, we have an adjacency matrix to learn, the learned adjacency matrix is identical to the true adjacency matrix, which further supports our theory.

Q3: "I didn't understand the significance of the linearly independent vectors (see, e.g., bullet point 2 of Thm 1). Please explain its importance."

A3: The linearly independent vectors are crucial to the proof of the theoretical result, and is commonly used in the literature [1]. Specifically, it allows us to obtain Eqs. (11), (12), and (13) from the equation above them that involves linear combinations of the corresponding terms. Intuitively speaking, it requires the distribution to vary sufficiently across different domains. We will explain this in the revised manuscript.

Q4: "On page 3, should it read $Z_j \to X_i$ if and only if $Z_j \in PA(X_i)$?"

A4: Thanks for spotting this. It should read $Z_j\rightarrow Z_i$ if and only if $Z_j\in\textrm{PA}(Z_i)$, and we have fixed this typo in the revision.

Q5: "How should we imagine that $(\hat{g}, \hat{f}, p_{\hat{Z}})$ is learned to achieve Eq. (2)?"

A5: Thanks for this insightful question. We estimate a model $(\hat{g}, \hat{f},p_{\hat{Z}})$ that assumes the same data generating process as in Eq. (1) and matches the true distribution of $X$. We did not specify exactly how to achieve it, and leave the door open for different approaches to be used, such as normalizing flow or variational approaches. For example, we adopt a variational approach in Section 5. We have included this discussion in the updated manuscript.

We greatly appreciate the reviewer's constructive comments. We have tried to address all the concerns in the following.

Q1: "Is the condition (sufficient change) of theorem 1 too strict? What are some practical use cases for this?"

A1: Thanks for this question. The requirement of a sufficient number of environments is actually standard in the literature (see e.g. a recent survey [1]). Various real-world experimental results (e.g., visual disentanglement [2], domain adaptation [3], video analysis [4], and image-to-image translation [5]) in this line of research indicate that it is likely to hold in practice. Meanwhile, for many tasks, the number of environments is actually accessible, just like the number of domains in transfer learning and the number of time indices in time series data. We have updated the manuscript to include this discussion.

Q2: "The experiment section is limited, only on simulated data and very limited graphs."

A2: Thanks for pointing this out. Since the main focus of our manuscript is the theoretical identifiability results, our experiments mainly serve to validate our theory. The figure 3 shows that we are able to recover the true components and true causal strucuture when the components are causally related instead of mutually independent.

Q3: "The number of latents needs to be known in advance."

A3: Thanks for this comment. Although it is a rather standard assumption in causal representation learning to assume that the number of latents is known in advance (e.g., [1, 6]), let us discuss how to deal with number of latent variables in practice. Some possible approaches are (1) selecting the number of latent variables via model selection, or (2) use a mask to automatically select the number of latent variables, similar to [7]. We will include this discussion in the revised manuscript.

Q4: "What does identifying potential shifts even mean? Even if I can learn $\theta$ up to component-wise invertible transformations? What does knowing $\theta$ tell me? And I don’t think theorem 5 is verified by experiments either."

A4: $\theta$ denotes the latent (changing) factor (or effective parameters) associated with each latent structural equation model, which governs the changes in causal mechanisms. Knowing $\theta$ tells us whether a particular causal mechanism changes across certain domains.

Q5: "Can you explain how the last line of the proof of theorem 5 leads to the conclusion? Also where in the proof do you need the modular change assumption?"

A5: Thanks a lot for this question. The statement $\frac{\partial \theta_l}{\partial \hat{\theta}_i} \frac{\partial \theta_l}{\partial \hat{\theta}_j} = 0$ implies that each $\theta_l$ is a function of at most one of $\hat{\theta}_1,\dots,\hat{\theta}_n$. By considering the inverse of Jacobian matrix, we also have $\frac{\partial \hat{\theta}_l}{\partial \theta_i} \frac{\partial \hat{\theta}_l}{\partial \theta_j} = 0$, indicating that each $\hat{\theta}_l$ is a function of at most one of $\theta_1,\dots,\theta_n$. Combining both of them, $\theta_i$ are identifiable up to component-wise invertible transformation.

Furthermore, the modular change assumption is needed to obtain the equation below Eq. (16). We have modified the manuscript to make these clear.

Q6: "the sparsity condition is not made explicit in theorem 2? Can you state it formally?"

A6: The sparsity constraint implies the minimal number of edges of Markov network $\mathcal{M}_{\hat{Z}}$ over $\hat{Z}$. We have upadated the manuscript to make this explicit.

Q7: "not sure how to interpret the results of theorem 1? it is not intuitive? If the main purpose of theorem 1 is to prove later theorem, maybe it should be a lemma?"

A7: Thanks for the constructive suggestion. We have modified Theorem 1 to be a lemma in the revised manuscript.

Q8: "The proof of theorem 1 uses assumption 3 which is not assumed by theorem 1?"

A8: Thanks for your careful reading and spotting this. We have fixed the typo in the proof of Theorem 1. That is, the proof does not involve Assumption A3, which instead should be Assumption A2.

Q9: "what's the relation between modular change and ICM? why do you use the "not related via equality constraint"? What if $\theta_i$ and $\theta_j$ are related by some inequalities? Does you result still hold?"

A9: Thanks for asking this question. One can consider the modular change condition as an instantiation of ICM. We will discuss this conenction in the revised manuscript to explain clearly the relation between modular changes and ICM.

In our case, we have not been aware the situation where $\theta_i$ and $\theta_j$ are related by some inequalities. Intuitively, if it is possible for them to be related by inequality cosntraints, then in this case, one should include inequality constraints into Assumption A2.

Q10: "How is your multi-distribution setting different from soft interventions?"

A10: If soft intervention is defined this way as in [8], then it is completely general, as long as there is change in the conditional distribution. If so, then we assume soft intervention for all latent causal variables. We will discuss this in the revised manuscript.

References:

[1] Hyvärinen, Aapo, Ilyes Khemakhem, and Hiroshi Morioka. "Nonlinear independent component analysis for principled disentanglement in unsupervised deep learning." Patterns 2023.

[2] Khemakhem, Ilyes, et al. "Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica." NeurIPS 2020

[3] Kong, Lingjing, et al. "Partial Identifiability for Domain Adaptation." ICML 2022

[4] Yao, Weiran, et al. "Learning temporally causal latent processes from general temporal data." ICLR 2022

[5] Xie, Shaoan, et al. "Unpaired Image-to-Image Translation With Shortest Path Regularization." CVPR 2023.

[6] von Kügelgen, Julius, et al. "Nonparametric Identifiability of Causal Representations from Unknown Interventions." NeurIPS 2023

[7] Xie, Shaoan, et al. "Multi-domain image generation and translation with identifiability guarantees." ICLR 2022.

[8] Eberhardt, F., et al. "Direct Causes and the Trouble with Soft Interventions." Erkenntnis 2014.

Thanks for your time and effort in reviewing our manuscript. Below we give a point-by-point response to the comments.

Q1: The presentation might be misleading since it makes the impression that the approach is a generalisation of a single environment case.

A1: Thanks for your question. We fully agree with you that our method exploits the distributional change to identify latent causal variables and the structure among them. Actually, we thought it is exactly the considered task and didn’t treated it as a generalisation of the single environment case. In fact, the single environment case is way more difficult and most previous works in causal representation learning assume a sufficient number of different environments (e.g., [1,2]). Therefore, the multi-environments case is not a generalization of a single environment case. We have modified the introduction in order to avoid any potential confusion.

Q2: Lack of some definitions.

A2: Thank you for the reminder, and apologize for the potential confusion. The term “modular changes” is defined in A3 of Theorem 5 (“$\theta_i$ across different $i$ are not related via equality constraints”). The sparsity constraint implies the minimal number of edges of Markov network $\mathcal{M}_{\hat{Z}}$ over $\hat{Z}$. We have updated the theorem statement to make this explicit.

Q3: Related work on multiple environments via causality and nomalizing flows for causal discovery.

A3: Thanks for the suggestion. As detailed in A1, our theoretical results take the existence of multiple environments as an assumption of the identifiability theory for causal representation learning, which differs from previous works trying to address the challenge of multiple environments (e.g., CD-NOD [3]). And one can consider causal representation learning as an extension of causal discovery, which aims at identifying both the latent causal variables and hidden structures. We have incorporated the discussion and futher highlighted the differences in the manuscript as well as the recommended references.

Q4: Observations are assumed to be noiseless and the map is invertible.

A4: We appreciate the insightful questions. Let us apologize for the typo in the paragraph before Eq. 9. The map is actually injective but not invertible, which is a standard assumption in the previous work in causal representation learning [2,4]. Moreover, since most identifiability theorems in the literature stems from the theory of nonlinear ICA, many works follows the setting of it, where the observation $X$ is a deterministic (nonlinear) function of latent variables $Z$. Our work lies in this line of research and thus follows the previous setting. We will provide additional discussion on it to make the context clearer, according to your constructive suggestions.

Q5: More discussion on the assumption of sufficient changes.

A5: Thanks a lot for your great suggestions. In light of these, we have added more discussions in the updated manuscript, including more real-world examples. At the same time, we would like to mention that this assumption is common in the literature. For example, [4, 5] require $2n+1$ environments in order for the system to be identifiable, where $n$ is the number of variables. Some real-world experiments (e.g., visual disentanglement [6], domain adaptation [7], video analysis [8], and image-to-image translation [9]) also suggests that this assumption is likely to hold at least in certain real-world scenarios.

We sincerely thank the reviewer for the time devoted and the thoughtful comments. Please find the response to your comments and questions below.

Q1: Connections to previous works.

A1: Thanks for the question. We fully agree with you that the connection between independence and second-order derivatives of the log-density was the key to the identifiability of some functional causal models. At the same time, the task we are trying to address is different from causal discovery, since we need to first identify the latent causal variables and then discover the hidden structure, all from some general nonlinear mixtures of these causal variables, i.e., causal representation learning. To the best of our knowledge, our proof technique has never been introduced in this literature. Furthermore, as you correctly mentioned, we are dealing with a setting that might be more general than existing identifiable SCMs, since we do not have similar constraints on the functional relations and we allow both the causal variables and structures to not be observed. At the same time, these two settings are different since we are incorporating additional information such as sufficient changes in the distribution.

Q2: Clarification of theorems and notations.

A2: Thanks for your suggestions and sorry for the potential confusion and typos. We have incorporated the clarification into the updated manuscript. Specifically, we have made the following clarifications/corrections:

• (1) $\tilde{Z}$ should be $\hat{Z}$, which denotes the estimated latent variables.
• (2) $h(\cdot)$ denotes the mapping between the estimated latent variables $\hat{Z}$ and the true variables $Z$. This is also the case for $q(\cdot)$. For consistency, we have changed them to $h$ or $h^{-1}$.
• (3) $m$ should be $u$, which denotes the auxiliary variable.
• (4) $\theta$ represents the latent factor that govern the change of hidden causal mechanisms. $u$ in Eq. 3 denotes the auxiliary variable.
• (5) For Theorem 1, we do not need to assume that $g$ is invertible as long as the function between estimated latent variables $\hat{Z}$ and the true variables $Z$ (i.e., $h$) is invertible, since the change of variable formula only applies between $p(\hat{Z};\theta)$ and $p(Z;\theta)$. Similar to previous work (e.g., [1,2]), $g$ only needs to be injective. We have corrected it in the manuscript.

Apologies again for the potential confusion. We sincerely appreciate your detailed suggestions, which are very helpful in improving the clarity of our manuscript.

Q3: Realisticness of the assumption.

A3: Thanks for your question. As mentioned in A2(5), we do not assume that $Z$ and $X$ are of the same dimension, similar to many previous works in causal representation learning (e.g., [1, 2]). And the determination of the dimension in practice is a typical model selection problem. In practice, one may extend model selection method such as cross-validation or sparsity regularization on a mask to determine the number of dimension of $Z$ [9]. The requirement of a sufficient number of environments is also standard in the literature (see e.g. a recent survey [3]). Various real-world experimental results (e.g., visual disentanglement [5], domain adaptation [6], video analysis [7], and image-to-image translation [8]) in this line of research indicate that it is likely to hold in practice. Meanwhile, for many tasks, the number of environments is actually accessible, just like the number of domains in transfer learning and the number of time indices in time series data. We have updated the manuscript to hightlight the related discussion.

Q4: Experiments and evaluations.

A4: Thanks for pointing this out. We assume that the influences from the domain and its parents are location and scaling transformations. Specificically, for each component $z_i$ in each domain, suppose the parents are $PA(i)$, the $z_i=\sum_{j\in PA(i)} s_{j,i}z_j+b_i$, where $s_{j,i}$ are scalars sampled from $Unif[0.5, 2]$ and $b_i\in Unif[-2,2]$. We use 100 domains such that the number of environments is sufficient. As for the nonlinear mixing function, we follow the standard nonlinear ICA literature and construct an MLP whose output dimension is the same as the input dimension. Each layer of MLP consists of an orthogonal matrix weight and a leaky-relu function. Since the weights are orthogonal and the activation is invertible, the MLP is invertible too. For the Y-structure, we have $z_1\rightarrow z_3 \leftarrow z_2$ and $z_3 \rightarrow z_4$. As shown in the Fig. 3(a)(b), the estimated $\hat{z}_1$ recovers $z_2$ and $\hat{z}_1$ is independent from $z_1$; the estimated $\hat{z}_2$ recovers $z_1$ and $\hat{z}_2$ is independent from $z_2$. Therefore, our estimated $\hat{z}_1, \hat{z}_2$ recovers the $z_2, z_1$ and their independence relationship. As for $\hat{z}_3$, it is dependent on $\hat{z}_1$ and $\hat{z}_2$. Therefore, we have $\hat{z}_1\rightarrow \hat{z}_3 \leftarrow \hat{z}_2 \Leftrightarrow z_2 \rightarrow z_3 \leftarrow z_1$. As for $\hat{z}_4$, it is dependent on all previous variables, i.e., $\hat{z}_1, \hat{z}2, \hat{z}_3$. Since we have the sparsity regularization, we can remove the redundant link from $\hat{z}_1$ and $\hat{z}_2$ to $\hat{z}_4$. Therefore, we are able to recover the true latent cause structure. In addition, in eq 6, we have an adjacency matrix to learn, the learned adjacency matrix is identical to the true adjacency matrix, which further supports our theory.

Q5: The necessity of learning latent SCMs and the nonlinear mixing function.

A5: We appreciate the great question, and would like to briefly summarize the discussion as follows:

• (1) Latent causal variables are crucial because they represent unmeasured or unmeasurable influences. For example, in social science research, a latent variable might represent an abstract concept (e.g., peer support in teacher’s burnout study [4]) that influences but is not directly measurable through observed variables.
• (2) It is common to have dependencies among those abstract concepts in practice, thus allowing an SCM among them (which also covers the empty graph) instead of assuming all variables are independent is much more general and can be applied in more real-world scenarios.
• (3) In addition, in order to identify the latent SCM, we need to first identify those latent causal variables $Z$ from only observed variables $X$. Since we do not want to put strong constraints on the relations between $Z$ and $X$, we just assume that the function $g$ in $x = g(z)$ is nonlinear, which is the nonlinear mixing function.

We have updated the manuscript in order to better highlight the practical significance of the considered general setting.

Q6: Why is $Z_i$ a function of the $\hat{Z}_k$ and $\hat{Z}_l$? What is $\frac{\partial Z_i}{\partial \hat{Z}_k}$?

A6: Thanks for pointing out this very interesting finding. In order to identify those latent causal variables $Z$, we would like to make sure that our estimated variables $\hat{Z}$ recovered the ground-truth $Z$ up to some mild indeterminacies. Without that guarantee, $\hat{Z}_k$ can be a mixture of an arbitrary set of latent causal variables in $Z$, making the identification less meaningful. This is similar to the task of disentanglement and we provided theoretical guarantees of the disentangled results. $\frac{\partial \hat{Z}_i}{\partial Z_k} \neq 0$ means the estimated variable $\hat{Z}_i$ is not a function of $Z_k$, removing $Z_k$ from the mixture.

References:

[1] Buchholz, Simon, et al. "Learning Linear Causal Representations from Interventions under General Nonlinear Mixing." NeurIPS 2023

[2] Khemakhem, Ilyes, et al. "Variational autoencoders and nonlinear ica: A unifying framework." ATSTATS 2020

[3] Hyvärinen, Aapo, Ilyes Khemakhem, and Hiroshi Morioka. "Nonlinear independent component analysis for principled disentanglement in unsupervised deep learning." Patterns 2023.

[4] Byrne, Barbara M. Structural equation modeling with Mplus: Basic concepts, applications, and programming. routledge, 2013.

[5] Khemakhem, Ilyes, et al. "Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica." NeurIPS 2020

[6] Kong, Lingjing, et al. "Partial Identifiability for Domain Adaptation." ICML 2022

[7] Yao, Weiran, et al. "Learning temporally causal latent processes from general temporal data." ICLR 2022

[8] Xie, Shaoan, et al. "Unpaired Image-to-Image Translation With Shortest Path Regularization." CVPR 2023.

[9] Xie, Shaoan, et al. "Multi-domain image generation and translation with identifiability guarantees." ICLR 2022.

Dear Review mmWQ,

Many thanks for the feedback! In our setting, X might have higher dimensions and redundant information (for instance, consider image pixels). Consider the following example in the simple linear case. Let