Learning Linear Causal Representations from General Environments: Identifiability and Intrinsic Ambiguity

We thank the reviewer for providing insightful feedback and suggestions. Below are our responses to the reviewer’s questions and concerns of our paper.

Q1: What does "general environments" mean? I don't think it is a widely used concept, so perhaps describe it more explicitly and accurately.

As the reviewer notices, the “general environments” setting is relatively new in the literature. As a result, we include discussions of this concept in several parts of our paper. The most notable one might be at the beginning of Section 4, where we wrote that “In this section, we consider learning causal models from general environments. Specifically, we assume that the environments share the same causal graph, but the dependencies between connected nodes (latent variables) are completely unknown, and, in contrast with existing literature on single-node interventions, we impose no similarity constraints on the environments.” The motivation of considering such a setting is discussed in Section 1 line 41-51. We thank the reviewer for pointing this out and we will improve exposition by moving the definition from Section 4 to the introduction so that the reader has a more precise understanding of the use of the term earlier on.

Q2: Definition 2: this definition is questionable, or at least inaccurate. Line 105: here "$i\in S$" and then "for all $i\in[d]$"? What is $S$ and what is the difference between $S$ and $[d]$? Further, when you write $\Leftrightarrow$, what does $i\notin S$ indicate?

Thank you for pointing at potential points of confusion/improved readability and let us clarify below the accuracy of the statements and ways that we plan to address the readability issues.

"i\in S" is part of the sentence “......a subset of latent variables $z_i,i\in S$”, so it describes for which zi’s we are defining the soft intervention. On the other hand, "if for all $i\in[d]$" belongs to the sentence that follows: if $\forall i\in[d]$, $\forall E_1, E_2 \in \hat{\mathfrak{E}}, E_1 \neq E_2$, we have $p_i^{E_1}=p_i^{E_2} \Leftrightarrow i\notin S$. So these two $i$’s are essentially the subscripts of different objects: latent variables $z_i$ and distribution $p_i$, not contradictory statements or typos. We will use different subscripts in the two sentences for improved readability in a revised version.

The notation $\Leftrightarrow$ stands for “if and only if”. We will replace the arrow with the if and only if text to make it clearer in a revised version.

So, to summarize, Definition 2 can be rephrased as follows: “We say that a collection of environments $\hat{\mathfrak{E}}$ is a set of (soft) interventions on a subset of latent variables $z_i, i\in S$ if the following statements hold for any $i\in[d]$: $\forall E_1, E_2 \in \hat{\mathfrak{E}}, E_1 \neq E_2$, we have $p_i^{E_1}=p_i^{E_2}$ if and only if $i\notin S$.

Q3: Similarly in Def. 4, line 121: it is difficult to interpret what you mean by <=>. Can you give a complete sentence of the condition?

Similar to Q2, we use the mathematical convention that <=> means if and only if. The sentence reads “For all $i,j\in[d]$, $i\in pa_{G}\left(j\right)$ if and only if $\pi(i)\in pa_{G}\left(\pi(j)\right)$. We will replace the arrow with the "if and only if" text for improved readability.

Q4.1: a motivation of considering soft interventions is that, ”the latent variables are unknown and need to be learned from data, it is unclear how to perform interventions that only affect one variable. “, but how do soft interventions avoid this concern?

We are afraid that the reviewer mixed up the following two settings: one is the “general environment” setting, which is the main focus of our paper (Theorem 1, 3 and the LingCReL algorithm), the other is the “soft intervention” setting, for which we only include a negative result (Theorem 2) to highlight its difference from the general environment setting. In fact, soft interventions exactly inherit the limitations that the reviewer quoted, and this is the main motivation for us to consider general environments that do not have this limitation. We included the soft-intervention result in our paper, as it only makes the negative result stronger. Obviously the negative impossibility result holds for general environments too, but proving it for soft interventions only strengthens the impossibility result. All our positive and constructive results are for general environments, which exactly bypass the requirement that the reviewer describes as a problem.

Q4.2: lines 123-124： "if there exists some i ∈ surG(j), then ambiguities may arise for the causal variable at node j, since any effect of j on any of its child k can also be interpreted as an effect of i." in pearl's book or the topic of mediation analysis, you can define clearly the effect of 𝑗 and the effect of 𝑖 on child 𝑘, and why would I interpret the effect of 𝑗 as that of 𝑖? That is not sound.

We thank the reviewer for pointing to the references. Note that this sentence is only an intuitive description and serves as a supplement to Definition 3, which is mathematically rigorous.

In this section, the word “interpret” is used in the following sense: from the learner’s perspective, the task is to find a suitable data model that matches the data. intuitively, there would be two indistinguishable but non-equivalent data generating processes, so in other words, there are two distinct causal mechanisms that match observation data. So you can of course define these theoretical causal quantities, but given the data you cannot conclude which of these two causal effects are at play. We hope this explanation would make things clear, and we point the reviewer to Definition 3 for a more rigorous mathematical formulation.

Due to space limits, the remaining parts of the rebuttal can be found in the official comment that follows this post.

Q4.3: I also have a question with the example with three causal variables in Appendix E, which aims to show the difference between hard and soft interventions. It seems that one can distinguish the two even with soft interventions, e.g., by setting $z_3=z_2+\epsilon_3$, $v_3=v_2+\epsilon_3$, then $z_2$ and $z_3$(or $x_2,x_3$) would still be dependent while in the right, $v_2$ and $v_3-v_2$(or $x_2,x_3$) are independent.

We thank the reviewer for raising this question. The reason why the two models in Example 3 cannot be distinguished is that, given any soft intervention on the first model, there always exists a soft intervention on the second one that yields the same data distributions. This is not to say that any two soft interventions on two models would yield the same data distribution – which is actually impossible. Thus, by constructing two soft interventions separately for the two candidate models and proving that they are different, one cannot conclude that these two models are distinguishable. In practice this translates to: "given the data that I have, I cannot distinguish whether they were generated by a true structural causal model SCM1 under an intervention SoftIntervention1 or whether it was SCM2 with intervention SoftIntervention2." The statement of Theorem 1 includes a precise statement of what “undistinguishable models” mean in our paper (a formal definition also used in prior work).

Returning to the reviewer’s example, a soft intervention $z_3=z_2+\epsilon_3$ on the first model is equivalent to $v_3=2*v_2+\epsilon_3$ on the second one. The key point here is that we are not assuming that the experimenter knows the specific form of the interventions, nor the nodes on which the interventions are performed (a commonly considered setting in CRL, see e.g. [3,4]). Thus, from the experimenter’s perspective, these two causal models cannot be distinguished.

[3] Squires, Chandler, et al. "Linear causal disentanglement via interventions." International Conference on Machine Learning. PMLR, 2023.

[4] Varıcı, Burak, et al. "Linear Causal Representation Learning from Unknown Multi-node Interventions." arXiv preprint arXiv:2406.05937 (2024).

Q4.4: lines 139-142: "in contrast with existing literature on single-node interventions, we impose no similarity constraints on the environments.": however, assumptions 4 and 5 seem to assume the environments are sufficiently diverse, which are not weaker.

Our Assumption 4 and 5 can be easily satisfied by single-node soft interventions. Indeed, soft interventions on node $i$ is equivalent to changing the entries of the $i$-th row of $B_k$. As a result, as long as there exists at least $i$ soft interventions in general positions for node $i$ (which is $\Theta(d^2)$ interventions in total), our assumption can be automatically satisfied. As a result, while our assumption allows for identification from diverse environments, it is not restricted to this case; it also holds for soft interventions as well. On the other hand, we also show in Theorem 2 that the number of interventions $\Theta(d^2)$ cannot be reduced, thereby providing a clear picture of the soft intervention setting.

We would like to stress that, however, the main motivation of considering general environments is that single-node soft interventions are too restrictive for applications. Hence, the focus of our paper is the setting of general environments, and we do not compare with the restrictive setting of soft interventions explicitly in the main text. We will add it in a revised version of the paper.

We hope that the above explanations are helpful, and we are also happy to answer further questions that the reviewer has.

We thank the reviewer for the positive feedback and comments. In the following, we respond to the questions raised by the reviewer.

Q1: Figure 2e looks like an undirected graph but on zooming looks like there might be directed edges. If so, this figure can be corrected.

That was indeed the case – the graph is directed. We thank the reviewer for pointing out this issue, and will correct them in a revised version.

Q2: Font size for the axis labels is too small to read in Fig 2a-2d

We thank the reviewer for pointing out this issue. We will update these labels to make them bigger.

Q3: Apart from the interventional CRL works already cited which mainly focuses on identifiability, the authors could also cite BIOLS [1] which studies CRL from a more empirical perspective and proposes an algorithm to learn linear latent causal graphs from high-dimensional data under hard, multi-node interventions.

We thank the reviewer for pointing to this super interesting and closely related paper, and we will definitely cite it in the revision.

We thank the reviewer again for appreciating our work and are happy to address any other questions that the reviewer might have.

We thank the reviewer for providing insightful comments and feedback. Below are our responses to the reviewer’s questions and concerns of our paper.

Q1: Could you provide an intuitive explanation or motivations for the assumptions? What do they represent in real-world data scenarios?

Certainly. Our paper considers the task of learning linear representations from general environments. Assumption 1 states that all environments share the same mixing function, which is a standard assumption in CRL literature [1,2]. For example, the generating process of images can be thought of as first generating a few causally related features (scheme, location, weather, etc.) and then these features are transformed by some mixing function into high level representations (pixels in images). Assumption 1 basically states that the relationship between features and the final image remains the same for all environments, but that those features are generated with different probabilities. Assumption 2 requires that the mapping from features to images is injective, i.e. there can’t be two different feature vectors that correspond to the same image. Assumption 3 requires the noise variables to be non-Gaussian and have different distributions. The non-Gaussian assumption is relatively standard in causal graph discovery, while requiring different distributions is not restrictive, since real-world noise distributions are seldom the same. Assumptions 4 and 5 are the main assumptions in our theory and intuitively state that the environments should contain enough information for all latent variables. Notably, it does not require single-node soft interventions, a widely-adopted assumption [1,4,5] but is questionable in real-world applications such as genomics (see e.g. Figure 1 of [3]), while interventions typically affect multiple nodes.

[1] Squires, Chandler, et al. "Linear causal disentanglement via interventions." International Conference on Machine Learning. PMLR, 2023.

[2] von Kügelgen, Julius, et al. "Nonparametric identifiability of causal representations from unknown interventions." Advances in Neural Information Processing Systems 36 (2024).

[3] Tejada-Lapuerta, Alejandro, et al. "Causal machine learning for single-cell genomics." arXiv preprint arXiv:2310.14935 (2023).

[4] Wendong, Liang, et al. "Causal component analysis." Advances in Neural Information Processing Systems 36 (2024).

[5] Varici, Burak, et al. "Score-based causal representation learning with interventions." arXiv preprint arXiv:2301.08230 (2023).

Q2: The authors mentioned that this work differs from Xie et al. [54, 55] and Dong et al. [11] because the latter requires structural assumptions. However, it seems that the proposed model in this paper also inherently assumes that there are no direct causal edges between observed variables.

We thank the reviewer for pointing out the possible confusion here, and we will modify the sentence in a revised version. Our point is that since these works rely on observational data only, the underlying causal models may at best be recovered up to Markov equivalence. If we hope to establish stronger identification guarantees, more structural assumptions must be made.

Q3: In the first step of the proposed algorithm, "any identification algorithm for linear ICA is used to recover the matrix $M_k$". This may introduce some errors, as perfect identification cannot be achieved. These methods typically assume that the dimensions of $X$ and $Z$ are the same and other restrictions.

As the reviewer noticed, standard linear ICA only applies to the case where $X\in\mathbb{R}^n$ and $Z\in\mathbb{R}^d$ have the same dimension. If $n>d$, then this means that our observation $X$ contains redundant information, and we can naively ignore the last $n-d$ components of $X$. Alternatively, we can run PCA to identify the top $d$ components of observation $X$ and use the new $d$-dimensional vector to run linear ICA. We will make this point more precise in a revised version of the paper.

Q4: In Figure 2(e), the arrows on the edges could be made larger, as they currently look like undirected edges.

We thank the reviewer for pointing out this issue and will make the arrows larger in an updated version.

Q5: How would the proposed method perform when applied to real-world data?

The experiments conducted in this paper mainly aims to verify the correctness of our theory and algorithm. We have not tested it on real-world data, because it is unclear how to evaluate the performance, since the underlying causal structure is generally unknown, and there is no benchmark in our setting. We agree with the reviewer that dealing with real-world data is an important future direction of CRL, and we will definitely explore it in future studies.

Q6: The authors only provided the results of the LiNGCReL algorithm on simulated data. It would be more objective and validate the effectiveness of the proposed method if results of other baselines or classical methods on the same data were also provided for comparison.

To the best of our knowledge, LiNGCReL is the first algorithm that can handle CRL problems given data from multiple environments. We are not aware of any other algorithms that work for this task.

Q7: The paper considers the problem of learning linear causal representations from general environments. Is the information about these different environments known? If so, it indeed provides some additional information for structure learning.

We assume that the environments are unknown; the weights in the linear causal models can be arbitrarily different across different environments.

We hope that the above explanations have properly addressed the reviewer’s questions, and feel free to let us know if the reviewer has any other questions.

We thank the reviewer for appreciating our work and for giving insightful comments. Below are our responses to the questions and weaknesses mentioned by the reviewer.

Q1: All environments share the same causal graph. My understanding of this is that soft interventions considered in this work do not allow for removal of a subset of the parents…… Regarding my comment above on "all environments share the same causal graph", if my understanding is correct, what's the main challenge of allowing partial removal of parents during a soft intervention?

We are sorry for the confusion here. There are actually three different settings that existing works consider:

1). single-node, hard interventions. Hard intervention means removing the edge between the intervened node and all its parents.

2). single-node, soft interventions. This means changing the weights of the edges between the interveded node and its parents.

3). general environments (a.k.a. soft interventions that allow simultaneously intervening an arbitrary number of nodes. This is more general than 2) and is the setting this paper focuses on.

While we highlight in our paper that we can deal with the case where the weights are nonzero, implying that the causal graph does not change, our setting does allow some weights to be zero. This is because our Assumption 4 (and 5) only requires a non-degeneracy condition, and this condition does not necessarily require all weights to be nonzero. for example,let there by a three-node graph with edges $1\to 3$ and $2\to 3$, and we have access to three environments $E_i,i=1,2,3$ where $E_1$ has nonzero weights for both $1\to 3$ and $2\to 3$, $E_2$ only has nonzero weights for the former edge and $E_3$ only has nonzero weights for the latter edge, then Assumption 4 and 5 are satisfied on node $3$ since the third rows of the $B_i$’s are of forms $(x_1,x_2,1),(y_1,0,1),(0,z_2,1)$ (where $x_1,x_2,y_1,z_2$ denotes some nonzero numbers) and they span $\mathbb{R}^3$. We will make this point more explicit in a revised version of our paper.

Q2: L113-114: "One may expect that identifiability with soft interventions is not much different from hard interventions, since soft interventions can approximate hard interventions with arbitrary accuracy". I am not really sure about this sentiment on soft interventions, I particularly would expect the identifiability to be different.

We thank the reviewer for pointing this out, and we will delete this sentence to avoid unnecessary confusion.

Q3: In line 133, it reads "each latent variable $v_i$" but $z_i$ was used to denote latent variables. L627 in appendix, typo "must stronger". In the model setup in eq.(3), you might want to index the variables z per environment too since these are not exactly the same r.vs. I think you want to cite the proceedings version of Squires et al 2023 and not the arxiv version from 2022.

We are sorry for these issues and will fix it in a revised version.

Q3: In Theorem 1, v is used to denote the candidate/hypothetical latent variables, wouldn't it be easier to use z^? This would keep the consistency with H^, A^, etc.

The reviewer is correct about this. We will change the notations in the revision.

Q4: In Definition 10 in the appendix, what is $\phi$? I also think there might be a better way to state Definition 4 instead of relying on a definition in the appendix as early as in page 3.

We are very sorry for the typo. We restate this definition below:

We write $(h,G)\sim_{CRL}(\hat{h},\hat{G})$ if there exists a permutations $\pi$ on $[d]$, and a diffeomorphism $\psi:R^d\to R^d$ where the $j$-th component of $\psi$, denoted by $\psi_j(z)$, is a function of $z_j$ for $\forall j\in[d]$, such that the following holds: For $\forall i,j\in[d]$, $i\in pa_{G}\left(j\right) \Leftrightarrow \pi(i)\in pa_{\hat{G}}\left(\pi(j)\right)$, $P_{\pi}\circ\hat{h}=\psi\circ\hat{h}$, where $P_{\pi}$ is a permutation matrix satisfying $(P_{\pi})_{ij}=1 \Leftrightarrow j=\pi(i)$.

Intuitively, this means that one can find a permutation of the nodes, such as the two models have node-wise correspondence.

We thank the reviewer again for the positive feedback and are happy to address any other questions that the reviewer might have.