Identifying General Mechanism Shifts in Linear Causal Representations

Thank you for your efforts on reviewing our paper! Below, we address your concerns.

From weakness

1. We provided some motivation from Line 34 to Line 38 in the paper, and also included a toy example in Figure 1. Additionally, we provided a Psychometrics data application (Section 5.2) to reflect the motivation and practical significance of our method. In addition, we would like to provide more potential applications here to illustrate in which scenarios one might be interested in localizing shifted nodes.

   For example, examining the mechanism changes in the gene regulatory network structure between healthy individuals and those with cancer may provide insights into the genetic factors contributing to specific cancers. Within biological pathways, genes could regulate various target gene groups depending on the cellular environment or the presence of particular disease conditions [1,2]. In the analysis of EEG signals [3], it is of interest to detect neurons or different brain regions that interact differently when the subject is performing different activities. All these questions can be formulated as lozalizing shifted nodes.

   The complementary question of finding shifted nodes is to find the shared non-shifted nodes, which can also be used for social science analysis. For instance, we may have data from different countries and different ethnicities but aim to identify the potential causes that influence the years of education pupils receive [4]. Based on the applications in these fields, I believe that estimating shifted nodes in CRL is an interesting and valuable topic.

   [1] Hudson, N. J., Reverter, A., & Dalrymple, B. P. (2009). A differential wiring analysis of expression data correctly identifies the gene containing the causal mutation. PLoS computational biology.

   [2] Pimanda, J. E., Ottersbach, K., Knezevic, K., Kinston, S., Chan, W. Y., Wilson, N. K., ... & Göttgens, B. (2007). Gata2, Fli1, and Scl form a recursively wired gene-regulatory circuit during early hematopoietic development. Proceedings of the National Academy of Sciences.

   [3] Sanei, S., & Chambers, J. A. (2013). EEG signal processing. John Wiley & Sons.

   [4] Ghassami, A., Salehkaleybar, S., Kiyavash, N., & Zhang, K. (2017). Learning causal structures using regression invariance. Advances in Neural Information Processing Systems, 30.

2. Assumption B is included for better presentation and understanding of our method's workflow, but it is not necessary for our identifiability results. As mentioned in line 148 and Appendix C, there are alternatives to this assumption. With estimated noise samples, we can perform distribution matching to achieve a consistent order of noise components. Therefore, this assumption is not necessary.

   Regarding sign and scaling ambiguity, no, we do not fix sign ambiguity in our assumptions, as ICA inherently has sign ambiguity. In Theorem 3, we prove that the unmixing matrix estimated from the ICA method, even with sign ambiguity, will not affect our estimation of shifted nodes. Additionally, in Theorem 1, it is proven that under certain assumptions, the ICA solution only has permutation and sign ambiguity, so there should be no scaling ambiguity.

   ICA is an important component of our method. The main contribution of our paper is to prove that the ICA solution can help us detect the shifted latent nodes. We also provide the identifiability result for the algorithm designed based on ICA. We believe this is not trivial since we are solving a new problem with theoretically backed methods.

3. Thank you for the question. We refer you to the global response for a detailed answer to this question. In short, in the "inadequate" intervention data setting (small number of intervention environments, inadequate number of interventions, and general intervention setting), estimating $G$ and $B$ from $M$ is impossible. Our method bypasses the step of estimating $B$ but can still estimate shifted nodes with fewer assumption constraints.

From questions

Thank you for the question. Assuming interventional distributions is a modeling choice, which may or may not hold in practice. Our setting is concerned with identifying the sources of distribution changes (shifted nodes) across two or more populations. In this case, one formal way to characterize distribution changes is through modeling interventions in a causal graph. We do not think one must assume this but we believe it can be a reasonable assumption in multiple scenarios.

We hope our answers will make you feel more positive about our work. Please let us know any follow-up questions!

We thank the reviewer's valuable suggestions and for recognizing the novelty of our work. We next address the reviewer's concerns.

From weaknesses

• Please refer to our global response regarding linear models.

• Thanks for pointing this out. We apologize for the typo in Proposition 2, line 187. It should state that "$Z_i$ is identified as a not shifted latent node between $k$ and $k'$ if and only if $M_i^{(k)} = M_i^{(k')}$", and Theorem 3 is correct. We now believe Proposition 2 and Theorem 3 are consistent after correction.

  Additionally, in Section 4.3, Step 3, line 213, it should read "$Z_i$ is a non-shift node between $k$ and $k'$ if and only if $\widetilde{M_i}^{(k)} = \pm \widetilde{M_i}^{(k')}$", and in line 215, it should state "there is no shift in node $Z_i$ if and only if $\widetilde{M_i}^{(k)}=\pm \widetilde{M_i}^{(k')}$". We will correct these in the revision. Thank you again for your careful review.

• Correct. Thanks for pointing this out. We will correct it in the revision.

From questions

• As we pointed above, there are some a couple of typos in Proposition 2 and Section 4.2, Step 3. After correction, as you suggested, the illustration of our method should be consistent. A node $Z_i$ is a shifted node if and only if $M_i^{(k)} \neq M_i^{(k')}$, which means that the $i$-th row of $M^{(k)}$ and $M^{(k')}$ are different, or $\widetilde{M_i}^{(k)} \neq \pm \widetilde{M_i}^{(k')}$.

• Great question! This is a non-trivial question that makes for an exciting future direction. It is important to note that the correlated noise assumption impacts the identifiability of $B^{(k)}$. Its effect on the ICA solution would come secondary. For our setting and for CRL in general, it is crucial that $B^{(k)}$ is unique to avoid identifiability issues. If we assume the noise component is correlated, it implies that $\Omega$ is not a diagonal matrix, and it is also possible that $\Omega$ is not full rank. Given that $\Omega^{1/2}B = I - A$, if $\Omega$ is not full rank, then $B$ is not unique.

We hope our answers will make you feel more positive about our work. Please let us know any follow-up questions!

We thank you for your efforts in evaluating our paper. Below, we address your concerns.

From weaknesses

1. In the population setting, step 3 will lead to identifiability provably, and it is proven in Theorem 3 that $L_i^{k,k'} = 0$ if and only if $Z_i$ is not a shifted node between environments $k$ and $k'$. However, in the finite sample setting, since we have no precise measure of the accuracy of the ICA estimation, the choice of $\alpha$ is heuristic.

From questions

• Yes, if we can estimate $B^{(k)}$ by the method in [1], SNA ambiguity does not affect the identifiability of our task. However, estimating $B^{(k)}$ up to SNA requires two assumptions: the number of environments $K$ should be at least equal to the number of latent nodes $d$, and there should be at least $\Theta(d^2)$ interventions. Our algorithm does not rely on these two assumptions. For example, for any number of latent nodes, our method can localize mechanism shifts even when given only $K=2$ environments.

• Even if we were able to estimate $B^{(k)}$, if the objective is to identify shifted nodes, estimating less can be more efficient. Our paper shows that estimating $B^{(k)}$ is not necessary at all for achieving that objective.

We will add these comments in the revision to emphasize the difference to [1].

We hope our answers will make you feel more positive about our work. Please let us know any follow-up questions!

Thanks for the reviewer's recognition of our paper's novelty and contribution. We now address the reviewer's concerns.

From Weaknesses

Significance of contributions...

Please refer to the global response for this concern.

Experiments...

The most recent CRL method with official code released is [2]. In their simulation, their official code is based on all Gaussian noise and samples data precision matrices from the Wishart distribution. However, in our case, our noise setting is non-Gaussian, requiring the calculation of the data precision matrix from samples. Since the estimation of the precision matrix involves the inverse of a not full-rank matrix, their code becomes numerically unstable. Thus, it is hard to accommodate their code in our setting.

Instead, as you suggested, it is a good idea to compare the existing methods of directly finding causal mechanism shifts in a fully observable setting. We compared with methods DCI [3], as they are the most recent methods with official codes publicly available. The results for the simulation are shown in the table of pdf. Our method achieves higher performance in most settings.

[2] Squires, C., et al. Linear causal disentanglement via interventions.

[3] Wang, Y., et al. Direct estimation of differences in causal graphs

Sample complexity....

Please refer to our gloabl response.

Real world experiment...

1. We believe that our experimental setting is relatively high-dimensional and involves larger graph sizes compared with existing CRL methods. For example, [2] considers settings where $d = 5$ latent nodes, [3] considers settings with $d = 5, 10$, and [4] considers settings where $d = 5, 8, 10$. In contrast, our experiments consider $d$ ranging from $5$ to $40$, which represents a relatively high-dimensional graph compared to existing baseline methods.

   [4] Jin, J., et al. "Learning causal representations from general environments: Identifiability and intrinsic ambiguity."

2. The main contribution of our paper is to provide an identifiability result to detect the shifted nodes under fewer assumptions. Thus, we believe that the experimental part is sufficient to validate our method and proof.

3. It is challenging to find CRL datasets, under which the detected shifted nodes can also be validated by scientific papers.

Limitations

Our method can be applied to any number of latent nodes, and our identifiability results holds. As we explained in previous responses, datasets with large number of latent nodes are hard to obtain, and we have already conducted synthetic experiments on relatively large latent graph sizes compared with other CRL methods. We will add more discussion about this in the limitations section of the revision.

From Questions

Line 56...

Thank you for your suggestion,we will add more discussion about it in our revision. We note that there are differences between our problem setting and the one in the mentioned paper, even in the direct observation setting. Using notations in [1], their goal is to find the SEM $A$ from the equation $X = C(I + \bar{A})$, where $(I - A)(I - \bar{A})=I$. They assume that $C$, the root cause, is sparse.

In contrast, our objective is to identify the difference in $A$ across different environments, where the difference in $A$ can be sparse. The sparsity in the difference of $A$ between different environments does not necessarily imply a sparse root cause $C$. To elaborate further, suppose we have datasets from two environments, $X^1 = A^1 X^1 + N$ and $X^2 = A^2 X^2 + N$. When we take the difference, we can consider $X^1 - X^2$ as arising from the process $X = (A^1 - A^2)X + N'$. Even though $B = (A^1 - A^2)$ may be sparse, $(I + \bar{B})^{-1}$ is not necessarily sparse.

Therefore, while the question addressed in that paper is relevant, it is not exactly the same as ours. We will add the citation and include a brief discussion about the differences between our work and that paper in the revision.

Fig1 caption...

Thanks for point this out. We will correct them in the revision.

L143...

This assumption is introduced to ensure a consistent order of latent noise components after applying ICA. When ICA is applied to each individual environment, the order of noise components may differ. Therefore, this assumption helps us eliminate permutation ambiguity. However, this assumption is not necessary. We discuss how to relax this assumption in Appendix C.

L231...

For any given $d$, a higher $p$ can be regarded as more data being provided since it offers additional auxiliary information. On the other hand, higher dimensional data makes optimization in ICA more challenging. Lower $p$ values provide less auxiliary information, but they are easier to optimize. Generally, we believe that higher $p$ will offer more benefits than disadvantages. For instance, in the third plot of Figure 2, we observe that higher $p$ values consistently yield higher F1 scores.

L293..

Yes. Sorry for the confusion. Figure 1 and Figure 3 are two different example, and line 293-294 is based on the example of Figure 3. We will make it more clear in the revision.

L297..

Please refer to the global response.

Line 303..

A good example of image distribution shifts is the WILDS dataset [1]. For instance, it provides a dataset where tissue slide images shift across patients' health conditions and hospitals. Identifying the mechanism shifts in this context involves using the tissue image dataset to determine, for example, whether some particular coloring changes across healthy and diseased patients. Since there is no evidence that the (mixing) mapping from latent to observation is linear, we did not include such an image example in our paper.

[1] Koh, P., et al. "Wilds: A benchmark of in-the-wild distribution shifts." ICML 2021.

We hope our answers will make you feel more positive about our work. Please let us know any follow-up questions!