DeepITE: Designing Variational Graph Autoencoders for Intervention Target Estimation

Q1 summarizing table

We agree that LIT does not require a given graph and will mention DeepITE requires a given graph explicitly in the introduction. In response to your suggestion, we have provided a comparative analysis of the assumptions inherent in various methods in Table R1 of the PDF attached to our global response. This table highlights that the primary advantages of DeepITE are its scalability and the flexible amortization during inference, allowing for adaptation without the need for retraining on new graphs and intervention targets. It also has limitations: it struggles to address confounders, and requires the graph structure as an input, despite its capacity to learn structural equations. These two limitations have been mentioned in Appendix H (Page 20).

Q2 soft interventions

Thank you for your valuable feedback! We have examined DeepITE's performance with various proportions of hard and soft interventions. Soft interventions are modeled by replacing the linear structural equations related to intervention targets with quadratic ones. The results, presented in Figure R1(c) of the attached PDF, show that DeepITE is robust to different mixtures of hard and soft interventions, demonstrating its capability to handle both types effectively. In addition, as detailed in Appendix G.3 on Page 18, we conduct further experiments based on classical ITE settings that also involve soft interventions. We will mention "soft interventions" explicitly in that section. The results, showcased in Table 5 in our paper, reinforce the superiority of DeepITE over other baseline models in these scenarios, further confirming its effectiveness in dealing with soft interventions.

Q3 consistency

In our paper, we primarily demonstrate that the decoder, as defined in Eq (5), can address interventional queries by modifying the adjacency matrix $A$ through graph surgery (see Propositions 1-3), since this mechanism lays the foundation for deriving DeepITE for ITE. Whether the estimated set of intervention targets is consistent with the ground truth relies on the properties of the amortized variational inference framework we use.

On the other hand, we notice that the consistency of VGAE for causal inference (the inverse process of ITE, see Lines 158-162) has been proven in [R4]. Due to the time limit, we are unable to prove the consistency of DeepITE at this moment, but will highlight this limitation in Appendix H (Page 21), identifying it as a promising area for future research:

Finally, proving the consistency and identifiability of DeepITE, and more broadly in the application of VGAEs for ITE, remains an interesting avenue for future work. Notably, such theoretical guarantees have been established for VGAEs in the context of causal inference (both observational and interventional) in [R4].

Q4 amortization

Thank for your constructive suggestion! We have investigated the performance of DeepITE as more graphs with different sizes are trained together. Specifically, we begin with 100 graphs, each containing 50 nodes and 1000 samples, and assess the model's performance on testing graphs containing 50 nodes. We subsequently introduce additional groups of graphs: 50 graphs with 100 nodes, followed by another 50 graphs also with 100 nodes, and finally 50 graphs with 500 nodes, followed by another 50 graphs also with 500 nodes.

As detailed in Figure R1(e), our findings indicate a gradual degradation in the performance of DeepITE (mix) as we incorporate more graphs of varying sizes, attributed to the amortization error. However, this reduction in performance is minimal.

Moreover, the results in Table 1 of our original paper show that while DeepITE (mix) performs slightly worse than DeepITE (sep), it even outperforms DeepITE (sep) training exclusively on 100-node graphs in terms of Recall@1. Based on this evidence, we maintain that the amortization process across graphs does not significantly hinder the performance of DeepITE.

Q5 performance boundary & components contribution

Thank you for your constructive feedback! We have conducted additional ablation studies to investigate the performance of DeepITE under various conditions, specifically focusing on graph size, the number of interventions, and sample size. The findings are illustrated in Figure R1 in the attached PDF. Our results indicate that the performance of DeepITE gradually declines as graph size and the number of interventions increase, and as sample size decreases, which aligns with our expectations.

To evaluate the contributions of different components within DeepITE, we performed a detailed ablation study, the results of which can be found in Appendix G.6 (Page 20). Our analysis revealed that the specific designs of both the encoder and decoder play critical roles in the model's empirical success.

Lastly, we have outlined the limitations of DeepITE in Appendix H (Page 20).

[R4] Zečević et al. Relating GNN to structural causal models, 2021.

Dear Reviewer HUbk,

We wanted to kindly check in on the status of the rebuttal, as there are 2 days remaining for the rebuttal period. Please let us know if there is anything else we can provide to contribute to our work. We would greatly appreciate it if you could provide your insights at your earliest convenience.

Thank you for your time and consideration.

Best regards,

Authors of DeepITE

Q1 theory

We acknowledge the importance of theoretical analysis. However, due to the time limit, we instead provide a comparative analysis of the assumptions inherent in various methods in Table R1 of the PDF attached to our global response. This table highlights that the primary advantages of DeepITE are its scalability and the flexible amortization during inference, without the need for retraining on new graphs and intervention targets (ITs). However, it struggles to address confounders, and it requires the graph structure as an input, despite its capacity to learn structural equations. These two limitations have been mentioned in Appendix H (Page 20).

Q2 scalability

Thank you for your insightful feedback! We have provided the performance of DeepITE as a function of the graph size and the number of interventions respectively in Figs R1(a-b) in the attached PDF. Our findings indicate that while performance in terms of Recall@1 declines as the graph size increases, Recall@5 remains stable (above 0.9), suggesting that the true ITs consistently appear among the top 5 candidates, even for graphs with 1000 nodes—a size that is already considered quite large for causal analysis. Moreover, as the number of interventions rises, DeepITE's performance decreases gradually; however, even with 5 interventions, the Recall@k remains at 0.825.

Q3 intervention types

Thanks for your constructive suggestion! We have examined DeepITE's performance with various proportions of hard and soft interventions. Soft interventions are modeled by replacing the linear structural equations related to intervention targets with quadratic ones. The results, presented in Figure R1(c) of the attached PDF, show that DeepITE is robust to different mixtures of hard and soft interventions, demonstrating its capability to handle both types effectively. In addition, as detailed in Appendix G.3 (Page 18), we conducted experiments based on classical ITE settings with soft interventions. We will mention "soft interventions" explicitly in that section. The results, shown in Table 5 in our paper, reinforce the superiority of DeepITE over other baseline models in dealing with soft interventions.

Q4 confounders

We acknowledge the importance of addressing confounders as a critical future work in Appendix H (Page 21). In line with other ITE works, where algorithms are initially developed for scenarios without confounders [3] before being extended to handle confounders [6], we also plan to extend DeepITE for confounders in future work.

Q5 identifiablity

In our paper, we primarily demonstrate that the decoder, as defined in Eq. (5), can address interventional queries by modifying the adjacency matrix $A$ through graph surgery (see Propositions 1-3), since this mechanism lays the foundation for deriving DeepITE for ITE. Whether the optimal set of ITs can be detected relies on the properties of the amortized variational inference framework we use.

On the other hand, we notice that the identifiability of VGAE for causal inference (the inverse process of ITE, see Lines 158-162) has been proven in [R4]. Due to the time limit, we are unable to prove the identifiability of DeepITE for now, but will highlight this limitation in Appendix H, identifying it as a promising area for future research:

Finally, proving the consistency and identifiability of DeepITE, and more broadly in the application of VGAEs for ITE, remains an interesting avenue for future work. Notably, such theoretical guarantees have been established for VGAEs in the context of causal inference (both observational and interventional) in [R4].

Q6 sample size

Thanks for pointing it out! We have depicted the performance of DeepITE as a function of the sample size in Figure R1(d) in the attached PDF. Here we choose 10 graphs for training and change the sample size from 25 to 1000 for each graph. Our findings indicate that DeepITE is generally robust to variations in sample size, though a larger sample size can enhance its performance. In particular, on graphs with 50 nodes, DeepITE achieves a Recall@1 of 0.812 and a Recall@5 of 0.984 even with just 50 samples for each graph (500 samples in total). This success may stem from the collaborative learning approach in DeepITE and the relatively few parameters in the GNN-based encoder and decoder. In contrast, traditional ITE methods [3,6,10] typically require thousands of samples for a single graph and intervention set to perform well. However, it is also important to note that the performance of DeepITE declines rapidly when the sample size is extremely small (e.g., 25 samples per graph), which aligns with our expectations.

Q7 random initialization

DeepITE is robust to random initialization. Since each node in a graph is initialized randomly and all graphs are trained collaboratively, the method can converge to optimal solutions regardless of the initial parameters. We will mention this point in our revised paper.

[R4] Zečević et al. Relating GNN to structural causal models, 2021.

Dear Reviewer bhQC,

We wanted to kindly check in on the status of the rebuttal, as there are 2 days remaining for the rebuttal period. Please let us know if there is anything else we can provide to contribute to our work. We would greatly appreciate it if you could provide your insights at your earliest convenience.

Thank you for your time and consideration.

Best regards,

Authors of DeepITE

Thank you very much for your positive evaluation and encouraging feedback on our paper. We deeply appreciate your constructive comments and valuable suggestions.

Q1 direct use of DAG-GNN

DeepITE does not directly use DAG-GNN. We have explcitly discussed the relation between DeepITE and DAG-GNN on Lines 260-270 on Pages 6-7. In summary, the key distinction lies in their objectives: DAG-GNN is designed for causal discovery (learning the DAG structure), whereas DeepITE focuses on ITE. Although both models share some similarities as variants of VGAEs, their inference models and latent spaces differ significantly. Furthermore, our ablation study in Appendix G.6 demonstrates the superiority of DeepITE over DAG-GNN for ITE tasks.

Q2 case study

Thanks for pointing it out! We have included a case study as follows:

We first present the causal graph in Figure R2. Recall that we focus on the RCA problem and we aim to identify and present the root causes of the system to users. Note that while we have labels for the root causes in our testing data, we only possess observations for the observable variables represented in the graph. Consequently, all methodologies employed can only localize observable variables as intervention targets (ITs), rather than directly identifying the root causes. For instance, RootCause 2 (a weak signal in marginal areas) can influence feature19, feature X, and feature Y. However, RootCause 3 also affects feature X. As a result, when a given method identifies either feature X as an IT, it becomes challenging to ascertain whether RootCause 2 is indeed the true root cause.

Interestingly, DeepITE model demonstrates superior performance on this dataset by effectively identifying features exclusive to a specific root cause. We evaluated the performance of all methods on samples 1758, 1760, and 1093, and summarized the results in Table R2. For sample 1758, where the true root cause is RootCause 2, DeepITE identifies feature19 as the IT, thus it is evident that RootCause2 should be the root cause. In contrast, other methods identify either feature X or feature Y, making it difficult to pinpoint the true root cause. Similarly, for sample 1760, DeepITE identifies feature60 as the IT, which is also exclusive to the true root cause, RootCause 3. On the other hand, in the case of sample 1093, DeepITE selects both feature60 and feature19, enabling us to conclude that both RootCause 2 and RootCause 3 are relevant root causes, a finding that aligns with the ground truth.

Q3 presentation

To address your concern, we will break up the third and fourth paragraphs into two shorter paragraphs each (e.g., the third paragraph can be divided at Line 46). Additionally, we will provide an introduction to less well-known priors, such as Jeffrey's prior, in the appendix, which will further enrich the reader's understanding of the priors.**

Q4 semi-supervised learning & labeled data

As detailed in Lines 283-309 in Section 5.3 (Page 7), the labeled data can be utilized to train the inference network, enabling it to more accurately identify intervention targets. In particular, the term $q(\gamma_i|\mathbf{x}, A)$ is replaced by the ground truth $\gamma_i$ when computing the ELBO (Eq. (14)), and an additional term is introduced to maximize the log-likelihood $\log q(\gamma_i^*|\mathbf{x}, A)$.

Q5 why use Recall

As defined in 326-328 on Page 8, Recall@k measures th proportion of true intervention targets (ITs) that are successfully captured within the top k ranked candidates proposed by each method. When k = 1, our goal is to pinpoint the intervention targets based on the highest-ranked candidate. We prioritize Recall@k because, in practice, false positives can be eliminated through further analysis, while false negatives are irrecoverable as they get lost among the numerous true negatives. This metric is widely adopted in the literature [4, 9, 29]. We will clarify this point in the revised paper.

Furthermore, we have incorporated additional metrics in our evaluations, such as Root.Acc and Score in the ICASSP experiments (see Table 7), and MMD/MSE in the ablation study (see Table 8).

Q6 why use GAT

We acknowledge that any inductive spatial GNN can be used as the inference network in DeepITE, and will mention this point in our paper. However, we choose GAT since it is more flexible compared to GCN and SGC [R3]. This flexibility stems from GAT's ability to dynamically weigh the importance of different nodes, thus allowing the variational distribution given by the inference network better approximate the exact posterior distribution. This advantage has been demonstrated in Appendix G.6 (Page 20), where we replace the GAT encoder with the encoder of DAG-GNN, a type of GCN [15], and show the benefits of using GAT.

Q7 typos

We will fix the typos accordingly.

[R3] Veličković, et al, Graph Attention Networks, ICLR 2018.

Dear Reviewer GccQ,

We wanted to kindly check in on the status of the rebuttal, as there are 2 days remaining for the rebuttal period. Please let us know if there is anything else we can provide to contribute to our work. We would greatly appreciate it if you could provide your insights at your earliest convenience.

Thank you for your time and consideration.

Best regards,

Authors of DeepITE

We greatly appreciate your valuable suggestions and the time you've taken to provide detailed feedback. We will carefully incorporate the case study results and clarifications into our paper.

For the different from DAG-GNN, the main theoretical results of this paper are from DAG-GNN. While this work applies the method to different problem settings, the technical novelty and contributions would be limited in this sense.

We would like to clarify that DAG-GNN does not prove any of the theorems presented in our paper, even within their problem settings. Instead, DAG-GNN provides two theorems specifically related to their proposed acyclicity constraints.

For Recall metric, the authors use different metrics for different datasets without enough justification. I understand what Recall means, but the rebuttal fails to properly justify why only Recall is used for Protein/Synthetic and the other metrics are used for ICASSP. This questions the robustness of the results.

Thank you for raising this concern. The additional metrics used for the ICASSP dataset were selected due to the unique characteristics and requirements of this particular dataset. As a result, these additional metrics could not be applied to to the Protein/Synthetic datasets in our study.

As explained in our rebuttal, we prioritize Recall@k because, in practice, false positives can be eliminated through further analysis, while false negatives are irrecoverable as they get lost among the numerous true negatives. This approach ensures that critical detections are not overlooked, which is why Recall@k is emphasized for certain datasets in our study.

On the other hand, it’s important to note that Recall@k inherently includes a ranking among all candidate intervention targets (ITs). A higher Recall@1 indicates that the true IT is ranked first, which implies that, with an appropriate threshold, the precision will also be very high. This is another reason why we focused on Recall@k in our analysis.

The argument that GAT is inductive is not rigorously correct, since common GNNs (e.g., GCN and SGC) are all applicable for inductive learning. The inductiveness is not a unique advantage of GAT.

Thank you for pointing this out. We agree that GCN and SGC are also inductive, as they, along with GAT, fall under the category of spatial GNNs rather than spectral GNNs, making them all applicable for inductive learning.

As mentioned in our rebuttal, we choose GAT not because only GAT is inductive, but because it is more flexible compared to GCN and SGC [R3]. This flexibility stems from GAT's ability to dynamically weigh the importance of different nodes, thus allowing the variational distribution given by the inference network better approximate the exact posterior distribution. This point is also validated in our experiments in Appendix A.6.

Once again, we sincerely thank you for your thoughtful feedback.

Q1 code availability

We are committed to open-sourcing the code upon acceptance of the paper.

Q2 lack of clarity

1. Datasets and Evaluations: We have provided descriptions of the datasets used, including their sources, preprocessing steps, and evaluation metrics in Appendix G (please refer to Appendix G.2-G.5)
2. Competing Methods: In Appendix B (pages 13–14), we have provided a review of the existing methods in XAI and RCA. This section includes a concise explanation of how each method operates. While we acknowledge that mathematical formulations can be rigorous, we believe that explaining these methods in plain language enhances clarity and comprehension. Additionally, we have referred readers to Appendix B within the experiment section (see Line 316 on Page 7) for easy access.

Q3 observation noise

DeepITE includes the observation noise variable $\epsilon$ that reflects uncertainty not present in the true SCM. In the true SCM, observed variables are deterministic functions of their exogenous variables and parent nodes via the structural equations (SEs). Since DeepITE does not have access to the true SEs or the distribution of the exogenous variables, it assumes the SEs can be approximated as $Dec(\mathbf{u}, \gamma, A) = f_2((I- A_\mathcal{I}^T)^{-1}f_1(\mathbf{u}))$ (Eq. (7)), with learnable $f_1$ and $f_2$. Thus, $\epsilon$ represents the uncertainty in the estimated observational distribution resulting from this approximation. Note that $\epsilon$ has also been used in VACA [21] for the same sake. We have clarified this point on Lines 222-224 in Section 5.1.

Q4 protein dataset

We clarify that the true intervention targets of the protein dataset are known because we utilize the dataset from IGSP [R1], which provides this information in Appendix E in their paper. Specifically, this datasets contain measurements of proteins and phospolipids under different interventional environments. In each environment, signaling nodes are inhibited or activated. Hence, these sites form intervention targets. This dataset has been previously employed in [3,6,10] for ITE. We will explicitly mention this in the dataset introduction in Appendix G.4.

Q5 Lasso

We respectfully believe that Lasso does not align with the scope of ITE as outlined Definition 1 on Page 4. First, Lasso cannot incorporate the causal graph structure available in ITE. The prediction G can be an arbitrary variable in the graph and there is no clear guideline on how to choose G given the observed data. Moreover, Lasso is designed to identify correlations rather than causal relationships, and so it is typically used for undirected graphical models [R2]. Consequently, Lasso may select all variables correlated with G, rather than correctly identifying the intervention targets that directly impact G.

Actually, Lasso has been used for RCA in [8], but only for bipartite directed graphs where causal and effect relationships between features and predictions have been provided. In contrast, our approach is designed to handle more complex causal graph structures where the predictions are unknown.

Q6 given causal graph

You are correct that DeepITE assumes a given causal graph (see Defintion 1 on Page 4). However, once trained, it can evaluate new samples on new causal graphs without needing to retrain. By directly inputting the sample and the new graph structure into the inference network, we can derive the intervention targets. This stands in contrast to classical ITE methods, which require retraining for each new causal graph and each new set of intervention targets, even if the graph remains unchanged.

Q7 limitations

We have included a discussion of the limitations in Appendix H (Page 21).

[R1] Wang et al, Permutation-based Causal Inference Algorithms with Interventions, NIPS 2017.

[R2] Meinshausen et al, High-dimensional Graphs and Variable Selection with the Lasso, Ann. Statist., 2006.

Dear Reviewer q8ww,

We wanted to kindly check in on the status of the rebuttal, as there are 2 days remaining for the rebuttal period. Please let us know if there is anything else we can provide to contribute to our work. We would greatly appreciate it if you could provide your insights at your earliest convenience.

Thank you for your time and consideration.

Best regards,

Authors of DeepITE

Thank you very much for your valuable feedback.

Regarding Q5:

• I think we may have a misunderstanding. If G is an auxiliary discrete variable (with discrete values denoting the interventional environment a data point came from), which has the same value for each point in the same interventional environment, a method like Lasso would indeed find all variables that are associated with G (it will yield the variables in the Markov Blanket of G, we can term them MB(G)). One can also assume that G is always a cause (by definition it cannot be an effect of any variable). Then, MB(G) will be either the direct effects of G, or parents of the direct effects. Then, one can think about using the known causal graph to distinguish these sets of variables.

Thank you very much for your clarification. We now understand that this is quite similar to the method implemented in [3], and we have compared DeepITE with it in our analysis.

Regarding Q6:

• Can the authors briefly elaborate why it is possible to generalize to new, previously-unseen, causal graphs when using their framework?

Sure. In simple words, after training the VGAE collaboratively using graphs with different sets of intervention targets, different structures, and even different sizes, the inference model (i.e., the encoder) in the VGAE successfully learns a mapping whose input is the observed data $\mathbf{x}$ and the adjacency matrix $A$ and output is the distribution of intervention indicator $\gamma$. As a result, when we replace the adjacency matrix $A$ by that of an unseen graph, the encoder can still output the distribution of the intervention indicator.

Thank you again for your thoughtful response, as they have provided valuable insights for improving our paper.