RegExplainer: Generating Explanations for Graph Neural Networks in Regression Tasks

Dear reviewer 4dqr, thank you for taking the time to review our work and providing feedback. In the following, we aim to address your questions and concerns.

A-1a: Thank you for your suggestion. We will include detailed descriptions of the datasets in future versions of the paper. In the appendix, we describe how these datasets were created and the rationale behind their creation.

A-2a: Thank you for pointing this out. Subgraph connectivity is indeed an important issue. However, in this work, our main goal was to extend GIB-based explainers to graph regression tasks, following the settings of previous works in other aspects. We are keen to explore and address the connectivity of explanatory subgraphs in future research.

A-2b: We incorporate the InfoNCE loss into the GIB objective with reference to previous work[1], where InfoNCE is a lower bound of mutual information (MI), and there is no constraint on the distribution of variables. Thank you for pointing out the issue in the appendix. We will provide a more detailed explanation in the next version.

A-2c: We are happy to address your concern. In Step 1 of Section 4.3, we describe in detail how to select $G^+$ and $G^-$. “We can define two randomly sampled graphs as positive neighbor $G^+$ and negative neighbor $G^−$, where $G^+$’s label $Y^+$ is closer to $Y$ than $G^−$’s label $Y^−$, i.e., $|Y^+ − Y| < |Y^− − Y|$.”

A-2d: They are continuous. The final explanatory subgraph is obtained by selecting the top-k edges based on their weights.

A-2e: Thank you for pointing this out. We will correct the typo in Table 3 in the next version of the paper. We use RMSE as the metric for measuring distances between graphs or graph distributions. Besides, metrics like KL-divergence can be challenging to compute in practice. Therefore, we use graph embeddings and prediction labels to measure distribution shifts between original graphs and explanation subgraphs.

A-2f: The purpose of Table 2 is to demonstrate that the explanatory subgraphs can indeed be out-of-distribution (OOD). The results in the last column indicate that these subgraphs are not sufficiently faithful and can introduce bias in the explaining. To address this issue, we introduced the mixup method to mitigate the OOD problem of the explanatory subgraphs.

A-2g: "Can explain" means that the explanation should identify the reasons behind the GNN's prediction. For example, in the defined datasets, different motifs and their corresponding features lead to different labels. The GNN relies on these motifs to make predictions, and the goal of the explainer is to identify these motifs as part of the explanation.

A-3a: Yes, our method can be extended to node-level regression tasks. In the case of a three-layer GCN, node-level tasks essentially involve graph tasks centered on 3-hop subgraphs around each node. Therefore, our method can be easily adapted for node-level tasks. We look forward to incorporating more datasets in future work.

A-3b: Thank you for pointing this out. "N" represents the number of graphs in the dataset. We will include this clarification in the next version of the paper.

A-3c: The term "regression label" refers to the prediction made by the GNN for a given graph or subgraph.

[1] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation Learning with Contrastive Predictive Coding.

Dear reviewer X7ek, thank you for taking the time to review our work and providing feedback. In the following, we aim to address your questions and concerns.

How does this method perform with/against classification explanation models? As the explainer can be modularly applied to existing trained GNNs, this would be interesting to see if it can improve the accuracy of explanation generations. This work [1] is a good work on instance and model-level explanations for graph classification tasks.

A1: Yes, we have compared our method with several existing methods for graph classification, including GRAD, ATT, MixupExplainer, GNNExplainer, and PGExplainer, as shown in Table X. It is important to note that these methods were originally designed for graph classification tasks and use cross-entropy loss in their objective functions, which cannot be directly applied to graph regression tasks. Therefore, we adapted them for comparison by using MSE loss.

Work [1] is an excellent study that explores graph neural network interpretability from a novel perspective. Our method could potentially be adapted to the framework presented in their Equation 6 and might improve the performance of their approach. We look forward to incorporating and citing this method as a baseline in our future work.

Is graph mix-up performed on the graphs explicitly or within latent space? If the mix-up occurs in graph space, then how is the subgraph G* combined with $(G^+)^{\Delta}$ and $(G^-)^{\Delta}$? How are edges added between these different subgraphs?

A2: We describe the mix-up process in detail in Appendix C. The mix-up is performed on the graph explicitly by mixing edge weights, and different subgraphs are connected through randomly sampled connection edges.

Furthermore, for more complex regression datasets, naively adding edges can drastically change the label for each graph. Is this strategy then limited to graph regression datasets which inherently rely on graph structures to derive labels?

A3: Yes, in this work, we follow the settings of previous studies and focus primarily on the graph structure without incorporating edge features. We are interested in exploring the impact of edge features on our method's performance in future research.

If the mix-up occurs within latent space, then how does this work differentiate itself from graph rationalization works [2, 3]? If the mix-up occurs in latent space, then additional studies to compare against graph rationalization baselines should be included.

A4: Our mix-up primarily occurs on edge weights. We will consider more related works[2, 3] and incorporate their strengths and citations in future studies.

Limitations: The work requires an accurate trained prediction model. The model explanations are not used to retrain the GNN in any way.

A5: Thank you for your suggestion. This work primarily focuses on post-hoc explanations. In future work, we will explore methods that involve retraining the GNN to enhance both the GNN and the explainer's performance.

Dear reviewer B4J7, thank you for taking the time to review our work and providing feedback. We appreciate your thorough review and aim to address your questions and concerns in detail. Due to character limitations, we will provide a detailed response to the remaining points in the official comment section. Please let us know if you have any further questions or need additional clarification, we are happy to discuss with you.

The paper is densely written ... problem.

A1: Thank you very much for your suggestion. In this work, our goal is to provide reliable explanations for graph regression tasks. Previous methods [1, 2, 3] used the vanilla GIB for graph classification, which cannot be trivially applied to graph regression tasks. Therefore, we introduced InfoNCE loss into GIB and theoretically validated its effectiveness. Next, we explain that the OOD problem is more severe in graph regression tasks. On this basis, we introduce the mixup method and combine it with InfoNCE loss to propose our method and model, which includes a contrastive learning objective function with contrastive loss. We aim to balance the effectiveness of the model design with the reliability of the theoretical foundation. In future versions, we will improve the organization of our paper, provide more detailed explanations of the model design, and move some of the theoretical derivations to the appendix.

Where are the ground truth vectors for your evaluation? How are they obtained? How can ground truth for explanations be in the datasets? This is the most important part of your paper, and it is left to the imagination of readers.

A2: (1). What the explanation subgraphs look like: As introduced in Section 3 Preliminary, line 119 - line 120, we follow the setup from previous work[1, 2], using a binary edge mask to represent the explanation subgraph of the original graph. Specifically, for each edge in the original graph, our explainer produces a prediction value. If this value is 1, it indicates that the edge is part of the explanation; if it is 0, it means the edge is not relevant to the explanation. For each graph, the mask containing these edge weight values is a vector.

(2)&(3). How the ground truth for explanation subgraphs is obtained: Establishing the ground truth for explanation subgraphs is a critical step. We adopt the approach from previous work, including both synthetic and real datasets. In synthetic datasets like BA-motif-volume, we designed the dataset such that the label is related to the motifs and their corresponding features within the graph. Therefore, the corresponding motif subgraph is the explanation subgraph and is used to evaluate the explainer's performance. Additionally, we designed a graph regression task with ground truth explanation subgraphs based on the chemical dataset Crippen. More detailed information can be found in Appendix E.1.

(4). How we evaluate the explainer’s performance and what metrics are used: As mentioned earlier, the explainer produces a weight for each edge in the graph and determines whether the edge belongs to the explanation subgraph based on this weight. In practice, the edge weights are floating-point numbers, and we assess the accuracy of the explanation by calculating the AUC-ROC with respect to the ground truth, thus evaluating the explainer’s performance.

(5). Additional Information: In some other works, evaluation methods without ground truth are used, such as fidelity/sparsity scores to estimate the quality of the explanation subgraph. However, these evaluation methods face issues with out-of-distribution (OOD) problems. Therefore, we did not use datasets and metrics without ground truth in this work. We plan to introduce more datasets, both with and without ground truth, in future work to better evaluate our method and facilitate related research.

I argue ... concern?

A3: From my understanding, your concern means that you believe the improvement in explainer performance comes from sampling and mixup, and thus the contrastive learning and contrastive loss components are unnecessary. Let me clarify this point; if my understanding is incorrect, I would be happy to further discuss and clarify the issue with you.

First, as mentioned in A1, our method aims to explain graph regression tasks, whereas previous methods were based on vanilla GIB and focused on explaining graph classification tasks. By introducing InfoNCE loss, we can better explain graph regression tasks. The sampling and mixup are employed to address the OOD (out-of-distribution) problem during the explanation process. The combination of these components is crucial for effectively improving the model's performance.

As shown in Figure 5 of the ablation study, simply using the mixup method and sampling (dropping the InfoNCE module) leads to a decrease in model performance. Therefore, just sampling is not sufficient to fully address the problem.

[1] Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks. Advances in neural information processing systems, 32, 2019.

[2] Dongsheng Luo, Wei Cheng, Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, and Xiang Zhang. Parameterized explainer for graph neural network. Advances in neural information processing systems, 33:19620–19631, 2020.

[3] Hao Yuan, Haiyang Yu, Jie Wang, Kang Li, and Shuiwang Ji. On explainability of graph neural networks via subgraph explorations. In International Conference on Machine Learning, pages 12241–12252. PMLR, 2021.

Why are the ablation results different for BA-Motiv-Volume?

A4: This is a very good observation, and I am glad you pointed it out so we can explain it. We will include the relevant explanation in future versions to improve the quality of the paper.

First, we observed that in the BA-motif-volume dataset and the Crippen dataset, the mixup module has a greater impact on method performance. In contrast, in the other two datasets, the InfoNCE loss (contrastive learning) has a more significant influence. We believe this is due to the characteristics of the datasets: in BA-motif-volume, all graphs are of the same size. Therefore, for a well-trained GNN, the explanation subgraphs (which are only parts of the original graphs) are significantly out-of-distribution (OOD), which reduces model and explainer performance. By using the mixup method to address this issue, we can effectively improve performance. In the other two datasets, the graph sizes are dynamic, and the trained GNNs are relatively more robust, so the performance loss caused by not using mixup is not as substantial.

What is the real ... mean?

A5: This means that our method cannot be trivially applied to tasks involving dynamic graph structures, such as spatio-temporal graphs (STGs). In STGs, the graph structure changes over time, which poses challenges for selecting samples for contrastive learning and mixup. The dynamic nature of the topology and node features makes it difficult to apply our approach directly. We aim to extend our method to handle dynamic graph structures in future work.

Limitations

A6: I understand your concern. However, the add-on approach actually offers advantages. Unlike single and fixed explainer methods, our framework can be flexibly applied to existing explainer models, making it suitable for graph regression tasks and improving their performance. In our code, we have also included implementations of Regexplainer based on GNNExplainer and PGExplainer. These implementations can directly replace the original GNNExplainer and PGExplainer and be used straightforwardly.

The authors have also not stated ... etc.

A7: Our method is specifically designed to improve explaining the graph regression tasks. For graph classification tasks, we can adapt our approach by replacing the MSE loss in the objective function with cross-entropy loss. Additionally, our method can be extended to node-level tasks. For example, in a 3-layer GNN, a node classification task can be viewed as a graph task on 3-hop subgraphs centered around nodes, and then a variant of our method with cross-entropy loss could be easily adapted to the dataset.

Dear reviewer LTFm, thank you for taking the time to review our work and providing feedback. In the following, we aim to address your questions and concerns.

Limited Discussion on Computational Efficiency

A1: Thank you for pointing this out. We are glad to supplement our analysis of computational complexity, which will also be included in future versions. In the implementation, we transform the structure of the graph data from the sparse adjacency matrix representation into the dense edges list representation. We analyze the computational complexity of our mix-up approach here.

Given a graph $G_a$ and a randomly sampled graph $G_b$, assuming $G_a$ contains $M_a$ edges and $G_b$ contains $M_b$ edges, the complexity of graph extension operation on edge indices and masks, which extend the size of them from $M_a$, $M_b$ to $M_a+M_b$, is $O(2(M_a+M_b))$, where $M_a>0$ and $M_b>0$. To generate $\eta$ cross-graph edges, the computational complexity is $O(\eta)$. For the mix-up operation, the complexity is $O(2(M_a+M_b)+\eta)$. Since $\eta$ is usually a small constant, the time complexity of our mix-up approach is $O(2*M_a+2*M_b)$.

We use $M$ to denote the largest number of edges for the graph in the dataset and the time complexity of mix-up could be simplified to $O(M)$.

In this work, GIB is used as Explainer, have you also studied other instance-based approach like GNNExplainer etc.

A2: GIB is a widely used theoretical foundation in related work. Both GNNExplainer and PGExplainer are based on GIB. In our experiments, we considered both GNNExplainer and PGExplainer and included a comparative study applying our method to GNNExplainer. The results are presented in Table 1 in the paper, as rows "GNNExplainer", "PGExplainer" and "+RegExplainer".

How does the proposed method handle dynamic graph topology changes, and what are the implications for real-world applications with evolving graph structures? (Directed Graph, Hyper Graph)

A3: Thank you for pointing this out. We also discuss the issue of dynamic graphs in the limitations section. This is a very valuable problem, and we plan to further investigate the explainability of dynamic graph topology, evolving graph structures, and spatio-temporal graphs. This is our direction for future work.

Any other general-used datasets are tested? Especially those real-world graph datasets.

A4: We would very much like to include more real-world datasets. However, the fact is that graph regression datasets containing ground truth explanation subgraphs are very rare. In this work, we created Crippen[1] as a real-world dataset. We will strive to discover, generate, and use more real-world datasets in future work.

[1] John S Delaney. Esol: estimating aqueous solubility directly from molecular structure. Journal of chemical information and computer sciences, 44(3):1000–1005, 2004.

Dear reviewer KBtu, thank you for taking the time to review our work and providing feedback. In the following, we aim to address your questions and concerns.

Some of the model designs are not moviated consistently over a pronounced challenge. The paper seems to be a combination of Mixupexplainer, GNNExplainer and G-Mixup.

A1: In this work, our goal is to provide reliable explanations for graph regression tasks. Previous methods [1, 2, 3], particularly the vanilla GIB used for graph classification, cannot be trivially applied to graph regression tasks. Therefore, we introduced InfoNCE loss into GIB and theoretically validated its effectiveness. We then recognized that the OOD (out-of-distribution) problem is more severe in graph regression tasks, so we incorporated the mixup method and combined it with InfoNCE loss to develop a contrastive learning objective function that includes contrastive loss. Our model design is cohesive and aims to address a specific problem, rather than simply combining Mixupexplainer, GNNExplainer, and G-Mixup.

The challgenges of graph regression explainer and distribution shift seem to be independent. The author doesn't make any justification on the graph topology and only embedding/predicted values are reported in Figure 3.

A2: The explainability of graph regression and distribution shift are not independent issues. We found that the distribution shift problem is more severe in graph regression tasks. To generate better explanation subgraphs, we need to correct the subgraph distribution during the explanation process to avoid biased interpretations. Figure 3 illustrates why addressing the distribution shift problem is necessary. It shows that the GNN makes completely incorrect predictions for the BA-Motif-Volume samples, which significantly affects our estimation of $I(Y^*, Y)$ and leads the objective function in the wrong direction. Therefore, we need to address the explainability of graph regression and the distribution shift of subgraphs together.

In Figure 6, why the selection of \alpha seems do not affect the overall performance? It seems the InfoNCE loss is negeligile in the optimization of the proposed method

A3: This is because our method exhibits strong robustness to the selection of hyperparameters. However, this does not imply that the InfoNCE loss is negligible. As shown in the ablation study in Figure 5, the performance of the method significantly decreases when the InfoNCE loss module is removed. Therefore, Figures 5 and 6 together demonstrate the robustness and effectiveness of the InfoNCE loss.

[1] Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks. Advances in neural information processing systems, 32, 2019.

[2] Dongsheng Luo, Wei Cheng, Dongkuan Xu, Wenchao Yu, Bo Zong, Haifeng Chen, and Xiang Zhang. Parameterized explainer for graph neural network. Advances in neural information processing systems, 33:19620–19631, 2020.

[3] Hao Yuan, Haiyang Yu, Jie Wang, Kang Li, and Shuiwang Ji. On explainability of graph neural networks via subgraph explorations. In International Conference on Machine Learning, pages 12241–12252. PMLR, 2021.

Dear reviewer KBtu,

Thank you for your valuable feedback on our paper. We truly appreciate the time you have taken to review our work and provide detailed insights. Your comments help us a lot in refining our next version of the paper.

We understand that you may have other commitments, but if you have any additional questions or require further clarification on any of our responses, we would be grateful for the opportunity to address them. Your feedback is crucial to us, and we are eager to ensure that our revisions align with your expectations.

Please feel free to reach out at your convenience, as we would be more than happy to discuss any remaining concerns or provide further information.

Thank you for your continued support.