Thank you for your time and effort reviewing our paper. We highly appreciate your inputs and would like to address your concerns as follows.

Experimental results on explainability are on very easy tasks with identical explanations

This is a reasonable concern. However, we respectfully disagree with the above statements by the reviewer and would like to clarify these 3 datasets on 3 points:

• The datasets are simple for classification, but not for explanation. Table 3 shows that most predictors can obtain near perfect classification on these 3 datasets, however, if you look at Table 1, the performances on explainability varies significantly. Such observation shows that while all methods excel at prediction, not all of them pick up the right signals. In this case, the ease of getting to the right prediction is a favorable attribute because it sets a fair ground to compare various methods on explainability and robustness to noisy signals, based on the ground-truth explanations. Notice that the point of ante-hoc explanation is learning both the good explanations and good predictions at the same time.

• The ground-truth explanations are not simple. In fact they are quite diverse. We encourage the reviewer to look at Figures 7, 8, and 9 in the appendix for some examples. For instance, Lactam groups can have varying ring sizes and configuration within a molecule (Figure 7). Various combinations of both Lactam groups and Benzoyl groups lead to diverse ground truth explanations in molecules (Figures 8 and 9). This makes our datasets more diverse in terms of explanations compared to existing datasets with house or cycle motifs. Moreover, our graphs are real-world molecules mined from ChemBl, not random graphs.

• We have various prediction settings, not just binary classification on whether the motifs exist. In the dataset BenLac, we assign labels according to various co-occurrence conditions of either the lactam and benzoyl group. Recognizing these logics is beyond a simple pattern matching problem. In the dataset BenLacM, we consider the explanability in a multiclass setting in which each class has a separate pattern. This setting is often overlooked in the literature. Our dataset Lactam is a binary classication task, however, in figure 7, we showed the weight assigned on both the positive and the negative examples, which is often overlooked in the literature.

Certainly, our synthetic datasets cannot capture all the complex interactions that may happen in Chemistry. However, as we show in our Table 2, many existing graph explainers fail to capture even these patterns. We believe these datasets are more complex than many existing benchmark graph datasets for explainability.

Dear reviewer,

We have resolved your concern regarding the paper's technical novelty and contribution. We also have responded to your last concern regarding the datasets and would like to know if it has been addressed. As the extended deadline of the rebuttal phrase is approaching, we look forward to your confirmation.

Best regards,

Authors

Thank you for your time reviewing our paper. We would like to address your concerns as follows.

Lack of Novelty. This paper primarily consists of a combination of methods from other studies. The model’s unique methodology is not clearly emphasized. For example, in the Information Bottleneck principle, the iterative algorithm from Tishby (2000) [1] is used as-is.

We respectfully disagree with the assertion that our method is merely a combination of approaches from existing studies. While our work draws inspiration from established principles, the development of our framework is far from trivial.

Firstly, the iterative procedure proposed by Tishby (2000) [1] is not applied as is, as doing so in the graph space would be intractable. This is a major challenge as even though the IB principle has been applied to explainability on graphs in many existing works, we are the first to approach the problem via this iterative principle. This constitutes our first key contribution.

Secondly, our contribution lies in aligning this iterative process within a bilevel optimization framework. Specifically, we replace the estimation of $P(S|G)$ and $P(Y|S)$ with neural approximation, achieved through a predictor trained in the inner loop using cross-entropy loss and an explainer trained in the outer loop via hypergradients from the inner loop. In our case, cross-entropy indirectly influences the optimization of mutual information, which is a distinction of our method compared with existing approaches that approximate mutual information via variational bounds. This design enables the predictor to generalize to unseen data, effectively transforming the entire framework into an ante-hoc predictive model. These contributions require deliberate and non-trivial insights.

Moreover, in the explainer and predictor sections, except for simple tricks like permutation invariance, the method of PGExplainer [2] is used directly.

Firstly, PGExplainer [2] is a post-hoc model, whereas our approach is an ante-hoc model. This distinction fundamentally affects how the models are trained and utilized.

Secondly, the structural similarity between our explainer and predictor modules and those of PGExplainer arises from the intuitive and practical design of using an edge-weighting module followed by a prediction module for weighted graphs. The novelty of our method lies elsewhere. Specifically, our approach introduces a bilevel optimization framework inspired by iterative IB principles. This innovative training formulation represents a significant departure from PGExplainer's methodology.

We opted for straightforward design choices for the submodules to focus on showcasing the advantages of our training framework. Our framework is flexible: if needed, it can accommodate more general explainer modules and sophisticated predictors without compromising its core functionality.

[1] Tishby, Naftali, Fernando C. Pereira, and William Bialek. "The information bottleneck method." arXiv preprint physics/0004057 (2000).

[2] Luo, Dongsheng, et al. "Parameterized explainer for graph neural network." Advances in neural information processing systems 33 (2020): 19620-19631.

Lack of Distinction from Existing Ante-Hoc Models. The paper does not present advantages that differentiate it from existing ante-hoc models. For example, it does not explain how the bilevel training approach provides any benefits over GSAT, which uses variational bounds. Furthermore, it lacks an explanation of advantages compared to other GNN models that generate predictions and explanations simultaneously, such as CAL and OrphicX.

We could have been clearer on presenting the advantages of EAGER over existing ante-hoc models. Information Bottleneck (IB) principle has been a great basis for explainability. In the graph domain, methods that rely on IB principle often approximate the mutual information via variational bounds, such as GSAT. In order to minimize $I(S;G)$, GSAT uses variational bound $I(S;G) \leq E_{G} [KL(P(S|G) || Q(S))]$, and minimizes the RHS, thus effectively minimizing an upper bound. This can lead to a loose approximation due to the complexity of the graph space. In particular, defining an appropriate variational distribution is difficult in the graph space and one has to make simplifying assumptions regarding features and edge independence. This overhead design is another significant burden. For EAGER, we decouple learning the explainer and the classifier by formulating the learning as a bilevel optimization problem. More specifically, EAGER is inpired by the IB iterative process (cite the paper) that optimizes the IB objective, guaranteeing convergence to local optima. In EAGER, cross-entropy loss indirectly influences the optimization of mutual information.

In terms of the input to the classifiers, EAGER is deterministic as the weighted graph produced by the explainer is used for prediction, taking advantage of modern GNN's ability to process edge features. The weighted graphs produced by EAGER explainer often has highly contrastive distintion between the foreground (high weighted edges) and the background, making the explanation highly interpretable (see Figures 1, 4, and 7-9). GSAT, instead, is stochastic as the method relies on generating random subgraphs based on the distribution represented by the weighted graph. This sampling process favors more uniform distribution to ensure well-behaved gradients, which explains the small difference between foreground and background edges in GSAT's explanations (Figure 4). This means that GSAT explanations are often not sparse and, given that ground-truth explanations are often not available in real-world application, GSAT explanations hard to interpret

Thank you for introducing causality-based methods like CAL and OrphicX . These works are highly related and we have included more discussions about them in our main text (Line 45-56, 135-139). Compared to EAGER or IB-based methods in general, CAL has more assumptions about the existence of causal and shortcut features. Additionally, modeling these features add to the complexity of designing and training the model. OrphicX is a post-hoc model because the target GNN is pretrained and fixed.

Need for Fidelity Score in Explanation Evaluation...

Fidelity is not relevant for EAGER or ante-hoc explanations in general because we learn both the explainer and the classifier at the same time, not adapting an explainer to an already trained classifier in post-hoc explanation. In ante-hoc explanation, both in graph and other domains, the explaner is learned together with the classifier and is part of the system. As such, producing a prediction requires both the explainer and the classifier. Fidelity requires comparing the differences between predictions using the input graphs and predictions using the explanations. However, in our case, the predictor never predicts based on the input graphs so this metric is not applicable.

Instead, we further evaluated the quality of the explanation using a related metric called Reproducibility, which compares how quickly predictive performances drop as we progressively sparsify the explanations. This is reported in Appendix B. We have updated the main text to better present this point.

Limited baselines...CAL and OrphicX are models that predict labels based on important explanatory subgraphs. It would be beneficial to include these as additional baselines for both explanation and classification performance.

As suggested, we have added these baselines. The reviewer can find results for CAL in Table 1 and Table 3, and the results for OrphicX in Table 1. Since OrphicX is a post-hoc model, we did not include it in Table 3.

...Could you show the training curve for loss and accuracy?

We have updated the appendix with this analysis. Please refer to the newly added appendix C. The plots show some variations in terms of training loss and validation AUC. Overall, we believe such level of variations is reasonable small. The general trends, reduction in training loss and improvement in validation AUC, are still clearly observed.

Thank you for reviewing our paper. We are happy to answer your questions.

The bilevel optimization, though effective, is computationally intensive and requires significant training time compared to other models.

We included the runtime analysis of EAGER and other baselines in Appendix D-Table 7. EAGER's training time is considerably long, however, we are still significantly faster than several other baselines. EAGER is just as fast as other methods during inference. That said, we believe that EAGER can be sped-up using recent advancements in faster bilevel optimization solvers, such as methods that apply decentralized processing, or single-loop algorithms. We also added a new section 3.5 discussing the runtime complexity of EAGER.

EAGER’s application is restricted to curated datasets; more real-world, large-scale evaluations could better demonstrate its adaptability.

During this project, we realized that many existing benchmark datasets for explanation (i.e, Bernoulli graphs with attached house of cycle motifs) are not reflective of real-world settings. Motivated by this problem, we curated datasets that are close to real-world settings.

Specifically, our datasets are real molecules mined from the ChemBl database. The ground-truth explanations are not simple and are quite diverse. We encourage the reviewer to look at Figures 7, 8, and 9 in the appendix for some examples. For instance, Lactam groups can have varying ring sizes and configuration within a molecule (Figure 7). Various combinations of both Lactam groups and Benzoyl groups lead to diverse ground truth explanations in molecules (Figures 8 and 9). This makes our datasets more diverse in terms of explanations compared to existing datasets with house or cycle motifs.

In addition, we have various prediction settings, not just binary classification on whether the motif exist. In the dataset BenLac, we assign labels according to various co-occurrence conditions of either lactam or benzoyl groups. In the dataset BenLacM, we consider the explainability in a multiclass setting in which each class has a separate pattern. This setting is often overlooked by the literature. Our dataset Lactam is a binary classification task, however, in Figure 7, we showed the weight assigned on both the positive and the negative examples, which is also overlooked in the literature.

Model performance is sensitive to hyperparameter settings,...

We generally agree with this notion. Tuning requirement and sensitivity to hyperparameter are reasonably expected in system that is complex (bilevel optimization) with multiple components (explainer and predictor). However, we believe this overhead cost can be amortized over a domain. For example, we use the same setting across multiple molecular datasets and the method obtain generally good results.

Just for suggestion, it would be better to have more real-world datasets. For those lacking a ground truth explanation, the fidelity score could be considered.

We thank the reviewer for the thoughtful suggestion. Fidelity is not relevant for EAGER or ante-hoc explanations in general because we learn both the explainer and the classifier at the same time, not adapting an explainer to an already trained classifier in post-hoc explanation. In ante-hoc explanation, both in graph and other domains, the explainer is learned together with the classifier and is part of the system. As such, producing a prediction requires both the explainer and the classifier. Fidelity requires comparing the differences between predictions using the input graphs and predictions using the explanations. However, in our case, the predictor never predicts based on the input graphs so this metric is not applicable. Instead, we further evaluated the quality of the explanation using a related metric called Reproducibility, which compare how quickly predictive performances drop as we progressively sparsify the explanations. This is reported in Appendix B.

Could you elaborate on the rationale for including the average AUC in Table 3?...

Being an ante-hoc model, EAGER should not only excel in explanation, but also perform well as a predictive model. Averaging AUC across diverse datasets provides a high-level overview of the model's general predictive performance. While it does not capture dataset-specific nuances, it offers a clear benchmark for comparing models holistically

Are there plans to include newer baselines in future evaluations? For instance, the addition of MixupExplainer...

We thank you for referring us to MixupExplainer. So far, in our revision, we have added CAL (2022) and OrphicX (2022). In general, there will always be more baselines that one can add. However, due to the time constraint of the rebuttal period and resource constraints on our side, we are still working on getting more results in. We, still, have added citations and discussion of MixupExplainer in the main text (Line 133-135).

Thank you for your time and insights in reviewing our paper. We are glad to address your concerns as follows.

My primary concern is about the efficiency of the proposed method,... The authors should thoroughly discuss the computational complexity of their method in the main section of the paper and include experiments on running time...

We thank the reviewer for the thoughtful suggestion. As requested, we have added Section 3.5 that discusses the time complexity of EAGER. Let $C_{GNN}$ be the running cost of the underlying GNN and $C_{hyper}(T,d)$ be the cost of calculating hypergradients, with $T$ being the number of inner iterations and $d$ being the number of dimensions. EAGER's inference time is $O(2C_{GNN})$: an explainer GNN followed by a predictive GNN. During training, EAGER's time complexity is $O((T+1)C_{GNN}+C_{hyper}(T,d))$: one outer iteration followed by T inner iterations plus hypergradient calculation. Notice that when used with a typical GNN, generally $C_{hyper}(T,d)$ scales linearly with batch size, $T$, and $d$ [1]. We report detailed training and inference time in Appendix D.

[1] Amirreza Shaban, Ching-An Cheng, Nathan Hatch, and Byron Boots. Truncated back-propagation for bilevel optimization. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1723–1732. PMLR, 2019.

More comprehensive testing on larger and more diverse datasets is necessary to establish a clearer understanding of the method's [running time] in real-world scenarios.

We generally agree with this suggestion. We would like to point out that since EAGER scales linearly with the number of input graphs, even if we repeat the experiments on larger datasets, we would obtain the same relative comparisons reported in Appendix D. Due to time and resource constraint, we could not repeat the runtime analysis (Appendix D) on larger datasets like HIV or PCBA (due to certain post-hoc baselines taking very long to run). We are doing our best to get more results in. However, we did add HIV to the classification benchmark (see below).

... it is common practice to maintain a consistent target model across different methods to ensure fair comparisons with baseline approaches. However, due to the unique architecture of the proposed method, it does not use the same GNN classifier as the one employed in the baseline methods.

We respectfully disagree and would like to provide more clarification. EAGER is a general framework that can work with any GNN backbone. In our experiments, we tried to the best of our ability to maintain the same GNN architecture across all baselines (see line 419-422). In particular, we rewrote parts of other baselines' source code to produce GNN architectures as similar as possible to the ones we implemented. Moreover, we altered the baselines' source code such that they take in the same atom and bond features as we used. For all the benchmarks and EAGER, if a method requires an underlying GNN backbone, we consistently used GIN. For post-hoc explainers, we also used GIN as the pretrained predictive model.

The datasets currently used in the study are relatively small. To more effectively demonstrate the capabilities of the proposed method in classification tasks, it would be beneficial to employ larger datasets, such as HIV or PCBA. Utilizing these more extensive datasets could provide a more robust evaluation of the method's performance.

We have added HIV as one of the benchmark in Table 3. Additionally, we would like to offer more explanations. The majority molecular datasets are small, especially those curated in real-lab settings. Performances on small datasets with limited data, at the moment, would best reflect expected real-world performances in this domain. That said, we do agree that evaluation on larger datasets is still beneficial in showcasing the method's performance in future uses. Unfortunately, due to time and resource constraint, we could not add more large datasets to the analysis besides HIV.

Figure 3 lacks clarity. A more detailed illustration is required to effectively display each component of the process. The figure should aim to distinctly outline and explain the functionalities of each part, ensuring that the figure conveys the intended information clearly and accurately.

We have updated Figure 3 with more details. Specifically, we put color boxes separating the inner and the outer optimization problems. We also added more descriptive texts on what each component learns, and the training loss used.

Dear reviewer,

We believe you have mistakenly submitted a review for another paper. Should you resubmit the correct review, we are happy to address any of your concerns and recommendations. Looking forward to hearing from you.