Can Classic GNNs Be Strong Baselines for Graph-level Tasks? Simple Architectures Meet Excellence

Thank you for your thorough review and for acknowledging our contributions. We sincerely hope our response below further strengthens your confidence in our work.

(1) Related Works

Thank you for sharing these related works. We have integrated detailed discussions of these relevant studies ([2], [3], [4], and others) into the revised manuscript to better contextualize our contributions.

(2) Performance vs. Efficiency Trade-off Analysis of Fast Baseline Models and GNN+

Thank you for your thoughtful question. The GT baselines used global attention mechanisms with quadratic computational complexity, resulting in higher computational costs without performance gains compared to GNNs. To address your question, we conducted additional experiments using two linear baseline models—HRN [3] and LDP [4]—and compared their performance with GNN+.

First, we evaluated these models on five small-scale graph classification datasets (IMDB-B, IMDB-M, COLLAB, MUTAG, and PTC) used in [3, 4]. These datasets consist of only 300–5000 graphs, substantially smaller than the main datasets (>10,000 graphs) used in our benchmarking study. Under identical experimental setups from their papers, GCN+ achieved comparable performance to these fast baselines. Given the small size of the datasets, their computation times are equally fast.

Further, we extended this comparison to large-scale OGB datasets (ogbg-molhiv and ogbg-molpcba) using identical experimental setups. Although LDP and HRN showed faster training times than GCN+, their performance was significantly lower. This highlights the importance of rigorous benchmarking on larger scales. Evaluating models solely on small datasets, like MUTAG and PTC, can mask true differences in generalization capabilities. Results from large-scale datasets clearly demonstrate GCN+'s superior predictive performance, validating our focus on comprehensive and systematic benchmarking.

Thank you once again for helping us improve our work. These results have been included in the revised version of our manuscript.

(3) Future Theoretical Research

We appreciate your comment and suggestion. We have incorporated a discussion of limitations and future research directions into the revised manuscript.

Our work is an empirical benchmarking study (please see our reply to Reviewer s2AU "(1) Theoretical Analysis"). While deeper theoretical exploration would be valuable, such analysis lies beyond the scope of this benchmarking study.

As noted by Reviewer 5voA, "While the paper does not introduce new theoretical innovations, it effectively synthesizes existing research, providing valuable insights and a comprehensive summary of current knowledge in the field."

The empirical insights derived from this work, particularly from the ablation studies (Tables 5 and 6), offer valuable guidelines for researchers in designing and applying GNNs to graph-level problems. These findings also lay a solid foundation for future theoretical investigations.

(4) Hyperparameter Tuning and Fairness of Comparison between GTs and GNNs

Thank you for highlighting this concern. We applied equally thorough hyperparameter tuning to all GT baselines, incorporating edge features, normalization, dropout, residual connections, FFNs, and positional encoding. All models were retrained using the same hyperparameter search space as GNN+ (lines 251–255).

Notably, GNN+ remains robust and dataset-agnostic. Please see our response to TFXX, "(2) Fair Comparison between GTs and GNNs".

(5) Hyperparameter Sensitivity Analysis

Thank you for your thoughtful suggestion. In response, we conducted additional experiments to systematically evaluate the sensitivity of GNN+ performance to critical hyperparameters, specifically dropout rates (link) and the number of layers (link).

• Dropout Rates: Our results indicate that a low dropout rate (≤ 0.2) is sufficient and optimal, whereas higher dropout rates significantly degrade performance.
• Number of Layers: Residual connections enable GNN+ to achieve optimal performance across a wide range. Increasing the number of layers does not lead to sudden performance drops, indicating effective mitigation of the over-smoothing problem.

For other hyperparameters that are binary (enabled or disabled), please refer to our ablation studies in Tables 5 and 6.

Thank you for recognizing the depth of our experiments and insights. We believe some key points may have been missed and hope our clarification encourages you to revisit and re-evaluate our work.

(1) Deeper Theoretical Analysis

Thank you for your valuable comment. Please refer to our response to Reviewer s2AU, "(1) Theoretical Analysis".

(2) Fair Comparison between GTs and GNNs

We'd like to clarify that our performance comparison is fair.

Firstly, the techniques used for GNN+, regarded as hyperparameters—including edge feature module, normalization, dropout, residual connections, FFNs, and positional encoding—have already been incorporated into all GT baselines. Importantly, we retrained the GT baselines using the same hyperparameter search space as the classic GNNs (lines 251–255). Therefore, the application of these techniques is consistent across both GNN+ and GT baselines.

Secondly, to comprehensively address your concern, we conducted additional experiments utilizing a fixed model with all six techniques integrated, denoted as GNN+ (fixed), as presented in the table below. The results show that the fixed GNN+ model achieves performance comparable to the best configurations reported in the paper, demonstrating its practicality and robustness.

(3) Justification for Architectural Choices

There are multiple ways to combine six techniques through operations like reordering, yet the GNN+ formulation is fixed as in Equation 11.

Indeed, the GNN+ framework is not limited to the formulation shown in Equation 11. Instead, it represents a flexible GNN architecture that integrates edge features, normalization, dropout, residual connections, FFNs, and positional encoding, which collectively define GNN+. Each component allows various implementation choices; for instance, both BN and LN are viable options for normalization.

The specific combination presented in Equation 11 was chosen because our experiments consistently showed strong and, in some cases, remarkable performance. However, our intention was not to advocate exclusively for this configuration. Rather, our primary goal was to clearly demonstrate, through systematic empirical benchmarking, that an enhanced GNN architecture can match or surpass GTs across diverse datasets. Future research can explore alternative architectures and refinements within this flexible framework.

(4) Insights into How to Enhance GNNs for Graph-Level Tasks

Our ablation studies (Tables 5 and 6) provide detailed, actionable insights. For example, normalization substantially impacts larger-scale datasets but has less effect on smaller datasets, and a low dropout rate (≤0.2) consistently proves optimal. These findings serve as practical recommendations for researchers seeking to enhance GNN performance.

(5) Why Attention Fails

Thank you for the thought-provoking question. This phenomenon is the core finding of our study. While GTs employ the global attention mechanism, our results suggest that it does not benefit graph-level learning as expected and may even degrade performance by introducing unnecessary complexity.

To investigate this, we visualized the attention scores of GraphGPS on the Peptides-func dataset (see this link). The visualization shows that the nodes in question (highlighted with green borders) predominantly attend to only one or a few distant nodes, or randomly attend to multiple distant nodes without clear, explainable patterns.

An ablation study is presented in the table below. The results clearly indicate that the global attention mechanism used in GraphGPS may negatively impact performance. We hypothesize that the global attention mechanism conflicts with the local message-passing mechanism of GNNs, causing the model to excessively attend to distant, less relevant nodes. This phenomenon, referred to as the over-globalizing problem, has also been experimentally validated in recent studies [1].

[1] Less is More: On the Over-Globalizing Problem in Graph Transformer, ICML 2024.

We greatly appreciate the very detailed feedback and your recognition of our contributions! We hope our response below will further enhance your confidence in our work.

(1) Comparison with SOTA GNNs Addressing Over-smoothing and Over-squashing

To enhance your paper's contributions, it is recommended to incorporate and compare your proposed method with state-of-the-art techniques addressing over-smoothing and over-squashing in Graph Neural Networks (GNNs) (you mentioned in the first sentence of the Abstract), such as Multi-Track Message Passing and Cooperative Graph Neural Networks. Conducting comparative experiments with these methods will provide a comprehensive evaluation of your approach's effectiveness and highlight its contributions to the field.

We appreciate the thoughtful suggestion. Following your recommendation, we conducted additional experiments on the Peptides-func and Peptides-struct datasets, comparing GCN+ with SOTA GNNs including Multi-Track Message Passing (MTGCN) [1] and Cooperative Graph Neural Networks (CO-GNN) [2]. These experiments adhered to the hyperparameter settings described in our paper (Lines 264-273).

For CO-GNN, we tuned its message-passing mechanisms, including SUMGNNs, MEANGNNs, GCN, GIN, and GAT, as recommended by the original paper. For MTGCN, we optimized the number of message-passing stages (from 1 to 4) to ensure a fair comparison.

The comparative results in the table below clearly demonstrate the effectiveness of our GNN+ framework in addressing over-smoothing and over-squashing. We have included the results in the revised version.

(2) The Impact of the Number of Network Layers on GNN+

Additionally, performing experiments that involve increasing the number of network layers to observe performance changes can help assess and address the over-smoothing issue in GNNs.

Thank you for the thoughtful suggestion. In response, we conducted additional experiments to examine the impact of the number of network layers on GNN+ for the PATTERN, CLUSTER, and PascalVOC-SP datasets. The detailed results can be found at this link.

Thanks to residual connections, GNN+ achieves optimal performance across a wide range of layers. Specifically, on PATTERN, both GNN+ variants (GCN+ and GatedGCN+) achieve optimal performance at 12 layers. In contrast, on CLUSTER and PascalVOC-SP, their performance continues to improve as the number of layers increases.

Overall, GNN+ maintains strong predictive performance even at greater depths, demonstrating its ability to effectively mitigate the over-smoothing issue commonly observed in GNNs.

(3) Why Not Consider Graph Node Classification

Why doesn't the author focus on graph node classification instead of graph classification?

Thank you for your question. Our work is inspired by a recent study [3] that showed classic GNNs can achieve performance comparable to, or even surpassing, state-of-the-art GTs for node-level tasks, such as node classification. However, there has been no similar conclusion or investigation for graph-level tasks, and our work aims to fill this gap.

Please note that we have addressed inductive node classification, which is considered one of the graph-level tasks. Specifically, Table 2 and 3 (in our original manuscript) include results on the PATTERN, CLUSTER, PascalVOC-SP, and COCO-SP datasets, which evaluate the performance of inductive node classification.

In addition, although our GNN+ framework is specifically designed for graph-level tasks, it can also be applied to node-level tasks. Below are the node classification results on four datasets from CO-GNN: roman-empire, amazon-ratings, minesweeper, and questions. The results indicate that GCN+ consistently achieves performance comparable to CO-GNN.

---

[1] Multi-Track Message Passing: Tackling Oversmoothing and Oversquashing in Graph Learning via Preventing Heterophily Mixing, ICML 2024.

[2] Cooperative Graph Neural Networks, ICML 2024.

[3] Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification, NeurIPS 2024.

Thank you for your constructive feedback. We believe there may have been some misinterpretations of our work that could have influenced your assessment. We hope our clarifications encourage you to reassess our work.

(1) Theoretical Analysis

This claim holds limited value as it lacks relevant theoretical analysis and deeper mechanistic insights.

We’d like to clarify that our work is an empirical benchmarking study, akin to previous notable benchmarking research such as [1,2,3,4,5]. Consequently, theoretical analysis is neither intended nor within the scope of this work.

Furthermore, we'd like to explain why the empirical study alone represents a significant contribution to the community.

In recent years, GTs have emerged as the leading approach for graph-level tasks, often dominating leaderboards, especially on small molecular graphs. This trend has fostered a growing perception that GTs, due to their global attention mechanisms, are inherently superior to GNNs, leading to the marginalization of GNNs in the field.

However, our comprehensive and fully reproducible experimental results, as presented in Tables 2-4, provide compelling evidence that classic GNNs, when enhanced with our GNN+ framework, can consistently match or outperform GTs across a diverse suite of graph-level tasks.

This finding has significant implications:

1. It suggests the global attention mechanisms of GTs may not be as useful as commonly believed. In fact, our ablation study shows they may degrade performance (see our response to Reviewer TFXX, "(5) Why Attention Fails").

2. It questions the need for complex architectures in graph-level tasks and could prompt a methodological shift from overly complex GTs to simpler GNN models, potentially reshaping the field's landscape.

3. It explains why state-of-the-art GTs often incorporate message-passing mechanisms into their models, either implicitly or explicitly, due to their high effectiveness.

As noted by Reviewer 5voA, "While the paper does not introduce new theoretical innovations, it effectively synthesizes existing research, providing valuable insights and a comprehensive summary of current knowledge in the field."

[1] Pitfalls of Graph Neural Network Evaluation, NeurIPS 2018.

[2] A Critical Look at the Evaluation of GNNs under Heterophily: Are We Really Making Progress, ICLR 2023.

[3] Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmarking, NeurIPS 2023.

[4] A Fair Comparison of Graph Neural Networks for Graph Classification, ICLR 2020.

[5] Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification, NeurIPS 2024.

(2) Insight into Why Certain Components Help

The study's utility in guiding researchers to design or apply GNNs for graph-level tasks remains constrained.

Our ablation studies (Tables 5 and 6) provide empirical insights into the contributions of each component of the proposed GNN+ architecture. For example, normalization has a more substantial impact on larger-scale datasets while being less pronounced on smaller ones; similarly, a very low dropout rate (≤0.2) consistently proves optimal for graph-level tasks. These findings greatly improve the usefulness of our work in helping researchers design and apply GNNs for graph-level problems. While we agree that deeper theoretical exploration would further benefit the community, such analysis lies beyond the scope of this benchmarking study. Nonetheless, our empirical results lay a solid foundation for future theoretical investigations into how these architectural components enhance graph-level expressiveness and effectiveness.

(3) Methodological Novelty

The core methodology (GNN+) constitutes a direct extension of Luo et al.'s framework for graph-level tasks.

It is important to note that GNN+ is NOT a trivial extension of Luo et al.'s model, which is intended for node-level tasks. Graph-level tasks, especially those involving small-scale molecular graphs, pose unique challenges. To address these, GNN+ integrates various techniques specifically tailored for graph-level modeling, such as edge feature module, FFNs and PE. The simple yet effective design of GNN+ was established through extensive experiments conducted over half a year, representing a new framework in the field.

(4) Visualization Results

Thank you for the suggestion. Following the advice, we have added additional visual analyses (e.g., t-SNE plots of learned embeddings (link), sensitivity analysis of network depth (link) and dropout rate (link)). In the t-SNE figure, we observe that the graph embeddings generated by GCN+ exhibit greater inter-class distances compared to those generated by GCN. We have incorporated the results in the revised version.