Diffusing to the Top: Boost Graph Neural Networks with Minimal Hyperparameter Tuning

[Response to Weakness 1] Heterophilic datasets.

Thank you for your valuable suggestion. We would like to clarify that 4 datasets proposed in [1] have been included in our experiments. Detailed results and analyses for these datasets are provided in Appendix I.2. We hope this addresses your concern, and we appreciate your attention to ensuring a robust evaluation of heterophilic cases in our study.

[Response to Weakness 2] Clarification of Table 5 results.

This is a good point. However, we would like to clarify the main benefit of the graph condition: it provides significantly more stable generation quality while achieving comparably higher average accuracy. During our experiments, we observed that while the average accuracy of GNN-Diff is not always significantly better than p-diff, its standard deviation is consistently lower. This can be attributed to the following reasons:

• Similar Parameter Distributions: GNN-Diff and p-diff are trained on the same set of samples, leading them to approximate similar parameter distributions when properly trained. As a result, their average accuracies are close, although GNN-Diff generally achieves better accuracy in most cases.

• Guidance from the Graph Condition: The graph condition in GNN-Diff serves as guidance to regions of promising parameters within the overall distribution. This allows GNN-Diff to generate more centered and consistently high-performing parameters. The newly added analyses in Appendix I.5 further support this hypothesis.

Thus, while the average accuracy of GNN-Diff may overlap with the variation seen in p-diff results, the primary advantage of the graph condition lies in producing dense, high-performing parameters. This contributes to the practical value of our method, especially in scenarios where reducing the impact of randomness is crucial.

Thank you for your detailed review and for providing constructive feedback on our work. We appreciate your assessment and acknowledge the concerns raised. Your comments have helped us identify areas for improvement, and we have added more analyses on the graph condition to our paper accordingly. Below, we provide detailed responses to your comments. We hope these clarifications will address your concerns and please feel free to ask any further questions. If possible, we kindly hope that you could consider increasing to a more favourable rating, as your support will be very helpful with the acceptance of our work. Thank you again for your time and efforts.

[Response to Weakness 3] Theoretical analysis.

Thank you for the feedback. We acknowledge the importance of theoretical analysis in advancing understanding in many areas. However, as this work is designed as an empirical study, our primary focus is on demonstrating the practical effectiveness, stability, and generalizability of GNN-Diff across diverse tasks, models, and datasets. Given that hyperparameter tuning is a practical challenge in graph neural networks, our goal was to validate the utility of GNN-Diff through extensive experiments rather than theoretical derivations. We believe that this practical focus aligns with the study's objectives and provides meaningful insights for both research and application. Nonetheless, we appreciate your suggestion and will consider including theoretical perspectives in future work.

[Response to Question 1] G-Encoder ablation study.

This is an excellent question which leads to a valuable suggestion. When designing the implementation code, we conducted comprehensive trials for the G-Encoder architecture and selected the current version as a generally promising default option for GNN-Diff. However, we recognize the importance of including such analyses in the paper to validate this choice. To address this, we have added Appendix I.5, which includes two analyses:

• An ablation study on the components of the current G-Encoder;

• Comparison of alternative graph convolutions beyond GCN and their integration with MLP as the G-Encoder.

Please refer to the revised paper for further details.

[Response to Question 1] Typo.

Thank you for pointing this out. The typo has been fixed.

[Response to Question 3] GNN-Diff for low-capability scenarios.

Thank you for raising the important issue of model capability—this is a good perspective we had not previously considered. It indeed represents a practical challenge when implementing GNNs. However, we would like to emphasize that our proposed method, GNN-Diff, is well-equipped to address these challenges in the following ways:

• Challenge of GNNs Becoming Ineffective: We agree that graph data are diverse, and there is rarely a single GNN that outperforms all others across all datasets and tasks. To address this, one can conduct an efficient coarse search as an architecture search to identify promising GNNs while filtering out those that clearly underperform. The generation process can then be applied to the well-performing GNNs identified during the coarse search, further enhancing their performance.

• Challenge of Oversmoothing: Oversmoothing is a common issue in GNNs, but our method effectively addresses it by including the number of layers as a hyperparameter. This approach is considered in our experiments on long-range graphs and can be smoothly extended to other tasks.

We have also conducted experiments to evaluate whether GNN-Diff can boost the performance of ineffective GNNs. Using GCN for node classification on Cora and addressing the oversmoothing issue as an example, the results are presented in the table below. These results demonstrate that GNN-Diff not only improves accuracy but also enhances stability in terms of lowering the standard deviation, even when the target GNN is significantly affected by oversmoothing. However, the best accuracy is still achieved with an appropriate choice of the number of layers. Therefore, we recommend treating the number of layers as an important hyperparameter to tune when oversmoothing becomes a critical concern.

[Reponse to Weakness 1 & Question 2] Concerns related to the graph autoencoder and graph condition.

Thank you for your questions. Please allow us to address Weakness 1 and Question 2 together because they are both related to the graph autoencoder and the graph condition.

Why such a GNN architecture?

For the basic node classification, we designed the GNN architecture to effectively handle both homophilic and heterophilic graphs. Usually, graph convolution such as $\mathbf{AXW}$ aggregate neighboring nodes to integrate information in related samples and form more informative representations. However, this inevitably similarizes the neighboring node representations. This is helpful for homophilic graphs where connected nodes are generally from the same class. Since more similar representations better classify the connected nodes to the same class. However, for heterophilic graphs, where connected nodes may be from different classes, the similar representations make it hard to distinguish them to various labels. This is why we add MLP, $\mathbf{XW}$, known as the source term in relevant literature, to add back some variation in node features. The concatenation of the graph convolution and MLP is shown to be effective on both homophilic and heterophilic graphs in [1], because it allows the model to automatically decide the similarity (or smoothness) among node representations.

[1]: Beyond Homophily in Graph Neural Networks.

Would GAT or GCN be effective?

The short answer is yes, compared to having no graph condition, but they may not perform as effectively as the current architecture. To validate this, we have included a comprehensive analysis of the GAE architecture in the revised paper (Appendix I.5). This includes an ablation study on components of the current GAE architecture and an evaluation of alternative graph convolutions such as GATConv, SAGEConv, and APPNP. The results confirm the effectiveness of the current architecture, establishing it as a reasonable default for all target GNNs. However, exploring alternative graph convolutions could also be beneficial if computational resources and time permit.

We sincerely thank you for your positive feedback and high evaluation of our work. We are delighted that you found our contributions valuable and appreciate your recognition of their significance. We have carefully considered your suggestions to enhance the clarity and completeness of our paper. Particularly, your suggestion on using other graph embedding frameworks such as GAT helped us to enhance the analysis on the graph condition. Below, we address your points in detail.

Thank you for your feedback, which has been very helpful in addressing my concerns. However, I must admit that I am not very familiar with this field, and after reading the responses from other reviewers, I noticed several issues that I hadn’t initially anticipated. As a result, I decided to adjust my rating to a more conservative score. Nonetheless, I still believe the paper has merit and recommend it for acceptance.

Dear Reviewer,

Thank you for your engagement with our response and for the feedback you have provided throughout the review process. We sincerely appreciate the time and effort you have dedicated to evaluate our paper. While we feel deeply sorry to hear that you decrease the score, we are still very grateful for your support.

If there are any following questions or concerns, we would be happy to provide further clarifications. Thank you again for your time and acknowledgment on our efforts.

Sincerely,

The Authors

[Response to Weakness 3] Transferibility and potential extension.

Thank you for bringing the transferability to our attention. We understand your concern as the transferability is indeed not in the scope of our work. Yet, we would like to emphasize that our method is designed for the scenario in which one would like to boost GNN performance for a specific task to achieve the top potential. If transferability is taken into account, it may lead to a sacrifice of accuracy. In addition, the current design of GNN-Diff shows strong ability to boost various GNN models across diverse tasks and datasets. Thus, we expect that, with some adjustments, it is possible for it to gain transferability. We plan to explore further generalizations of our approach in future work, which may involve the method proposed in [1]. We appreciate your insights, and we will take them into consideration for future improvements.

[1] Diffusion-based Neural Network Weights Generation.

[Response to Question 1] Clarification on the upper bound of GNN-Diff.

Thank you - this is a good question. The global upper bound of our method for a specific task or dataset is indeed hard to measure, but we would like to clarify two key points regarding the intention of our method.

Firstly, our method is designed to be model-agnostic. The primary goal is to enhance the performance of any given GNN model, allowing it to reach its maximum potential. As such, the upper bound is not tied to any particular GNN but to how well our method can help fine-tune the performance of various models across different tasks and datasets.

Secondly, it is important to note that different GNNs may be better suited for different types of tasks or datasets. Some models may excel in certain scenarios, but underperform in different contexts. So, a more expressive model may not necessarily exist for all graph tasks. Classic GNNs, such as GCN, may even serve as strong baselines as shown in [2]. Therefore, rather than focusing on determining which GNN is inherently the most powerful, our method aims to optimize the performance of the target GNN on a specific task, helping it achieve its best possible results.

[2] Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification

[Response to Weakness 1] Technical contribution.

Thank you for raising this concern. In this response, we will first highlight the novelty and contribution of our method, and then discuss the modifications we have made in the revised paper to better improve the technical contribution.

To start with, we would like to clarify that the main contribution of our work is not to propose a new model component that has never been adopted before in relevant literature; but to explore an efficient and novel training strategy for GNNs with the minimal hyperparameter tuning, which has not been sufficiently studied so far. We agree with your concern that LDM and the GAE architecture are not new, and we have cited relevant works in our paper. However, lowering the hyperparameter tuning costs for GNNs with a parameter generative model is a brand new area for GNN research. The relationship between model components and our novelty is just like the ingredients and the recipe. The ingredients such as LDM, the graph convolution in GAE, and the sample collection are easy to obtain, but the design of the recipe (i.e., what ingredients to include and how to combine them for effectiveness and efficiency) is the true value of our work.

Then, to better improve the technical contribution of our work, we carefully considered the suggestion of more baselines for comparison. We will discuss more details in the response to Weakness 2.

[Response to Weakness 2-1] 120 supplementary experiments with Bayesian optimization as a baseline.

Thank you for the suggestion. We have revised the paper accordingly. Here are a few key points that we would like to highlight:

• In "Advanced Methods for Hyperparameter Tuning", we mentioned 3 types of techniques: parameter-free optimization, Bayesian optimization, and Hyperband. For Hyperband, we have used one of its variants, the coarse-to-fine (C2F) search. So, our revision majorly focused on parameter-free optimization and Bayesian optimization.

• Parameter-free optimization: While parameter-free optimization is a useful tool for tuning hyperparameters related to neural network training, such as the learning rate, this method is incapable of handling model hyperparameters such as the ChebNet K and APPNP alpha. So it is not considered as our baseline. We further clarify this limitation in Appendix A - Advanced Methods for Hyperparameter Tuning.

• Bayesian optimization: Bayesian optimization was not chosen as our comparison baseline because it is not common to apply Bayesian optimization for GNN tuning in GNN-related research. However, it indeed can be applied. Given that our paper is related to hyperparameter tuning, we do find your suggestion very valuable, and we have added experiments of Bayesian optimization. Please refer to Appendix I.4 in the revised paper for more details.

[Response to Weakness 2-2] Clarification on the coarse search results.

We suppose here is some misunderstanding. We will provide an explanation on why this is not an issue - and maybe even the "advantage'' of our method. But please kindly advise if our explanation does not address your question :)

Firstly, we agree with your observation that the coarse search underperforms compared to random and grid search in most cases. This is expected, as the coarse search operates over a very limited search space, which sacrifices exploration for tuning efficiency. However, our method, GNN-Diff, uses the coarse search as an initial step, then improves upon it through parameter generation.

Based on our experimental results, we make two key observations:

• GNN-Diff is able to outperform both random and grid search, even when starting from a relatively weak point provided by the coarse search;

• GNN-Diff requires significantly less time than both random and grid search.

These findings together demonstrate that GNN-Diff is an effective method for enhancing GNN performance within a short time frame. The coarse search provides a reasonable starting point, and GNN-Diff is able to build upon that to achieve superior results efficiently.

Thank you very much for the time and effort you dedicated to reviewing our paper, as well as for your valuable suggestions. We especially appreciate your insights on strengthening the technical contributions of our work. In the revised paper, we have added additional experiments and discussions to further clarify and enhance our work. Here, we would like to take this opportunity to address your concerns and clarify some potential misunderstandings. If possible, we sincerely hope that you could consider increasing the rating. Again, we greatly appreciate your expertise and support in the review process.

Dear Reviewer,

Thank you for your thoughtful and constructive feedback on our paper. We greatly appreciate the time and effort you have dedicated to reviewing our work and have carefully addressed your comments and suggestions.

To facilitate your review of our explanations and revisions, we have summarized our previous response below:

• We have conducted 120 additional experiments to further strengthen the technical contributions of our work. These include a detailed comparison of GNN-Diff with Bayesian optimization implemented using Optuna (Appendix I.4, page 32).

• For other concerns, such as the underperformance of coarse search, we have provided clarifications to demonstrate how it reflects the effectiveness of GNN-Diff. However, further suggestions are always welcome if our current explanation does not meet your expectations.

If you have any further comments or suggestions, we would be happy to address them promptly. Thank you again for your valuable feedback and thoughtful engagement with our work.

Sincerely,

The Authors

[Response to Weakness 5] Memory and time concerns.

Thank you for your questions and suggestions. Below, we address them point by point:

• Memory Costs: Memory costs were not reported as they were not a significant issue in our experiments. All experiments were conducted on a single NVIDIA 4090 GPU with 24GB memory, which was sufficient for all tasks. For large datasets, such as large graphs and long-range graphs, we employed clustering and batch training to mitigate memory usage. Most experiments required far less than the available memory.

• Why GNN-Diff is Faster Than Random Search: GNN-Diff outperforms random search in efficiency for two main reasons:

  • Reduced Search Space: GNN-Diff uses coarse search, which has a search space only half the size of the random search.

  • Minimal Training Overhead: Although GNN-Diff involves three trainable components, their training time is very short compared to the coarse search. The GAE training is equivalent to one run of GNN training, while search methods involve many more runs depending on the search space. The PAE and G-LDM training are computationally efficient due to the partial generation technique in our implementation. This approach considers only the last layer parameters, significantly reducing computational costs. Additionally, PAE encodes parameters into a low-dimensional space, which enhances the training and inference efficiency of G-LDM.

• Comparison with Coarse Search: Following your suggestion, we have updated Figure 7 to include the coarse search time. The revised figure clearly demonstrates that the coarse search accounts for the majority of GNN-Diff's time, supporting our claim that the training and inference time of GNN-Diff is relatively short.

We hope these clarifications could address your concerns.

[Response to Weakness 6] Graph classification.

Yes, our method can be adapted to graph classification. In most cases, the major difference between graph-level and node-level classification tasks is the graph pooling process. So, we may extend to graph classification by adding a pooling layer to the GAE architecture and also the target models. The pooling layer obtains global graph representations for downstream tasks. In Appendix I.3, we discuss more details related to graph classification along with the experiment results on Fake News datasets [4] to show the effectiveness of our method in graph classification.

[4] User Preference-aware Fake News Detection

[Response to Question 1] Explanation of G-Encoder architecture.

Thank you for the question, and here is the explanation. As we know, graph convolutions such as $\mathbf A\mathbf X\mathbf W$ and $\mathbf A^2\mathbf X\mathbf W$ aggregate neighboring node features to construct more informative representations, which inevitably similarizes neighboring node representations. This is useful for homophilic graphs, where connected nodes are usually from the same class. As similar representations better map the connected nodes to the same class. However, this may be harmful for heterophilic graphs, where connected nodes are usually from different classes. This is why some GNNs may perform even worse than MLP on heterophilic graphs.

GNNs that can handle both homophilic and heterophilic graphs well typically incorporate dynamics that can bring the variations of features back, e.g., through adding the source terms in a similar way as skip connection [5, 6]. In this way, the model can choose to smoothen or differentiate node representations in the learning process. Accordingly, we adopt equation (2) with the concatenation of different receptive fields (e.g., graph convolutions $\mathbf A^2\mathbf X\mathbf W_1,\mathbf A\mathbf X\mathbf W_2$, and the source term $\mathbf X\mathbf W_3$). Furthermore, we note that our message passing structure (e.g., equation (2)) is also aligned with some recently developed GNNs, such as H2GCN [7], which serves as one of the state-of-the-art models in terms of learning with both homophilic and heterophilic graphs.

[5] From Continuous Dynamics to Graph Neural Networks: Neural Diffusion and Beyond.

[6] GREAD: Graph Neural Reaction-Diffusion Networks.

[7] Beyond Homophily in Graph Neural Networks.

[Response to Weakness 2] Motivation of GAE.

Thank you for the question; it’s an excellent point. We previously put the motivation of GAE in Section 5.3 - Ablation Study on the Graph Condition. However, we recognize that placing it in Section 4.2 - Graph Autoencoder, where GAE is formally introduced, will make it clearer for readers. We have made this adjustment accordingly. Please see line 194-197 in the revised paper.

To answer the question, the motivation for including GAE is threefold. (1) Contribution: The role of specific data characteristics in parameter generation still remains underexplored. To fill this gap, GNN-Diff leverages the graph data and structural information as the generative condition. (2) Usage: GAE is used to construct the graph condition, which is considered as a data and graph structure-aware guidance on the GNN parameter generation. (3) Accuracy & Stability: We show with the ablation study in Table 5 that GNN parameters generated with the graph condition generally lead to better average accuracy and higher stability across various tasks.

Further, in terms of the training process, we train the three components one by one in a consecutive training flow. The training of GAE is prior to the training of G-LDM, because its encoder will be used to construct one of the inputs of G-LDM, that is, the graph condition.

[Response to Weakness 3] Ablation Study.

We appreciate this suggestion and have enriched the ablation study accordingly. Please refer to Appendix I.5 for detailed experiment results and analyses.

[Response to Weakness 4] SOTA GNNs.

Thank you for highlighting three very intriguing papers on large-scale graphs. As a matter of fact, we originally considered such works during our experiment design, but we eventually decided to focus on exploring "graph convolution architectures'' rather than "efficient GNN training algorithms'' (i.e., these papers share the same characteristic in emphasizing efficient training strategies rather than proposing new convolutions). Nevertheless, we appreciate your reminder of their relevance, and we acknowledge their importance for scalability in real-world challenges. To address this, we have cited these papers and added a discussion on the potential of combining our method with these algorithms. Please find the discussion in Appendix I.7 of the revised paper.

To further clarify, please see below for more explanations of why the three papers all focus on efficient training algorithms rather than proposing new architectures:

• [Training Graph Neural Networks with 1000 Layers] This paper proposes reversible connections to enable the training of very deep or very wide GNNs for large graphs. The proposed algorithm is generic and can be theoretically applied to any GNNs, including GCN, SAGE, GAT, etc.

• [Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced Training] SAGN proposed in this paper gathers multiple-hop representations with the attention mechanism. This is very similar to one of our target model, MixHop. Since MixHop is not included in the large graph experiments due to its computational complexity, this paper can be a very good direction for our future work. This paper also proposes Self-Label-Enhanced training, which is also a generic training algorithm.

• [LazyGNN: Large-Scale Graph Neural Networks via Lazy Propagation] This paper introduces lazy propagation and combines it with generic sampling strategies. Lazy propagation can be regarded as a way to organize and compute GNN layers, and the graph convolutional layers of LazyGNN are very similar to one of our target model, APPNP.

Accordingly, we did not include these works as target models in our main experiments but have discussed GNN-Diff's extension to SOTA training algorithms in Appendix I.7.

Thank you for highlighting this point. We also had the same observation during our experiments. As discussed in Section 5.1 - Reproducibility and comparability, this discrepancy is primarily due to differences in data split and model architectures. We would like to emphasize that our experimental settings mostly align with standard practices in the original paper of OGBN datasets [1], except for the following aspects:

• Cluster vs. Full training: As mentioned in Section 5.1 - Experiment details, we conducted all experiments on NVIDIA 4090 GPU with 24GB memory. Given the limited memory, the large graphs are split into clusters following [2]. This may not be necessary for relatively smaller datasets such as Flickr, but we apply the same setting to maintain consistency. Clusters sometimes lead to inferior performance especially comparing with the full graph training adopted by most Leaderboard GNNs. However, this does not affect the validity of experiments to show GNN-Diff's ability in boosting GNN performance on large graphs.

• No tricks vs. Tricks applied: We didn't apply tricks such as skip connection, which is considered in GraphSAGE [1]. The inititial goal of our experiments is to investigate the top potential of vanilla GNNs.

• 2-layer GNN vs. 3 or more layers: We adopted 2-layer GNNs in large graph experiments following the same setting as the basic node classification experiments. In comparison, GNNs in [1] adopt 2-3 layers, while GNNs in [3] adopt 2-10 layers. This factor may not be as influential as the previous two, but we suppose it may still cause some differences in accuracy.

To further validate the effectiveness of our method, we repeat the experiment for GCN and SAGE on OGB-arXiv strictly following the settings in [1]. In the table below, we present the result, and it is shown that GNN-Diff can achieve better accuracy with lower standard deviation than the result reported in [1] (GCN 71.74% $\pm$ 0.29% and SAGE 71.49% $\pm$ 0.27%).

[1] Open Graph Benchmark: Datasets for Machine Learning on Graphs

[2] Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks

[3] Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classification

Thank you for your thoughtful review and valuable feedback. Based on your comments, we recognize your expertise in large-scale graphs and appreciate your insights. We have revised our paper to address your concerns, clarified potential misunderstandings, and made improvements as detailed below. We hope these revisions demonstrate the soundness of our work. We also hope you could kindly reconsider the rating. Thank you again for all the constructive suggestions and your time.

Dear Reviewer,

Thank you for your thoughtful and constructive feedback on our paper. We sincerely appreciate the time and effort you have devoted to reviewing our work and have carefully considered your comments and suggestions.

To assist in reviewing our explanations and revisions, we have provided a brief summary of our previous response below:

• We have addressed several of your concerns through revisions to the paper and additional experiments, particularly focusing on the motivation for GAE (Section 4.2, page 4), the ablation study (Appendix I.5, pages 33-34), and graph classification (Appendix I.3, pages 30-31). These updates aim to improve the clarity and contributions of our work, and we hope they align with your expectations.

• Some of your other recommendations, such as including additional SOTA GNNs on large graphs, extend beyond the immediate scope of this study. However, we have included discussions on these directions as potential future work in our revised submission (Appendix I.7, page 35) to illustrate how our approach could be further expanded.

If you have any additional comments or suggestions, we would be happy to address them promptly. Thank you again for your valuable feedback and engagement with our work.

Sincerely,

The Authors

Dear Reviewer,

We would like to extend our warmest thanks to you for taking the time to review our responses and for increasing our score. We deeply appreciate your constructive suggestions and insights, which will undoubtedly contribute to the further improvement of our work.

We are committed to addressing the concerns you have raised and will follow up with our detailed responses as soon as possible. We believe it won't take long and thanks in advance for your patience.

Best regards,

The Authors

Dear Reviewer,

Thank you very much for further increasing the score of our submission and for your thoughtful questions! We also find these questions very interesting and inspiring to explore. While we are unable to revise the paper at this stage, we are committed to investigating these points and will share any notable findings through an anonymous GitHub link (the updated findings can be included in the camera-ready version of the paper if it is accepted). We will follow up as soon as possible.

Thank you once again for your valuable feedback and support.

Warm regards, The Authors