The Snowflake Hypothesis: Training Deep GNN with One Node One Receptive field

Thank you for your thoughtful and constructive reviews of our manuscript! Based on your questions and recommendations, we give point-by-point responses to your comments and describe the revisions we made to address them.

Weakness 1 & Question 1: The motivation is unclear; The motivation is unclear, why enhance deepGNN? many GNN use 2 or 3 layers but still achieve good performance. For example, [1] use 3 layers and achieve 73% on ogbn-arxiv, higher than the model's performance.

Thank you for your insightful question. We would like to respectfully state that, although it is true that shallow GNNs can outperform deep GNNs on certain datasets, recent explorations in deep GNNs are positioning them as the mainstream in the field of graph learning. It is noteworthy that a considerable number of SOTA methods ranking highly on the OGBN benchmark surpass not only 6 layers [1,2,3] but, in some cases, 32 layers [4]. Hence, we believe our exploration of deep GNNs contributes meaningfully to the community.

Weakness 2 & Question 2: overclaim, SnoH does not always improve deepGNN's performance; in Table 4, performance drops after using SnoHv1, so SnoHv1 is not always helpful for deepGNN.

Thank you for your keen observation! We acknowledge that SnoHv1, as evidenced in Table 4 with the 16-layer ResGCN+, led to a slight performance decrease (0.78$\downarrow$). We faithfully reported this based on the average results from 5 runs. However, we would like to humbly state that, in numerous other scenarios (refer to Tables 1 & 5), SnoH consistently enhances the performance of various GNN backbones. Overall, we believe it is not an overclaim to assert that SnoH can bring valuable performance improvements to all GNN backbones.

Weakness 3 & Question 3: The GNN backbone is far from STOA, both on Ogbn-Proteins and Ogbn-arxiv dataset; Is SnoH only limited to the GNN proposed here or it also enhance other models' performance, I would suggest evaluating SnoH on GNN models that have demonstrated strong performance on Ogbn-Proteins and Ogbn-arxiv.

Thank you for the constructive feedback! Indeed, the backbones (GCN/ResGCN/Res+) we selected are not the SOTA ones for current OGB datasets. However, as stated in our paper, the snowflake hypothesis is not confined to a specific GNN backbone; it is a universal operator that can be embedded in any (deep) GNN. To substantiate our standpoint, we have supplemented experiments demonstrating the application of the snowflake hypothesis to RevGNN [4] (ranked 4th and 26th on the Proteins/Products Benchmark respectively).

Table 3. Comparisons with SnoHv2 on Ogbn-Proteins. We report an average of 3 runs.

Table 3. Comparisons with SnoHv2 on Ogbn-Products. We report an average of 3 runs.

It can be observed that SnoHv2 has brought about a substantial performance improvement for RevGNN, with the enhancement becoming more pronounced as the number of layers increases (consistent with our observations in Tables 1 & 5).

---

References: [1] Label Deconvolution for Node Representation Learning on Large-scale Attributed Graphs against Learning Bias, Arxiv [2] Simplifying Graph Convolutional Networks, ICML'19 [3] Simple and Deep Graph Convolutional Networks, ICML'20 [4] Training Graph Neural Networks with 1000 Layers, ICML'2021

We express our sincere thanks for the detailed and thoughtful review of our manuscript and for the encouraging appraisal of our work. We give point-by-point responses to your comments and describe the revisions we made to address them:

Weakness 1:the text contains many filler-sentences ("We have another interesting observation") and hyperlatives ("These astonishing results clearly verify the effectiveness of our algorithm."). Please go over the text, and keep in mind that good writing should have active voice, be clear and concise. There is no need to write "we meticulously conduct a multitude of experiments to validate our hypothesis..." when one could just write "we validate our hypothesis...". This will make the text easier to read.

Thank you very much for pointing out the shortcomings in the writing of our paper. Based on your suggestions, we have read through the paper and removed the meaningless filler-sentences and revised some hyperlatives. For example, the sentence "These astonishing results clearly verify the effectiveness of our algorithm." We modify it to "These observations confirm our algorithm's effectiveness.”.

Weakness 2:The current abstract contains many awkward sentences (what does "Given that the potency of numerous CV and language models is attributable to that support reliably training very deep architectures" even mean?)

I appreciate your insightful suggestions regarding the shortcomings in our manuscript's abstract. We have tried our best to modify the vague sentences in it. For example the sentence "Given that the potency of numerous CV and language models is attributable to that support reliably training very deep architectures", we modify it to "The success of artificial intelligence in computer vision and natural language processing largely stems from its own ability to train deep models effectively.”.

weakness 3:I would also strongly suggest to remove the "red numbers" in Table 1, and instead add variances: The relevant part is what the average performance over 5 runs is, plus how much variance there will be between runs. The "best result out of N runs" is not valuable information, it's random-seed-lottery.

I'm grateful for for your suggestions on the experiments of our work. And I am very sorry that this is actually a mistake in our writing, all the values in Tabel1 are the average result of 5 runs, not the best result of 5 times.

Question 1: What is the running time (wall clock, FLOPS) of your method vs the baselines?

Thanks for your insightful question. Due to limited time, we have only presented part of the experimental results(The results are shown in Table 1), and we will try our best to add more experimental results to the revised manuscript in the future.

Table 1. Running time of different baseline using SnoHv2 on ogbn-arxiv. For each backbone we trained 500 epochs

Aggregation operations through adjacency matrices are a time-consuming part of graph neural networks. Therefore, the running time of our method (whether it is inference time or training time) depends on the sparsity of the adjacency matrix after pruning. The sparser the adjacency matrix, the shorter the time spent, while the original adjacency matrix is the densest. After adding SnoHv2, the baseline FLOPS will also decrease a bit, because the number of edges will decrease and the corresponding calculation will decrease.