Scalable Message Passing Neural Networks: No Need for Attention in Large Graph Representation Learning

Cons:

1: The SMPNN can maintain its performance but does not gain any advantages from a deeper network. What causes this?

2: For larger graph datasets, the improvement from SMPNN is less pronounced. For example, on the ogbn-papers-100M dataset, the improvement is only 0.2%. Could this suggest that the model size is still inadequate? If we use a larger network for both SMPNN and Graph Transformer, SMPNN should experience less oversmoothing and demonstrate a more significant accuracy improvement.

1. In Section 3.2, the authors label their proposed block as a "transformer block." However, the SMPNN framework lacks any attention mechanism, which is a core component of transformers. Consequently, SMPNN functions more like a deep GCN with residual connections rather than a genuine transformer model. This categorization is misleading, as attention mechanisms fundamentally distinguish transformers by enhancing scalability and representation capacity in large graph models. Existing models such as GCNII [1], EGNN [2], and DeeperGCN [3] have already explored architectures with enhanced depth. Although these models improve scalability, they are still linear and inherently limited compared to transformers.

2. Table 1 mentions that the SMPNN has training time complexities comparable to other transformer-like models, such as GraphGPS and Exphormer. However, the experiments do not include comparisons with these transformer-based models, especially models with higher computational complexity, like Graphormer, that might showcase different trade-offs in performance versus scalability. While SMPNN is intended to scale to larger graphs, demonstrating performance across various data scales, especially small to medium datasets, would test its broader applicability.

3. SMPNN relies on local message-passing operations without incorporating global attention mechanisms, which may limit its ability to capture long-range dependencies effectively. For tasks where global context is essential, SMPNN's performance could be suboptimal compared to transformer-based models that use global attention mechanisms. A discussion on this limitation and potential strategies would strengthen the paper.

[1] Chen, Ming, et al. "Simple and deep graph convolutional networks." International conference on machine learning. PMLR, 2020. [2] Zhou, Kaixiong, et al. "Dirichlet energy constrained learning for deep graph neural networks." Advances in Neural Information Processing Systems 34 (2021): 21834-21846. [3] Li, Guohao, et al. "Deepergcn: All you need to train deeper gcns." arXiv preprint arXiv:2006.07739 (2020).

• The contribution of this paper is weak. The main focus is replacing the attention module in the transformer with a message-passing module and using the residual connections to alleviate the over-smoothing problem. However, the use of residual connections to address over-smoothing has already been explored in DeepGCN[1], which this paper does not mention or compare. Additionally, the implementation of deep GNNs has been studied in several other works[1]-[4].

• The paper contains a substantial amount of repetitive descriptions of existing work, such as message-passing neural networks and GCN (Eq. 2/3 vs. Eq. 6). Additionally, the theorem introduced in Section 4.1 is neither utilized later in the paper nor a contribution of this work.

• The contributions of the paper also involve repetitive work, such as a) using residual connections to mitigate over-smoothing; b) based on reference [5], Theorem 4.4 is evidently valid.

• Empirical analysis lacks heterophilic benchmarks to validate the effectiveness of SMPNN on over-smoothing.

• In tab. 7, SMPNN achieves the best performance on ogbn-proteins with 12 layers. Does SMPNN require significantly more computational resources and parameters to achieve comparable or better performance to baseline models?

[1] DeepGCNs: Can GCNs Go as Deep as CNNs? ICCV'19

[2] Graph Convolutional Networks via Initial residual and Identity Mapping, ICML'20

[3] Training Graph Neural Networks with 1000 Layers, ICML'21

[4] Revisiting Heterophily For Graph Neural Networks, NeurIPS'22

[5] How Powerful are Graph Neural Networks? ICLR'19

The experiments are not strong enough. For instance, all SMPNN variants should be included consistently across tables and figures in the section. The same applies to baselines, unless there are reasonable explanations for exclusions. Additionally, the distinctions among SMPNN variants and the strengths of SMPNN are not clear. For example, SMPNN uses significantly more GPU memory than SGFormer, yet the paper still claims that it does not use more memory than the baselines. Some presentation needs to modified, for example: FF notation needs to be written consistently with the rest of the paper.