GraphFM: A generalist graph transformer that learns transferable representations across diverse domains

1. A primary limitation of the model is its dependency on unique initial MLPs and final predictors for each graph dataset, which necessitates fine-tuning for every new dataset or task. This requirement significantly hinders the model’s practicality in real-world applications.

2. The pre-training results were largely anticipated, given the model’s supervised training approach. However, in real-world scenarios, labeled data is often scarce, making these results difficult to scale for large-scale graph model training. Notably, what stands out about large language models (LLMs) is their ability to leverage self-supervised learning, achieving scaling laws and emergent abilities.

3. Despite extensive pre-training and fine-tuning on downstream datasets, the model’s performance only matches that of specialized models, without offering substantial improvement. Considering that the model still requires fine-tuning on new datasets, it raises questions about the benefits of this approach over simply training specialized models directly.

4. In Figure 5, the authors compare the proposed model to GCN and NAG on heterophilic graphs and the Coauthor-CS dataset. Could the authors provide comparisons between the proposed model and models with comparable performance on homophilic graphs? Additionally, how does varying the hyperparameter settings affect the performance of the proposed model?

1. How does the model handle more challenging graph tasks, such as link prediction, given its emphasis on node classification?
2. The paper’s focus on node classification limits its scope and raises concerns about the broader applicability of the approach, especially given the minimal evaluation on other graph tasks.
3. The method is not novel. The paper is more like a technical report.

• Some notations are not clearly defined. For example, the expression $\tilde{\mathbf{u}}_i=\operatorname{MLP}_g\left(\mathbf{u}_i\right)$ appears only once in the paper, and the meaning of $\tilde{\mathbf{u}}_i$ is unclear. Additionally, the calculation of the position encoding $\mathbf{p}_i$ and how $\mathbf{x}_i$ concatenates a projection of the node features are not sufficiently explained.
• The novelty of GraphFM architecture is limited. GraphFM builds upon transformer-based architectures like the Perceiver to encode graph-specific details into a shared latent space. Although this improves generalization across diverse domains, the underlying architecture does not introduce fundamentally new mechanisms for graph representation learning. A similar mechanism has also been widely used in some related research papers, such as in [1] and [2].
• The experimental results are not convincing, and the baseline methods need to be stronger. Although the authors compare the GraphFM method with some widely used baseline methods like GCN, GAT, SAN, and NAG, these are neither the latest nor the most competitive, undermining the reported performance's significance. Additionally, GraphFM does not achieve state-of-the-art (SOTA) performance in most cases.

[1] Liu C, Zhan Y, Ma X, et al. Gapformer: Graph Transformer with Graph Pooling for Node Classification[C]//IJCAI. 2023: 2196-2205.

[2] Gui A, Ye J, Xiao H. G-adapter: Towards structure-aware parameter-efficient transfer learning for graph transformer networks[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2024, 38(11): 12226-12234.

1. The novelty of this work is not high. It mainly uses the graph transformer cimbining with some engineering effort, like distributed tranining.

2. The paper claim that most GNN train on individual graph, which is not true. GNNs can tranin on mutiple grpah as well.

i) GPT-GNN: Generative Pre-Training of Graph Neural Networks, KDD 2020 ii) GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training, KDD 2020

3. Traning one model for multiple graphs is not new, in this work, it seems that the backbone is just chaaning from GNN to Graph transformer. Any different insight?

The contributions of the paper are fairly limited, as the authors have not established the premise of developing their foundation models vis-a-vis some of the existing works, and the evaluation is also not convincing.

First, I recommend that the authors consider the survey paper, "A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspective," and also the tutorial on Graph Foundation Models in WWW'24. The authors have not cited the former, and also not compared and contrasted with the methods discussed in the survey paper.

Second, the authors should also look at, "Learning MLPs on Graphs: A Unified View of Effectiveness, Robustness, and Efficiency," in ICLR'24. While the paper is not focused on pre-training, but the MLP construct holds similarities to the work developed by the authors in their paper.

Third, the authors do not provide any context on why the particular data were used for validation, and not others. As such, it is fairly unconvincing, and there are also no statistical significance offered in the table of results.

Fourth, the authors have not compared their performance to heterogeneous graph neural networks (there are a number of recent papers on this topic), and hence the performance comparisons are not appropriately contextualzied, especially on heterogeneous graphs.

Fifth, why the focus only on node classification? If it is a pre-trained foundation models, then should there not be more generalizability offered on downstream tasks?