GraphProp: Training the Graph Foundation Models using Graph Properties

1. The pre-computed graph properties are a set of manually selected, discrete values. The motivation that cross-domain structural features can be extracted through regression of these values requires empirical validation. Additionally, the computational complexity varies significantly with different graph sizes, resulting in limited scalability of the method.
2. According to the formal definition of the method, the authors assume consistency in node counts, input spaces, and output spaces across domains. While using language models to extract node attributes can ensure input space consistency, other assumptions are difficult to satisfy in real-world applications, which limits the method's applicability.
3. The experimental comparisons only include OFA as a baseline method for graph cross-domain generalization. This limited comparison is insufficient to demonstrate the superiority of the proposed method.

• From the novelty and contribution perspective, while the idea of universal graph representation is interesting and promising, the proposed pipeline to acquire such an ability seems implausible to me. Specifically,

  • Can a graph transformer with the proposed positional encoding predict all proposed graph properties? Note that the graph transformer and the spectral encoding are constrained by 3-wl expressivity, and is their combination (theoretically) capable of predicting all the properties? You should theoretically or empirically justify that the proposed methods can indeed predict the targets.
  • Suppose the model has the expressivity to predict the set of properties, does this learning pipeline suffice to be called a "foundation model?" On one end, you will always need to design new properties to fit the new data. On the other, the current model only tackles graph classification problem, but ideally one would want a foundation model to solve all tasks in a domain.

• From the presentation perspective, the motivation example seems hand-waving to me. The representation for TSG is much more uniform across datasets, and most words are "Node X: Connected to", causing high correlation among datasets. Whereas the representation for TAG can be a lot more diverse as the node description involves atom names, which can differ quite significantly across datasets, leading to lower correlation. However, this difference in semantics does not say much about the transferability of TAG and TSG, it also measures the semantic similarity, which is heavily influenced by how you setup the text description but not by the inherent information. I understand the message you try to convey, yet the example does not really make sense to me. I suggest providing a more rigorous analysis of the transferability not based on the textual description.

• From the experiment perspective, you should consider adding more baselines for a comprehensive comparison, and including several interesting works, such as GIMLET and LLM4Mol, is still important. It seems like you also only conducted graph-level tasks. Moreover, you should compare your model with a variant where you do not use a pre-trained graph property predictor, but, instead, you can directly concatenate the set of graph properties to the representation you obtained from the LLM. This is particularly important to validate the property prediction pre-training. You should also report how well your property predictor perform, because it also makes sense to use such a predictor when it's doing its job well. Does your method apply to link and node tasks? If so, it would be nice to have those results.

W1. It's not clear how to apply GraphProp to zero-shot scenarios. Although I know that the used baseline (OFA) can be applied to zero-shot, as a reader, I am more curious to see how the proposed model can enhance the model performance in the zero-shot scenario. Especially how to use comprehensive training part in a zero-shot scenario. I suggest the authors provide a detailed explanation for this.

W2. Figure 1 is missing a comparison with noise. From the prompts of TSGs and TAGs, there are more meaningless but similar words (e.g. "connected to" and "and") between TSGs of different graphs. This may be a reason why it seems that Figure 1 (a) has higher similarity than Figure 2 (a). I suggest the authors add meaningless noise prompts for comparison (e.g. TSGs generated by randomly linked graphs) to further prove the point.

W3. There are some writing errors in the paper. For example, Appendix A should be deleted. In addition, although I think the section 2 is well written and makes it easy for the reader to understand the basics, it is too long, and the introduction of the methods and experiments section seems a bit inadequate. Perhaps some of the subsections in section 2 could be combined or bolded instead of being separate subsections, which would make good use of the blank space.

1. The primary concern with this paper lies in its misalignment between the proposed goal of achieving a GFM and the actual experiments conducted. The current experiments are limited to datasets designed for graph classification, focusing on a single task type, all at the graph level. This approach does not substantiate the broader scope implied by a GFM. We recommend either narrowing the scope explicitly to a GFM designed specifically for graph classification tasks or enhancing the experimental framework by incorporating more diverse graph datasets for pre-training and downstream testing. Additionally, the paper does not provide any explanation as to why this approach could contribute to graph-level tasks. The only related mention occurs at line 362, where it is stated that GraphProp faces difficulties with larger graphs due to its use of computationally complex graph properties. However, this is a matter of methodological design rather than a theoretically justified reason.
2. The observations related to Figure 1 devote substantial space to discussing an intuitively evident point. Specifically, Figure 1 merely illustrates that when only graph structure is present without node information (a), the representations generated by the LLM exhibit low discriminative power across different graph datasets. Conversely, when only node information is present without graph structure (b), the LLM-generated representations display higher discriminative power across these datasets. This is straightforward to understand, as it’s clear from the input text provided to the LLM that the TSG lacks distinctive features, whereas the TAG demonstrates significant discriminative capability.
3. From a methodological perspective, Structural GFM merely trains a model capable of predicting various structural indicators of a graph and applies this model in downstream tasks. However, the critical question arises at best, what the model learns is to predict structural characteristics for any given graph. Why, then, can’t the structural characteristics of downstream graph data be used directly as $Z$, instead of relying on the output of a frozen Structural GFM model, to train the so-called Comprehensive GFM? Theoretically, this approach should perform at least as well as the current method.
4. Modeling graph structure and node features separately and training the structure-related component during pretraining is a common approach, as demonstrated in [1]. More importantly, the paper does not substantiate, either theoretically or empirically, why this particular design of Structural GFM is superior to other self-supervised models that learns the graph structure. Additionally, it remains unclear why the representation $Z$ output by Structural GFM can be directly added to the representation $E$ obtained from the LLM, given that they belong to different representational spaces. If $Z$ is intended to function as a positional encoding (PE), it should be compared with other existing PE methods.
5. As a GFM paper, the experiments are relatively limited, especially in terms of baseline comparisons. There are some papers that could serve as valuable baselines for comparison [2,3,4].
6. In practical application, when node features are available, the paper suggests using TAG. However, even with node features, it is possible to construct TSG as input for the LLM. Why is this option not utilized? Using TSG could potentially provide the LLM with richer information, leading to higher quality embeddings $E$.
7. In the Appendix, none of the results are highlighted to show the best or second-best performance, either through bolding or other visual indicators. This makes it challenging for readers to clearly understand the conclusions conveyed by the other experiments.

[1] GraphControl: Adding Conditional Control to Universal Graph Pre-trained Models for Graph Domain Transfer Learning

[2] THUDM/GraphAlign: GraphAlign: Pretraining One Graph Neural Network on Multiple Graphs via Feature Alignment (github.com)

[3] All in One and One for All: A Simple yet Effective Method towards Cross-domain Graph Pretraining

[4] AnyGraph