Towards Graph Foundation Models: Training on Knowledge Graphs Enables Transferability to General Graphs

We thank the reviewer Z3ck for the insightful and valuable comments. We sincerely hope that our rebuttal adequately addresses your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know if you have further concerns, and we will continue actively responding to your comments.  

1. W1&Q1. Can you compare the performance of the proposed SCR with other methods that also incorporate semantic features into the conditioned message passing framework to show the necessity of the proposed late-fusion method?  

We sincerely thank you for this comment! We first discuss the three methods [1,2,3] mentioned by the reviewer:  

1. The initializer $INIT^3()$ in Huang et. al. [1] incorporates the semantic feature of the query node $u$, but ignores the semantics of other nodes in the neighborhood. As a result, using $INIT^3()$ would not significantly drop the performance of ULTRA in Figure 2, because it still cannot exploit the full range of node semantics. This is also one of the motivations that we propose the semantic isolation issue.  

2. The second work [2] is not related to graph foundation models and does not utilize CMP-based encoding. It employs PLM-based textual embeddings as the input node feature of GNN, and alternates the training of GNN and PLM within a single dataset. This method cannot handle our zero-shot GFM settings, especially for non-textual semantic features in general graphs. Additionally, the performance drop reported in their Table 2 when directly using pre-trained PLM embeddings indicates the necessity of re-training a PLM for enhanced text representation. It is consistent with our findings in Table 2.  

3. The third work [3] is a contemporaneous work (published in May 2025) for KG foundational reasoning. While [3] achieves semantic injection, it focuses solely on text embeddings derived from a single semantic space (i.e., an LLM). Consequently, their pretraining approach cannot process zero-shot features originating from different semantic spaces. We note that their late fusion between query‑conditioned structural encoding and global structural semantic encoding is similar to our proposed strategy, indirectly confirming the feasibility and rationale of our design choice.

In summary, the first method cannot utilize complete node semantics, while the latter two methods, reliant on either an LLM's semantic space or graph-specific training, cannot effectively integrate multi-source zero-shot features. Consequently, none of these approaches provides a viable alternative to our method within Graph Foundation Model. We will incorporate the above discussion and analysis into the revised manuscript to strengthen our method.  

2. W1&Q1. When setting the dimension of the node features closer to the dimension of the semantic features and using $INIT^3$ fusion, would the trend in Figure 2 still hold?

In terms of the impact of the dimension gap between semantic features and node features, we conduct additional experiments with 50d Glove Features (with zero-padding), Clipped 64d Bert features, random features, and all-ones features. The results below indicate that using node features with closer dimensions still invokes the performance drop. Additionally, we test the variant using the semantic features of the query node in $INIT^3()$. There still exists a performance drop compared with the original ULTRA on most datasets.  

Because Bert-based embeddings have 768 dimensions, it is much larger than the dimension of the node features (64). Sorry that we cannot pretrain such a 768-d ULTRA due to limited computational resources in the rebuttal phase.  

3. W2&Q2. How does the proposed method handle the semantics of the edges? In the homogeneous graphs node classification, are there only two types of edges (i.e., original edges and label edges)? How does the proposed method generate meaningful relation representations from the relation graph for reasoning?  

We sincerely appreciate these comments and questions.  

We focus on node semantic features in this work and acknowledge the value of handling explicit edge features. It is straightforward to extend SCR to incorporate edge semantics by replacing the first CMP module with our proposed SCMP. Similar to our work, the contemporaneous work, SEMMA [4], employs an additional CMP-based semantic relation encoding process and the late-fusion strategy to handle edge features.  

We clarify that, in homogeneous node classification, besides original edges and label edges, there are semantically-similar edges involved in the relation graph and CMP-based encoding. Although there are a few relation types in the relation graph, it still follows strict topological rules. For example, label edges only link from nodes to their labels, while original edges connect among nodes. This creates meaningful asymmetric constraints, i.e. there are no “t-h” and “t-t” interactions from the label edge type to the original edge type. By learning how different relation interactions interact topologically across the graph, our model derives transferable reasoning capabilities that generalize beyond explicit relation semantics.  

In Section 5.5 (3), we conducted ablation studies to verify the necessity of the relation representations from the relation graph. As shown in the right panel of Figure 3, despite the presence of only a few relation types in node/graph-level graph structures, the observed performance decline highlights the essential role of relational information.    

4. W3&Q3. Why does the performance of the model drop after a few-shot fine-tuning? Is the model trained from scratch, or is it fine-tuned from a pre-trained model? Would fine-tuning from a pre-trained model improve the performance of the model?  

Thank you for your comment. We guess the mentioned drop performance is from SCR-5 and SCR-20%. We clarify that these two variants are not few-shot fine-tuning, but few-shot labeling. As explained in the caption of Table 2, SCR-20% uses 20% of the “node-label” edges from the training set, while SCR-5 includes five edges per class. Both variants have no fine-tuning process, but utilize different scopes of “label information” when inference on the pre-trained model.  

Additionally, we observed that fine-tuning would enhance the model performance on target tasks. For example, we have conducted additional experiments, which fine-tune on heterophilic graph-derived KGs. We discovered that SCR’s performance is improved on heterophilic node classification benchmarks, as shown in the table below.

[1] Huang, Xingyue, et al. "A theory of link prediction via relational weisfeiler-leman on knowledge graphs." Advances in Neural Information Processing Systems 36 (2023): 19714-19748.

[2] Combining Structure and Text: Learning Representations for Reasoning on Graphs, https://openreview.net/forum?id=hJ8OQAiTrl

[3] Hua, Yin, et al. "Beyond Completion: A Foundation Model for General Knowledge Graph Reasoning." arXiv preprint arXiv:2505.21926 (2025).

[4] Arun, Arvindh, et al. "SEMMA: A Semantic Aware Knowledge Graph Foundation Model." arXiv preprint arXiv:2505.20422 (2025).

Thank you for your thoughtful feedback. We appreciate the opportunity to clarify our position. Your interpretation aligns closely with our intended meaning: In the $INIT^3()$ method in Huang et.al., only the source node (i.e., query node) receives injected features in the initialization stage, while semantic features of neighboring nodes remain unutilized.

As shown in the table of our second response, there still exists a performance drop when using $INIT^3()$ compared with the original ULTRA on most datasets. Nevertheless, because $INIT^3()$ using semantic features can distinguish between nodes in most cases, the performance degradation is less severe than approaches that directly apply semantic features to all nodes.

We thank the reviewer zZR9 for the insightful and valuable comments. We sincerely hope that our rebuttal adequately addresses your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know if you have further concerns, and we will continue actively responding to your comments.  

1. W1. The link prediction performance on transductive datasets does not demonstrate a significant advantage compared to ULTRA.  

We sincerely thank you for this comment. We agree SCR doesn't surpass ULTRA on transductive link prediction, but it is not the primary objective of SCR.

Our key contribution is to validate the semantic and structural transferability of our pre-trained KG reasoning model to general graph tasks. Towards graph foundation models, achieving consistent performance across diverse tasks with zero-shot topological graph and semantic features is challenging and valuable.  

We believe incorporating more diverse training KGs could enhance reasoning capabilities, which we plan to rigorously explore to strengthen SCR's applicability in future work.

2. W2. Besides, in the introduction, a brief explanation of "semantic features" should precede the discussion of the "semantic isolation issue", ... I encourage the authors to revise this section to reflect this context more clearly.  

Thank you for this essential feedback. We will restructure the related paragraphs in the introduction section to:  

1. First define semantic features and their role in KG reasoning/general tasks;  
2. Then introduce the semantic isolation issue as a fundamental barrier to cross-task generalization;  
3. Explicitly connect both concepts to motivate SCR’s design.   This logical progression will clarify why resolving semantic isolation is critical for graph foundation models.

3. W2&Q1. The reliance on Semantic Conditional Message Passing (SCMP) may constrain the model’s reasoning ability on knowledge graphs that lack rich textual features. Can SCR be applied to knowledge graphs (KGs) that lack textual features?

Thank you for your insightful comment. We clarify that SCR is not restricted to graphs with textual features, as shown in the ablation study in Section 5.3. In Table 1, for SCR pretrained on textual features, replacing semantics with all one features (SCORE-One) only marginally degrades performance while still outperforming existing methods, indicating that semantic features do not significantly affect generalizability.

Besides, when evaluating on transductive KG datasets, SCR did not employ textual features (unavailable for some KGs) but instead ontology features derived from the KG’s inherent structure. SCR also achieved great performance.  

The key lies in SCR’s pre-training design: it systematically trains the model to handle diverse scenarios—from KGs with rich semantics (text/ontology) to those with no input features. SCR learns to reason over KGs with diverse semantics rather than depending on auxiliary textual data.  

4. Q2. Does SCR face limitations when scaling to large KGs? Specifically, is it capable of handling large-scale transductive datasets?  

We sincerely appreciate this comment. Regarding scalability, we analyzed the computational complexity of SCR in Appendix H, which exhibits comparable scalability and running time to ULTRA. We acknowledge that both methods face practical limitations when applied to KGs with millions or billions of triples. Specifically, the time cost of subgraph extraction and message passing becomes non-trivial compared to traditional embedding-based models. This limitation is inherent to subgraph-based inductive frameworks but does not preclude SCR’s applicability to typical large-scale KGs.  

We will prioritize this in future work. For large-scale KGs, recent acceleration techniques like TIGER [1] (enabling efficient subgraph extraction for inductive reasoning on Freebase) are critical. While SCR’s current implementation does not yet integrate these optimizations, its framework is compatible with such methods. We have discussed this point in Appendix J: Limitations.  

To further ease the concern, we recall the results on several large-scale transductive datasets in Table 7. The results demonstrate that SCR outperforms ULTRA, confirming its scalability and effective generalization.  

[1] TIGER: Training Inductive Graph Neural Network for Large-Scale Knowledge Graph Reasoning, Proceedings of the VLDB Endowment, 2024.

We thank the reviewer wV6S for the insightful and valuable comments. We sincerely hope that our rebuttal adequately addresses your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know if you have further concerns, and we will continue actively responding to your comments.  

1. W1: Graph-level and node-level tasks often require modeling local neighborhoods, community structures, or global graph semantics—factors not typically present in KG link prediction tasks. Without pretraining on such structures, it is unclear how the learned representations transfer effectively.  

Thank you for highlighting this concern. Here we transform these two tasks into KG formats by introducing two extra entities, “label□” and “super graph△”, and a new relation, “is attributed with”, to predict the labels for a node or a graph. We believe that many KG relations (e.g., "person → hasGender → gender") are functionally similar to classification tasks ("is_attributed_with"), as they map numerous nodes/graphs to shared attributes.

Although the pre‑training objective is link prediction, it learns the neighborhood connectivity and therefore captures local neighborhoods and community structures. Global semantics are retained via our non‑parametric semantic representation, which aggregates all original node features in the CMP process. Together, these components enable representations learned during KG pre‑training to transfer to downstream node‑ and graph‑level tasks that rely on local structures and semantic cues.  

2. W2: The comparison with ULTRA is potentially unfair. ULTRA is trained solely on KGs and not designed for general graph tasks. In contrast, the proposed method is designed a broader range, giving it a task diversity advantage.  

We appreciate the reviewer’s valid concern regarding comparison fairness. We clarify that SCR and ULTRA use the same pretraining knowledge graphs (FB15k-237, WN18RR, CoDExMedium). ULTRA can handle general graphs by utilizing the proposed Unified Reasoning Format, that is why we obtain its performance on general graph tasks.  

The core difference lies in SCR’s complementary use of semantic features (e.g., text descriptions, node features) beyond raw graph structure. The advantages of SCR stem from the semantic-augmented graph encoding in SCMP and pretraining with multi-source features.  

3. W3: The model performs better on homophilic graphs than heterophilic ones, suggesting it relies heavily on feature similarity or local neighborhood consistency. This raises concerns about its robustness across structural regimes, which are common in real-world graphs.

We sincerely appreciate the reviewer’s insightful observation. We agree that SCR currently exhibits a degree of homophily bias, which we believe primarily stems from the CMP-based backbone architecture that emphasizes local relational connectivity.

Our main objective in this work is to validate the semantic and structural transferability of our pre-trained KG reasoning model to general graph tasks. To this end, we adopt the standard CMP-based backbone, whose generalization ability has been well established in the KG reasoning domain.

To improve robustness on heterophilic graphs, several promising strategies can be explored, such as pre-training or fine-tuning on heterophilic structures, or replacing standard aggregation functions with operators more suitable for heterophilic settings. In our preliminary experiments, we introduced a fine-tuning stage on heterophilic benchmarks, and the results demonstrate clear improvements in SCR’s performance:

These findings suggest that SCR can indeed be enhanced through additional training on heterophilic graphs. We believe that incorporating a broader set of training KGs—especially those exhibiting diverse structural properties—will further improve generalization and reasoning capabilities across varying graph regimes.

In summary, our work aims to improve transferability across graph tasks, and the observed performance gains of SCR over ULTRA validate this contribution. We will include a more in-depth discussion on the challenges of heterophily and potential mitigation strategies in the revised manuscript, and we plan to explore this direction further in future work.

4. W4: Over-smoothing and over-squashing are known issues in message-passing-based models, especially in deep architectures or long-range dependency tasks. The paper does not provide any discussion or evaluation on these issues. Benchmarks like LRGB or synthetic long-range tasks would help assess whether the model is scalable in such settings.  

We thank the reviewer for highlighting the critical issues of over-smoothing and over-squashing in message-passing models. While these are well-known challenges, our current focus is to investigate the semantic and structural transferability of KG pre-training to diverse graph tasks—a core goal in the emerging Graph Foundation Model (GFM) paradigm.

We agree that a discussion of these challenges is valuable. Over-smoothing typically occurs in deep architectures, where repeated message passing leads to uniform node embeddings. In our model, however, message propagation occurs over relatively shallow subgraphs (3-6 layers), which mitigates this risk. Furthermore, our message passing is source-conditioned: only the source node $e_q$ is initialized with a non-zero embedding, and information flows outward. Thus, each node’s embedding is a view from $e_q$, not a globally shared representation, preserving diversity.

Over-squashing, caused by bottlenecks in aggregating long-range messages into fixed-size vectors, may arise due to dense multi-hop paths. To address this, we employ the expressive PNA aggregator and set the hidden dimension to 64, balancing representational capacity with computational efficiency, while also accommodating both BERT-based textual features and common graph-domain features.

It is worth noting that unlike conventional GNNs, Graph Foundation Models are still in their early stages, and many associated challenges remain open. Our work takes a step forward by focusing on cross-task and cross-graph transferability, which we believe is largely orthogonal to the over-smoothing, over-squashing, and long-range issues. Nonetheless, we agree that evaluating performance on deep or extreme-range benchmarks would further validate scalability. We will include a dedicated discussion on these challenges and possible mitigation strategies in the revised manuscript, and we consider this an important direction for future work.

5. Q1: Section "CMP-based Backbone Model" is crucial in understanding the paper, but it is not carefully written. Could the author elaborate more, such as what is "conditioned on the query" or the initialization process of $h_v$?  

Thank you for pointing this out. We will revise the manuscript accordingly, which first gives the high‑level intuition and then provides the formal update rules.

In CMP, the learned node embeddings are conditioned on a query $(e_q, r_q)$, where $e_q$ is the source node and $r_q$ is the specific relation embedding. At initialization, only the source node carries information: its hidden state $h_q$ is set to a non‑zero vector determined by $r_q$, while all other nodes are zeroed out. During message passing, this signal propagates outward, and each target node $e_v$ ultimately learns an embedding $h_v$ that reflects how it is viewed from the perspective of $(e_q, r_q)$.

Thank you for the constructive suggestion! We fully agree that evaluating on long-range graph datasets will further clarify our method’s capability.

Due to the time and resource limit, we selected a subset of PascalVOC-SP (685K nodes, 5M edges) to evaluate the node classification task. While the current LRGB benchmark assumes fully-supervised training on very large graphs, our proposed SCR was designed for zero-shot generalization. SCR only saw the label information during inference but no any fine-tuning on them. SCR delivers a substantial gain over ULTRA, but under full supervision, it still trails GCN and GraphTransformer. We believe that large-scale training enhances the performance of GCN and GraphTransformer.

To mitigate the effect of training, we use a few-shot setting. GCN and GraphTransformer are trained on 20% samples, while SCR remains strictly zero-shot, merely seeing those labels at inference. Under this setting, SCR matches GraphTransformer and outperforms GCN, demonstrating that it retains long-range ability even without task-specific training.

Sorry again for no running the entire LRGB benchmark (hundreds of millions of labels) during the rebuttal phase. Nevertheless, our method is compatible with the optimization strategies outlined in Response 4 to Reviewer zZR9 to solve the scalability challenge. We will include these new results, along with a dedicated subsection discussing long-range ability and scalability challenges, in the revised manuscript.

We thank the reviewer cmwp for the insightful and valuable comments. We sincerely hope that our rebuttal adequately addresses your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know if you have further concerns, and we will continue actively responding to your comments.  

1. W1: The main figure lacks clarity and does not effectively convey the model’s workflow.

We sincerely thank you for this comment. Because external figure links are not permitted during the rebuttal phase, we describe the points we plan to revise. First of all, we will redraw all the arrows and reorganize the two bottom subfigures, so that the workflows for pre‑training on KGs and reasoning on general graphs are shown clearly. Besides, we also modify the caption to clarify the role of each component. The updated figure will be added in the revised version.

2. W2: The main innovation appears to be a practical and effective integration of node semantic information based on the ULTRA framework. The introduction of virtual nodes to convert various tasks into reasoning tasks on graphs primarily serves this integration, which to some extent reduces the novelty of this work.  

Thank you for raising this point. We agree that the use of virtual nodes is a common design choice, but it remains essential to unify different graph tasks. The novelty of our work is to generalize the inductive KG reasoning method within the ULTRA framework to handle diverse tasks on general graphs, providing a practical recipe for building graph foundation models.

3. W3: Additional experiments isolating the effect of semantic features on graph classification would strengthen the claims. Additionally, did SCR (3g), ULTRA (3g), and ProLINK(3g) consistently use the same KGs?  

We sincerely appreciate these valuable comments.

In Section 5.5, we conducted ablation studies to verify the impact of semantic features on node/graph classification. As shown in the right figure of Figure 3, the reduced performance of “w/o NPSR” emphasizes its important role in leveraging semantic diversity. Meanwhile, the Semantic-Injected Entity Initialization and Relation Representations also play essential roles. Additionally, in Table 3, two graph classification datasets we evaluated (IMDB-BINARY, COLLAB) are featureless, which indicates that the performance gain is not only derived from semantic features.  

The three compared methods (SCR, ULTRA, ProLINK) use identical underlying topological graphs. The critical distinction is that only SCR incorporates semantic features to construct additional semantically-similar edges, while ULTRA and ProLINK rely solely on raw graph structures.
 

4. W4: The methodology section could be better organized. An initial high-level overview of the entire approach is needed.  

We sincerely appreciate this constructive suggestion. To improve clarity, we will relocate the workflow description currently in Section 4.3 to the beginning of Section 4, serving a high-level overview of the entire approach accompanied by Figure 1. This will provide readers with a clear roadmap before delving into implementation details.