Towards Effective Federated Graph Foundation Model via Mitigating Knowledge Entanglement

We sincerely thank you for your constructive comments, and initially address all of your concerns below:

Question 1: What is the difference between knowledge entanglement in FedGFM and vocabulary collapse in VQ-VAE?

We appreciate the opportunity to clarify the distinction between the two challenges.

• Knowledge Entanglement is a unique challenge in FedGFM caused by the lack of centralized optimization over heterogeneous, domain-specific data. This leads to different semantics being mapped to the same token positions, resulting in conflicting updates during server-side aggregation. For example, token ID 1 may represent financial concepts on one client and biomedical knowledge on another. These conflicts degrade the quality of the aggregated global model and cause domain confusion downstream.

• Vocabulary Collapse is a common issue in VQ-VAE, where only a few codebook tokens dominate usage while others remain inactive. This self-reinforcing dynamic occurs because tokens slightly closer to inputs are selected and updated more often, further increasing their dominance. Consequently, token utilization decreases and representational diversity suffers.

Question 2: Why is the federation mechanism not applied during the fine-tuning stage?

Thank you for your comment. FedGFM consists of two stages. During pre-training, clients act as data holders, collaboratively training a unified and generalizable graph foundation model via federated communication over distributed, domain-specific datasets. In contrast, during fine-tuning, clients serve as model users or task adapters, working independently without federation. Notably, the fine-tuning data often differs from the pre-training data, reflecting practical deployment scenarios where distributed coordination is unnecessary or impractical. Based on the above considerations, federated communication is employed during the pre-training stage, but not during the fine-tuning stage.

Question 3: Explain the fairness of the comparison between FGL baselines and FedGFM+

Thank you for the question. We appreciate the opportunity to clarify the fairness of our comparison.

All methods (including FedGFM+ and the FGL baselines) are evaluated under the same federated protocol, with consistent feature pre-processing, identical data splits, and shared optimization settings. The only difference lies in the modeling paradigm: FedGFM+ adopts a federated pre-training and fine-tuning framework, while FGL baselines follow a task-specific federated training approach. This paradigm shift allows FedGFM+ to enable generalized collaboration across domains and tasks (see Lines 64–78), whereas FGL baselines are inherently limited to single-task, task-bound collaboration (see Lines 38–50).

In summary, the performance superiority of FedGFM+ arises from its modeling innovations, not from any unfair advantage in experimental setup. We therefore consider the comparison between FedGFM+ and FGL baselines to be fair and meaningful.

Question 4: Clarify the meaning of Figure 2 in more detail.

Figure 2 is designed to illustrate the core challenge of the FedGFM paradigm, named knowledge entanglement.

Specifically, we consider three datasets from distinct domains and tasks: Cora (citation network, node classification), WN18RR (knowledge graph, edge classification), and HIV (molecular graph, graph classification). As shown in Figure 2(a), these datasets exhibit significantly different structural patterns, while Figure 2(b) ('Raw Feat.') shows notable feature disparities across domains.

We then compare GFT and its naive federated variant GFT*:

• GFT, trained centrally across all datasets (jointly optimized);

• GFT*, trained in a decentralized manner following the FedGFM setting.

As shown in Figure 2(b), the embeddings produced by GFT ('GFT Emb.') preserve clear domain-specific separability, while those produced by GFT* ('GFT* Emb.') collapse into indistinguishable representations.

This result highlights a fundamental limitation in the FedGFM setting: knowledge entanglement, where multi-domain knowledge is encoded into indistinguishable representations, thereby limiting downstream adaptation.

Thank you for your follow-up. We are glad our previous response addressed most of your concerns. Below we provide further clarifications on the remaining points.

Question 2: Clarify how this work differs from these federated fine-tuning methods. We appreciate your reference to existing work on federated fine-tuning. Here we clarify how the FedGFM paradigm differs from these approaches, and why this motivates our choice to execute the fine-tuning phase in isolation rather than in a federated manner.

In existing federated fine-tuning methods for foundation models, the primary goal is to adapt the model to multiple local datasets held by clients; in such settings, the fine-tuning data directly corresponds to the clients’ local data.

By contrast, in FedGFM, multiple decentralized datasets from clients are used only in the pre-training stage to learn a GFM. In the fine-tuning stage, the learned GFM is adapted to a target graph that may originate outside of any participating clients. This naturally allows (and in some scenarios necessitates) performing fine-tuning in isolation rather than through a federated process.

Question 3: Explain the reason for each FGL baseline does not apply to certain tasks.

Thank you for your suggestion. Below, we provide a more detailed explanation of why existing federated graph learning methods are not directly applicable to our setting:

• FedSage+, Fed-PUB: Designed for subgraph-level federated learning; their model architectures and optimization objectives are not suitable for graph classification datasets.

• FedGTA, FedTAD, FGSSL, FGGP: Developed for subgraph-level federated learning, but their algorithms rely on node label inputs, making them applicable only to node classification tasks—not to edge classification datasets (which lack node labels).

• GCFL+, FedStar: Designed for graph-level federated learning; their model architectures and optimization objectives are not suitable for subgraph-level federated learning with node or edge classification tasks.

We hope this resolves the remaining ambiguities. Thank you again for your thoughtful feedback and for recognizing our contributions.

We sincerely thank you for your comments, and hope to address your concerns as follows:

Question 1: The motivation of the FedGFM paradigm (combining FGL and GFM) is not convincing.

We respectfully highlight that FedGFM is not a superficial field combination, but a practical paradigm that leverages the complementary strengths of FGL and GFM.

• The advantage of FGL is that it can achieve cross-silo data collaboration, computing and storage resource collaboration under the scenario of privacy protection. However, its disadvantage is that the scope of collaboration is extremely limited (across small number of fields or even different shards of a single dataset) due to the data heterogeneity (including features, space, topology) and task heterogeneity (bound to specific model architecture, algorithm process, and optimization goals).

• On the contrary, the advantage of GFM is that the language model encoding unifies the data format, while the pre-training mechanism is task-format-agnostic. The disadvantage is that the actual training of GFM faces the difficulty of centrally collecting multi-domain datasets (graphs from different domains are collected by different institutions and are not allowed to be made centralized and public) and the opportunity cost of wasting multi-party cross-silo computing and storage resources.

Why centralized GFMs are not enough to fully meet real-world needs: Existing centralized GFM studies are typically trained on datasets from a small range of domains (e.g., 4 domains for OFA[1] and GFT[2]), which is significantly fewer than those used in vision or NLP foundation models. However, building a generalized graph foundation model likely requires access to graphs from a much broader range of domains. In reality, such data (e.g., financial networks, disease graphs) are often privacy-sensitive and siloed across institutions, where legal, ethical, and commercial constraints prohibit direct data sharing. As a result, aggregating these graphs for centralized GFM training is not only impractical but often infeasible.

FedGFM addresses this limitation by aligning GFM’s task-agnostic modeling capabilities with FGL’s decentralized training infrastructure, enabling scalable foundation model pre-training across siloed, heterogeneous, and sensitive domains, which is the core motivation behind FedGFM.

Question 2: The various modules of the proposed framework FedGFM+ lack technical novelty.

We respectfully argue that the core novelty of this work lies in paradigm innovation: to the best of our knowledge, this is the first work to identify and systematically characterize the complementary relationship between FGL and GFM, leading to the proposal of a new paradigm, FedGFM. We further conduct an in-depth empirical analysis to uncover a paradigm-specific challenge named knowledge entanglement.

Within this new paradigm, the novelty of technical design should be evaluated in terms of how our method addresses the above challenge. While certain components like prototypes and personalized prompts have been widely used in various machine learning research, our design motivations and functionalities are uniquely aligned with the needs of FedGFM, rather than being direct adaptations from any FGL, GFM, or VQ-VAE studies. Below, we provide a detailed clarification of the novel aspects of each module:

• Anchor-Based Domain-Aware Initialization (AncDAI): In FedGFM, where centralized optimization of multi-domain knowledge is infeasible, clients from different domains may encode distinct semantics into the same codebook tokens. When these locally updated codebooks are aggregated on the server, such semantic conflicts can compromise the global representation quality. To mitigate this, AncDAI leverages each client’s local domain prototype as a semantic anchor to initialize the global codebook, encouraging early separation of domain-specific semantics and reducing conflict during aggregation. This design is unique to FedGFM paradigm, as existing GFM or VQ-VAE models operate in centralized settings, where token semantics can be aligned through joint optimization, eliminating the need for such initialization strategies.

• Adaptive Domain-Sensitive Prompt Pool (AdaDPP): To mitigate knowledge entanglement in FedGFM from the local perspective, AdaDPP learns domain-specific prompts during pre-training and adaptively injects them during fine-tuning. These prompts provide domain-aware guidance that modulates the shared encoder’s behavior, enabling localized adaptation while reducing cross-domain interference. Unlike prior GFM or FGL methods that rely on a shared encoder without domain-specific adaptation, AdaDPP introduces explicit domain-aware modulation to preserve semantic boundaries across domains.

Question 3: Need more explanation about the baseline selection strategy and the reason behind it.

We sincerely apologize for any possible oversights. Here we provide further explanations about the reasons for the setting and selection of each category baseline:

• Isolated Supervised Learning: These models are trained independently on each client without collaboration, serving as the performance lower bound under non-federated and non-pre-training settings. They help assess the benefit of FedGFM over traditional per-graph training. Model choices are also aligned with task types: GCN, GAT, and GraphSAGE for node/edge classification, and GIN for graph classification.

• FL/FGL Methods: These approaches support only limited collaboration across domains and are often tailored to specific downstream tasks, limiting their generality. In contrast, FedGFM enables broad, scalable collaboration across heterogeneous domains and tasks. The performance comparison with FL/FGL baselines further highlights the effectiveness of generalized, task-agnostic pretraining over narrow, task-specific federated learning.

• Federated Variants of Centralized GFM: These methods adapt centralized GFM architectures to federated settings by performing multi-round parameter aggregation across clients. While they enable federated pretraining, they do not account for the knowledge entanglement challenge arising from cross-domain heterogeneity. In contrast, FedGFM+ is explicitly designed to mitigate such entanglement, and comparisons with these baselines highlight its superior ability to preserve domain-specific semantics and improve performance under the FedGFM paradigm.

Finally, we kindly remind you that detailed descriptions of each baseline catagories and specific methods can be found in Appendix C.3. We promise you that we will add these supplementary content to the appendix during the revision phase.

References

[1] Liu, H., Feng, J., Kong, L., Liang, N., Tao, D., Chen, Y., & Zhang, M. One For All: Towards Training One Graph Model For All Classification Tasks. In The Twelfth International Conference on Learning Representations.

[2] Wang, Z., Zhang, Z., Chawla, N., Zhang, C., & Ye, Y. (2024). Gft: Graph foundation model with transferable tree vocabulary. Advances in Neural Information Processing Systems, 37, 107403-107443.

Thank you for your follow-up and the score adjustment.. We agree that the privacy implications of prototype sharing merit deeper theoretical analysis, and will evaluate these privacy risks in future work.

We sincerely thank you for your comments, and appreciate that you acknowledged the strengths of our work. Here we initially address all concerns below:

Question 1: Theorem B.1 and B.2 lack quantitative estimation or empirical evidence.

Thank you for your insightful feedback. We hope to address your concerns.

For Theorem B.1 (Domain Prototype Distinguishability), prior work has shown that even randomly initialized encoders can produce distinguishable embeddings for multi-domain data, especially in the text domain [1]. This supports our assumption that distinguishability can emerge from structural biases and feature heterogeneity.

For Theorem B.2 (Semantic Separability of the AncDAI-Initialized Codebook), we have added empirical evidence demonstrating that FedGFM+ encodes multi-domain knowledge into semantically separable codebook representations. Please refer to our response to Question 2 from reviewer cFwZ.

Question 2: Whether the theorem B.2 hold under other similarity metrics?

Thank you for your comments. Theorem B.2 remains valid for other similarity metrics (e.g., Euclidean distance), as long as the codebook’s query strategy is adapted accordingly. Since the codebook selects discrete tokens based on encoder outputs, the similarity metric used in the query must match the one assumed in the theorem. In our case, Theorem B.2 is based on cosine similarity, which is also the metric used in our codebook querying.

Question 3: Lack of discussion about the potential privacy risks associated with shared domain prototypes and prompts.

We sincerely thank you for your comments and agree that a deeper privacy risk analysis of the shared domain prototypes and prompts in FedGFM+ would provide stronger safeguards. However, we respectfully emphasize that we did not upload any raw local data; instead, it was encoded using a randomly initialized model, which helps reduce the risk of privacy leakage to some extent. It is also important to note that privacy preservation is not the primary focus of this work. Our main contribution lies in defining a new paradigm named federated graph foundation model (FedGFM). In future revisions, we plan to include a theoretical analysis and explore the use of noise injection on the uploaded domain prototypes to enhance privacy protection.

Question 4: Eq. (7) results in same feature dimension for all nodes, which may limit the applicability of the method to heterogeneous graphs.

Thank you for your comments. Equation 7 illustrates the feature augmentation process when applying the pre-trained GFM and personalized prompt pool to a specific target graph during fine-tuning. We would like to clarify that the statement "all nodes must have the same feature dimension" does not imply that our method is unsuitable for heterogeneous graphs. On the contrary, our approach is applicable to all textual attribute graphs with textual descriptions for nodes and edges. We ensure a consistent feature dimension by encoding all textual descriptions into unified vector representations using a shared pre-trained language model (e.g., Sentence-BERT, as used in this paper).

Question 5: Need more justification and analysis of computation and communication costs about would prompts and projections.

Thank you for your comment. Our prompts are very lightweight and contain far fewer parameters than the backbone network. Although each prompt shares the same feature dimension as the node features, the total number of prompt parameters remains minimal, resulting in negligible additional computation and communication overhead in the federated setting. Specifically, in our experiments, for each client, we use only 3 learnable prompts (shape: 3 × 768) along with a projection matrix (shape: 768 × 3), which is extremely lightweight.

Question 6: How is the model trained to ensure well-separated prototypes across domains?

Thank you for your comments. We respectfully clarify that FedGFM+ relies on the AncDAI module to yield distinguishable domain prototypes for graphs from different domains. As outlined in Section 4.1 and Theorem B.1, prototype computation is independent of a pre-trained encoder; it only presupposes that every client employs an identically randomly initialized encoder.

Question 7: provide a pseudo-code or algorithm sketch for FedGFM+.

We sincerely appreciate your constructive comments. Due to word limit, please see our rebuttal to Question 1 from reviewer cFwZ.

Question 8: How does the model perform on domains with limited or imbalanced data?

Thank you for your constructive feedback. We respectfully remind you that the original manuscript briefly discussed the performance of FedGFM+ on limited data and domain imbalanced data. Specifically:

• Limited Data: In Appendix D of the original manuscript, we evaluated the performance of FedGFM+ in the few-shot setting. As observed, FedGFM+ consistently outperforms naive federated adaptations of centralized GFM training strategies across all evaluated settings
• Imbalanced Data: We’d like to clarify that our original experiments were conducted on data with imbalanced domains (see Appendix C.1 Table 4 and C.2 Table 5). The graph classification dataset involves significantly more nodes than the node classification dataset, based on local codebook access counts. To address this, we use a fixed batch size and number of batches with random sampling during pre-training, ensuring each client sees similar improvement per training round. Our results show that FedGFM+ handles domain imbalance effectively. We will include a clearer explanation of this training strategy in the revised version.

Question 9: What happens to performance if the data is randomly partitioned or if the number of clients increases?

We sincerely appreciate your constructive comments. Due to word limit, please see our rebuttal to Question 3 from reviewer cFwZ.

We sincerely thank you for your constructive comments, and we will initially address all of your concerns below:

Question 1: Lack of formal algorithmic pseudo code for FedGFM+

We fully agree with this constructive feedback and apologize for the oversight. Here, we present the pseudo code for the federated pre-training phase and isolated fine-tuning phase of FedGFM+ in Algorithms 1 and 2, respectively. We are committed to revising this content to provide a more rigorous and formal exposition of the FedGFM+ approach in the revision stage.

Algorithm 1 - Federated Pre-Training Phase of FedGFM+

Input: Number of clients $K$, each client's local graph $G^k$, initialized global gVQ-VAE parameters $\Theta^{g}$, each client's local learnable prompt parameter $\Phi^k$ and projection vector $w^k$, number of communication rounds $T$, number of local epochs $E$, noise scaling factor $\sigma$.

Output: learned global GFM $\Theta^{g}$, domain-aware prompt pool $\rho$ and projection pool $w$

for each client $k$ in $1,...,K$ do

------------ Initialize local gVQ-VAE with global gVQ-VAE, $\Theta^k \leftarrow \Theta^g$.

------------ compute domain prototype $p^k$ via Eqs. (3), (4)

------------ upload domain prototype to central server

end

compute perturbed embeddings and initialize global codebook via Eq. (5)

for each round $t$ in $1,...,T$ do

------------ for each client $k$ in $1,...,K$ do

------------------------- replace local gVQ-VAE with global gVQ-VAE, $\Theta^k_t \leftarrow \Theta^g_t$

------------------------- for local epoch $e$ in $1,...,E$ do

------------------------------------- local graph feature prompting via Eq. (6)

------------------------------------- local pre-training to update local gVQ-VAE $\Theta^k_t$ and learnable prompt $\Phi^k$, $w^k$ via Eq. (2)

------------------------- end

------------------------- upload gVQ-VAE parameters $\Theta^k_t$ to central server

------------ end

------------ aggregate local gVQ-VAE parameters to update global gVQ-VAE, $\Theta^g_{t+1} \leftarrow \frac{N_k}{N}\sum_{k=1}^{K}\Theta^k_t$

end

Algorithm 2 - Isolated Fine-Tuning Phase of FedGFM+

Input: Target graph $G$, learned global GFM $\Theta^{g}$, domain-aware prompt pool $\rho$ and projection pool $w$, number of fine-tuning epochs $E^t$, task head $W$.

Output: fine-tuned global GFM $\Theta^{g}$ and task head $W$.

for each epoch $e$ in $1,...,E^t$ do

------------ target graph feature prompting via Eq. (7)

------------ downstream task-specific optimization to fine-tune global GFM $\Theta^{g}$ and $W$.

end

Question 2: Missing encoding results to demonstrate how FedGFM+ alleviates the knowledge entanglement problem.

Thank you for your valuable feedback. In Section 3, we show the similarity of initial node features and GFT/GFT* embeddings for Cora, WN18RR, and HIV, which highlights the knowledge entanglement issue in FedGFM—where representations from different domains become indistinguishable. As suggested, we further report FedGFM+ embeddings on these datasets. The results are summarized in Table. 1:

Table. 1: Comparison of the distinguishability of FedGFM+ and baseline embeddings on cross-domain datasets.

As observed, FedGFM+ produces significantly lower embedding similarity across the three domains compared to GFT and GFT*, indicating that its AncDAI and AdaDPP modules effectively help encode multi-domain semantics into distinguishable representations, thereby mitigating the knowledge entanglement issue.

Question 3: How does FedGFM+ perform under increased number of clients?

We appreciate the insightful suggestion. Following your advice, we evaluated FedGFM+ and 5 representative baselines, including FedAvg, Fed-PUB, GCFL+, OFA*, and GFT* on the Cora, PubMed, WikiCS, OGB-arxiv, FB15K237, WN18RR, HIV and PCBA datasets, each evaluated under both 3-client and 5-client partitions. The results are summarized in Table. 2.

Table. 2: Performance comparison between FedGFM+ and baselines under 5 client partitions.

As observed, FedGFM+ consistently outperforms all baselines. Compared to the 3-client setting in Section 3, all methods show performance drops under 5-client partitioning, which is a common phenomenon in federated learning when data becomes more sparsely distributed (increased client heterogeneity).