One Model for One Graph: A New Perspective for Pretraining with Cross-domain Graphs

Q1 What is the means of t in Equation (3)? How do you ensure that each expert has different parameters? Based on Equation (3), it seems that each expert has the same input and parameters, leading to identical outputs. What, then, is the purpose of having multiple experts?

A1. Thank you for your question. In Equation (3), $t$ indicates the number of training nodes in a batch. Equation (3) indicates the loss calculation to train a single expert for one graph. Each expert has the same architecture but is trained on different datasets. Thus, their inputs and outputs are different. In other words, for one graph, we pretrained one distinct model.

Q2. In line 241, the mask matrix seems to be replaced by an MLP. Why are the node embeddings transformed by the MLP considered to be negative embeddings?

A2. Thank you for your question. For the MLP-generated mask matrix $a_i$, we tend to let it learn to mask the the domain-irrelevant features through the two loss terms $dis(\tilde{f},f_{center})$ and $\frac{1}{dis(\mathbf{o_i,f_{center}})}$ in Equation (4), where $\tilde{f}$ is embedding of original features masked by $a_i$ in the output space, $o_i$ is embedding of $a_i$ in the output space, and $f_{center}$ can be viewed as the representative feature of the training domain. With the loss terms decreasing, the distance between the mask matrix $o_i$ and $f_{center}$ will become larger but $\tilde{f}$ masked by $o_i$ will become closer to the $f_{center}$. Hence, $a_i$ are supposed to learn to mask the domain-irrelevant features of the input vector.

Q3 In Equation (4), is f_center the average output of a single graph across multiple experts, or the average output of multiple source graphs through their respective experts? How does this function as an anchor point for training the gate?

A3 Thank you for your question. $f_{center}$ is the average out of one source graph through its own expert. It is a centroid point for the feature distribution of the source graph. Thus, we could view the feature vectors close to $f_{center}$ as in the similar domains of the source graph, while vectors distant from $f_{center}$ in different domains. In this way, $f_{center}$ serves as an anchor point for training the gate as we described in the answer for the last question.

Q4. What are the parameters of the Gate in Equation (5)? Why is the Gate used before the Expert?

A4. Thank you for your question. The Gate and Expert in Equation (5) are pretrained on the $p$th source dataset following the steps in Section 3.1. The gate is used before the Expert because Equation (5) is used to calculate the relevance score with the help of graph.

Q5. According to the inference process, an unseen graph activates the top k experts based on the highest correlation between each expert's output and the output average. How do you ensure that the pre-trained graph domain encompasses a sufficiently heterogeneous feature space to handle all potentially unseen graph domains?

A5. Thank you for your questions. We agree that there is possibility that the framework will have difficulty if encountering datasets from unseen domains. Actually, the out-of-distribution problem is a general problem for pretraining models, even for LLMs [1,2]. To deal with the data heterogeneity problem, researchers usually add data from more sources to pretraining datasets. In our case, we try to include the typical datasets from different graph domains to enlarge the scope of the model bank. In this way, the framework could find similar models for most of the common downstream tasks.

Reference:

[1] Unsupervised Out-of-Domain Detection via Pre-trained Transformers; Xu et al. [2] Multi-Level Knowledge Distillation for Out-of-Distribution Detection in Text; Wu et al.

Q6. How is the total number of experts determined, especially when the number of graph domains during pre-training and testing is uncertain?

A6. Thank you for your question. For the pretraining datasets, we select representative datasets from common domains to construct the pretrained model bank. Hence, the number of experts in the model bank is the same as the number of pretraining datasets. For the number of experts in downstream tasks, we choose top-k relevant experts from the whole model bank. Specifically, we conduct experiments as shown in Figure 6, which reveals that 2 or 3 experts could be an ideal choice that boosts the useful knowledge to transfer and mitigate the negative transfer.

Q1. The primary motivation for adopting the “one model for one graph” approach is to alleviate the negative transfer limitations observed in the “one model for all graphs” method. It would be beneficial to provide comparisons and discussions on how this method differs from prior approaches that aim to reduce negative transfer through better pretraining data selection as [1].

A1. Thank you for your question. The data selection approach such as [1] implements data selection in the pre-training stage. After the pre-training, the model is fixed and applied to all the downstream tasks. This method will exclude knowledge from the pretraining data and would cause generalization difficulty on downstream datasets. Compared to this approach, our method is more flexible. We do not eliminate any knowledge from the pretraining data but store all in the bank. During inference, we adaptively choose the most relevant models (knowledge) for a downstream dataset (or task), which encourages the positive knowledge transfer while mitigating the negative transfer more effectively.

Q2. It is recommended to identify which models, pretrained on specific graphs, are selected as the best match for various test graphs, with explanations for these selections. Additionally, I’m curious about whether pretraining data from different domains can contribute effectively or if only similar/same-domain data is more beneficial and would strengthen the analysis. A case study is recommended to evaluate whether the proposed gating strategy actually mitigates issues stemming from conflicts across pre-training data from diverse domains.

A2. Thank you for your question. We have provided a case study in Section 4.5. We consider the 9 downstream datasets for zero-shot node classification. Specifically, we select 10 random samples from each dataset and visualize the average relevance score given by different gate functions. From the figure in Section 4.5, we find that in most of the situations, the gates will choose expert models trained in similar domains, which indicate that datsets from similar domain will contribute more in general. But in several cases, the gate will choose models from different domains. For example, Sports dataset of e-commerce domain have the highest relevance scores to the Wikics dataset in academic datasets, which indicates a sign of cross-domain transfer in certain situation.

Q3. It would be valuable to explore whether this pipeline could be adapted to other self-supervised learning approaches for pretraining graph-specific experts. and additional ablation studies are expected.

A3. Thank you for your suggestion. In the table below we attach the results attained by switching self-supervised approaches to GraphMAE. As a comparison, we find that both the original GRACE method and GraphMAE methods could attain similar performance. We will add the results in the revision.

Q4. OMOG’s design requires a separate model for each dataset and can result in a large model bank when many datasets are involved. Will it lead to high storage costs and maintenance overhead, especially in resource-constrained environments? A complexity analysis would also be helpful to understand OMOG’s computational feasibility at scale.

A4. Thank you for your question. Our models are light-weighted. The storage cost of one pair of expert and gate models is 579KB. Considering the storage capability of current devices, the cost is relatively low and it is feasible to maintain the model bank.

For a single expert model, the training time complexity is $O(d^2+K^2d)$, where $K$ is the number of hops aggregated for a target node and $d$ is the dimension of the feature vector. For a single gate model, the training time complexity is also $O(d^2+K^2d)$. Thus, the pretraining time complexity on a single dataset is just $O(d^2+K^2d)$. This manifests the feasibility of scaling up our frameworks. Moreover, since our framework allows us to train multiple experts and gates on different datasets parallely, it actually would gain efficiency advantages compared to the previous cross-domain pretraining methods.

Q1. The paper does not clearly explain the differences from other MOE-based methods, such as GraphAlign and AnyGraph. The approach seems very similar to these methods, leaving it unclear what specific advantages OMOG has over them and why it achieves improved performance.

A1. Thank you for the question. Previous MOE methods have the following characteristics: (1) Have a fixed number of experts with a model. (2) Pretrain all the experts on different datasets together. (3) Apply the one pretrained model for all downstream datasets. Compared to the previous datasets, our new method has following advantages: (1) Train different expert models on different datasets parallelly, avoiding the potential conflicts from different data domains during pretraining. (2) Flexibly choose suitable experts for different downstream tasks, which boosts the positive knowledge transfer while mitigating the negative transfer. Furthermore, as empirical evidence shown in the table below, our proposed method outperforms the two methods on the node classification tasks when the datasets are numerous and from different domains.

Q2. A core idea of OMOG is that each pre-training dataset requires a dedicated expert. This approach poses challenges for scalability: as the volume of pre-training data increases, the model grows linearly with the data, which is detrimental to pre-training efficiency.

A2. Thank you for your question. We would like to point out that to construct the pre-trained model bank, it is not required to train a large model on various datasets. Instead, our approach only trains light-weight expert and gate models on each dataset, respectively. Hence, the pre-training process on multiple datasets can run in parallel, which actually provides our method efficiency advantages over other cross-domain pre-training approaches.

Q3. Why is the expert model specifically a Transformer? How would the performance change if other models, such as GNN, Graph Transformer, or MLP, were used instead? Additionally, prior to entering the experts, the features and structure are fused through SGC. Why couldn’t this fusion step be incorporated within the experts themselves? After all, different graphs may require varying levels of neighbor aggregation.

A3. Thank you for your question. We would like to point out that our method can fuse the different levels of neighborhood. Specifically, we deploy SGC of different layers to aggregate different hops of neighborhood information. Then we apply the self-attention module to adaptively fuse the varying hops of features. Compared to MLP, our backbone applies SGC to capture the graph structure information. Compared to GNN and GPS (graph transformer), our backbone can selectively assign attention weights to aggregated embeddings from different hops, which is more suitable to be generalized to different datasets (different dataset might require a different number of aggregation layers). As a comparison, we also test the performance of zero-shot node classification when using the GCN, GPS and MLP. The results are shown in the table below.

From the table, we could find out that our expert backbone outperforms all other models.

1. You highlighted as an advantage of your method that “Train different expert models on different datasets parallelly, avoiding the potential conflicts from different data domains during pretraining.” However, I find this reasoning unclear. If conflicts exist among different data domains, the expert models trained on these domains independently should also inherit these conflicts. In such a case, there should be no need to combine multiple experts during downstream tuning.

Thank you for your comment. Based on the previous research [1][2], graph datasets from similar domains are more likely to produce positive transfer, while datasets from different domains tend to produce negative transfer. Hence, if all datasets are pretrained together, the model would inevitably suffer negative transfer. Instead, our framework will first pretrain the expert models on different datasets, which prevents the negative transfer during pretraining. Moreover, for inference on a new dataset, the framework will adaptively choose experts of high relevance to merge, thus mitigating the conflicts/negative transfer to a large extent and boosting the positive transfer. This is addressed by our experiments in Figure 5, when merging all the models or random k models, the performance will drop due to the conflict/negative transfer compared to merging the top-k models. Hence, combining top-k relevent experts is useful for downstream task infering.

Reference: [1] Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insights, Chen et al.

[2] Better with Less: A Data-Active Perspective on Pre-Training Graph Neural Networks, Xu et al.

2. You mentioned that pretraining on multiple datasets can be done in parallel. Could you report the training time and GPU memory consumption of your method compared to the baselines?

Thank you for the comment. We list training time and GPU memory consumption in the following table, which manifests that our method has advantages in efficiency due to the lightweight models and parallel framework.

3. In your response to Q3, you stated that “we deploy SGC of different layers to aggregate different hops of neighborhood information.” However, Figure 2 and the description on line 197 of your paper suggest that you repeatedly apply Equation 2, which does not imply using different numbers of repetitions of Equation 2 for different datasets. In fact, on some datasets, increasing the number of repetitions does not necessarily improve performance. Additionally, the experiment you provided in your response is not what I intended to see. My suggestion was to replace the Expert model with other architectures, not to replace the entire pre-training model with a different one.

Thank you for your comment. There are some misunderstandings we would like to clarify. For the SGC aggregations, we keep the ego-subgraph embeddings $h^{(\alpha)}$ after each repetition. This will result in a list of embeddings containing different hops of information, $[h^{(1)}, h^{(2)}, ..., h^{(\alpha)}]$, of a target node as stated in line 205. We did not just repeat aggregations and only kept the final embedding. Then we apply the self-attention module to adaptively fuse these embeddings of different hops since different datasets might favor different levels of aggregations. In this way, our expert model is flexible and can be applied to different datasets.

Regarding the experiments in Answer 3, what we did is indeed replacing the expert model backbone with other architecture, not replacing the whole model. We are sorry that we did not specifically enhance this point in the answer and caused the confusion. If the reviewer has more questions, we are more the glad to answer.

I noticed that both in [1] and [2], comparisons are provided against “using semi-supervised method training from scratch (e.g. gcn, gin, etc)”. However, I am curious why Table 2 in your paper does not include a direct comparison with methods that is trained from scratch on downstream datasets. As widely known, basic semi-supervised learning achieves good performance on these datasets (e.g., results on Cora can exceed 80%), while your method only achieves 75.41 with extensive pre-training. Additionally, in terms of memory usage and computational time, training directly on the downstream graph data from scratch is likely the most cost-effective choice. Could you clarify whether your approach is only better when the downstream dataset does not have any labels?

Thank you for your comment. We would like to point out that [2] does not provide comparisons against semi-supervised methods. All their models and baselines are trained under the unsupervised learning setting (e.g., pretraining + fintuning). Regarding [1], it actually discusses two different settings: co-training (supervised learning) and pertaining (unsupervised learning setting). However, the results from different settings are never compared. Unsupervised learning requires the model to access neither any node label nor the Laplacian matrix of the whole test graph during training, which differs from semi-supervised learning. There have been plenty of works that try to build unsupervised graph learning frameworks. Our work follows this research line and focuses on the unsupervised setting. Building a framework that has superior performance in (semi-)supervised setting is beyond the scope of this paper.

Moreover, in your response to the third part, you mentioned that your method uses a “self-attention module to adaptively fuse these embeddings.” However, I could not find any equations describing this module in the paper. The lack of such crucial details significantly hinders understanding of your method. Even based on your explanation during the rebuttal, if this self-attention module learns different parameters for different datasets, then it should be considered part of the expert model rather than a shared module across all experts. Could you clarify which parts of the model belong to the expert modules and which parts are shared among all experts? I suggest revising the paper to connect the steps of the model more clearly through equations. The current model description in the paper is quite confusing and does not align with the explanations provided in your rebuttal.

Thank you for your comment. We indicate the backbone of the expert model is a transformer block (self-attention module) in line 210 and provide the equations in Appendix C. Nonetheless, we agree with the reviewer that the current presentation could lead to certain misunderstandings and we will revise the sections of model descriptions.

Q1. What is the time complexity or running time of the proposed method given that the proposed method needs to pretrained the model on several graphs from different domains?

A1. Thank you for your question. For a single expert model, the training time complexity is $O(d^2+K^2d)$, where $K$ is the number of hops aggregated for a target node and $d$ is the dimension of the feature vector. For a single gate model, the training time complexity is also $O(d^2+K^2d)$. Thus, the pretraining time complexity on a single dataset is just $O(d^2+K^2d)$. Moreover, since our framework allows us to train multiple experts and gates on different datasets in parallel, it actually would gain efficiency advantages compared to the previous cross-domain pretraining methods.

Q2. In line 207, the authors mention that the node-level feature are randomly maked, resulting in two maked views. Are these two masked views mutually exclusive? For instance, given a 10-d feature matrix, the first masked view is generated by masked 5 features and the second view is generated by masking the rest 5 features?

A2. Thank you for your question. For the contrastive learning on node features, we follow the approach of GRACE to mask the features randomly. We would pinpoint this detail in the revision.

Q3. How do you ensure that the generator only generates a matrix to mask the domain irrelevant features such that the filter features are domain-related?

A3. Thank you for your question. For the generated matrix $a_i$, we tend to let it learn to mask the domain-irrelevant features through the two loss terms $dis(\tilde{f},f_{center})$ and $\frac{1}{dis(\mathbf{o_i,f_{center}})}$ in Equation (4), where $\tilde{f}$ is embedding of original features masked by $a_i$ in the output space, $o_i$ is embedding of $a_i$ in the output space, and $f_{center}$ can be viewed as the representative feature of the training domain. With the loss terms decreasing, the distance between the mask matrix $o_i$ and $f_{center}$ will become larger but $\tilde{f}$ masked by $o_i$ will become closer to the $f_{center}$. Hence, $a_i$ are supposed to learn to mask the domain-irrelevant features of the input vector.

Q4. The authors mention that one key issue is negative transfer. Since the knowledge in the proposed methods is extracted from graphs in different domains, it inevitably increases the probability of facing the negative transfer issue. How does the proposed method address this issue? The top-k strategy seems to only filter out the low confidant knowledge, while it can not directly alleviate the negative transfer issue as the irrelevant knowledge might be included in the top-k expert models.

A4. Thank you for your question. We agree that the negative transfer is inevitable in cross-domain graph learning, and our proposed framework aims to mitigate the level of negative transfer. Specifically, we train the gates and let them to select top-k models that are most relevant to the new task. To deal with possible negative transfer within the top-k models, the gates would calculate the relevant scores for the k models, and their parameters will be weighted according to the score. This is validated by our experiments in Figure 6. When increasing the number of experts, the variance of the performance of top-k strategy is minor. This indicates that even including irrelevant knowledge, the performance will not be affected because of the low relavent score used in weighting. In this way, the proposed method tends to let the most relevant model contribute the most and thus alleviate the possible impacts of irrelevant knowledge included in the top-k models.

Q5. I try to reproduce the experimental results, but there is no instruction and datasets available in the provide GitHub link. The readme seems to be empty. Could you provide the datasets and the instruction to reproduce the results?

A5. Thank you for your question. The dataset sizes are too large to upload to the anonymous github repo. We are trying to re-organize them into smaller ones and will provide them and the instructions shortly.