Cross-Domain Graph Data Scaling: A Showcase with Diffusion Models

Q4. The authors present experimental results with varying pretraining scales in Figure 3. Although there is a general performance improvement between the SMALL, FULL, and EXTRA scales, there are instances where the model pretrained on smaller graph datasets outperforms those pretrained on larger datasets, such as on the Enzymes and Erdos datasets. Could the authors provide a detailed explanation for these observations

In the SMALL, FULL, and EXTRA scales, we progressively increase the distribution coverage of the pre-training set, resulting in an overall upward trend in downstream performance. We emphasize that the downstream performance is closely tied to the distribution coverage of the pre-training set, which explains the observed performance fluctuations on the Enzymes and Erdos datasets.

For the Enzymes dataset, we observe a slight performance drop when adding 1000 graphs from the Github Star dataset to the FULL set. Graphs in the Enzymes dataset represent small molecules, which differ significantly from those in the Github Star dataset in terms of both the number of nodes and structural properties. Consequently, it is expected that UniAug trained on the EXTRA set does not substantially outperform the FULL set for this dataset. However, it is important to note that the addition of these 1000 graphs results in performance improvements across all other datasets, with only a negligible drop in performance on Enzymes.

Regarding the Erdos dataset, its distribution occupies the scattered space illustrated in Fig. 2, with minimal overlap with graphs in the Network Repository. This limited overlap explains why we do not observe significant performance gains when training UniAug on the SMALL or FULL sets. However, in the EXTRA set, the inclusion of 1000 additional graphs effectively fills the scattered space, leading to a substantial performance boost on the Erdos dataset.

Q5. Can the proposed method be applied to more practical scenarios, such as few-shot or zero-shot learning?

Please see the answer to Q2.

Q6. While the model demonstrates empirically desirable performance, I am curious whether there is a theoretical understanding that supports its efficacy.

One of the key motivations of this manuscript is that GNN model architectures should be aligned with task-specific inductive biases [7]. For instance, GCN performs well on node classification for homophilic graphs, while GIN demonstrates superior performance on graph classification. This highlights the inherent difficulty in designing a single GNN that performs well across all types of tasks.

To address this, graph data augmentation (GDA) methods aim to mitigate the issue by directly modifying the data, effectively aligning data patterns with model designs. For example, Half-Hop [8] adds nodes on each edge to improve GCN's performance on node classification tasks for heterophilic graphs, achieving substantial gains. CFLP [9] generates counterfactual links to enhance link prediction. However, while GDA methods offer performance improvements, they often rely on handcrafted augmentation strategies tailored for specific tasks or datasets, limiting their generalizability to broader applications.

UniAug addresses this issue by leveraging a diffusion model. Intuitively, UniAug learns diverse data patterns by pre-training on graphs across domains, enabling it to adaptively augment downstream graphs with task-specific and data-specific guidance. This allows UniAug to "automatically" align data patterns with the downstream GNN architecture, resulting in empirical success across a wide range of tasks and datasets.

[1] Chen, Zhikai, et al. “Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insights.” NeurIPS, 2024

[2] Hassani, Kaveh, and Amir Hosein Khasahmadi. "Contrastive multi-view representation learning on graphs." ICML, 2020.

[3] Zhu, Yanqiao, et al. "Deep graph contrastive representation learning." arXiv, 2020.

[4] Thakoor, Shantanu, et al. "Large-Scale Representation Learning on Graphs via Bootstrapping." ICLR, 2022

[5] Hou, Zhenyu, et al. "Graphmae: Self-supervised masked graph autoencoders." KDD, 2022.

[6] Liu, Zemin, et al. "Graphprompt: Unifying pre-training and downstream tasks for graph neural networks." WWW, 2023.

[7] Mao, Haitao, et al. "Demystifying structural disparity in graph neural networks: Can one size fit all?." NeurIPS, 2024

[8] Azabou, Mehdi, et al. "Half-Hop: A graph upsampling approach for slowing down message passing." ICML, 2023.

[9] Zhao, Tong, et al. "Learning from counterfactual links for link prediction." ICML, 2022.

---

Thank you again for your constructive feedback and suggestions, which have greatly contributed to improving our work. We believe the additional results and explanations address your concerns and provide a clearer picture of the contributions of our study. We welcome any further input and hope for your strong support for the paper, reflected in an improved score.

To further illustrate the performance improvements of UniAug compared to existing SSL methods, we conducted two additional experiments:

1. Comparison with self-supervised baselines: We compare the performance of UniAug with four representative self-supervised baselines on node classification and link prediction, as summarized in the following tables. The best results are highlighted in bold, with the last column indicating the average ranking. Our results demonstrate that UniAug consistently outperforms the baselines in terms of average ranking, particularly on heterophilic node classification datasets.

2. Few-shot graph classification: In response to the reviewer’s suggestion, we include results for 5-shot graph classification following [6]. These results show significant performance improvements of UniAug over the self-supervised baselines, highlighting the effectiveness of our method under few-shot settings.

Q3. When applying it to downstream tasks, such as node and graph classification, the authors use node labels to guide the generation of graph structures. However, the method may struggle with graphs that lack sufficient label information. It would be helpful to understand whether the approach can be effectively applied to downstream graphs with limited or no label data.

In Section 3.3, we described various guidance objectives at the node, edge, and graph levels, including both supervised and self-supervised objectives. To demonstrate that UniAug can still achieve comparable performance without label guidance, we conducted experiments using node degree as the guidance objective on graph classification datasets. The results are presented in the following table, where we observe that UniAug, guided by node degree, consistently outperforms the GIN baseline across all datasets.

Additionally, as mentioned in our response to Q2, we provide 5-shot graph classification results, which show that UniAug significantly outperforms all self-supervised baselines by a large margin. In summary, while we leverage label guidance when labels are available to achieve better performance, UniAug remains effective with limited labels or self-supervised guidance objectives, demonstrating its adaptability.

---

To be continued in the next reply

Q1. The authors claim to propose a graph foundation model for cross-domain graphs. However, the method appears to be more of a universal graph structural augmentor rather than a foundation model. Additionally, the concept of using graph diffusion models to create structures is not entirely new, which may limit the novelty of the approach.

We agree that UniAug functions as a universal graph structural augmentor, as stated in line 18 of our manuscript:

• We propose UniAug, a universal graph structure augmentor built on a diffusion model.

We emphasize that UniAug is a universal method capable of leveraging graphs from various domains and is applicable to a wide range of downstream tasks. Furthermore, we want to highlight UniAug’s flexibility compared to the models mentioned by the reviewer. Specifically, both GraphAny and Zerog are designed exclusively for node classification, limiting their application scenarios. In contrast, UniAug’s data augmentation paradigm allows it to be applied to any desired downstream task, including but not limited to node classification, link prediction, and graph classification. According to TSGFM [1], the “one-for-all” approach leads to performance drops for node classification when compared to the GCN baseline. However, UniAug consistently provides performance improvements over vanilla GNN baselines across various tasks, including node classification, link prediction, and graph classification.

One of the key contributions of our work is cross-domain data scaling. Existing approaches to graph data augmentation and graph pre-training are restricted to in-domain datasets or rely on handcrafted data filtering strategies, which utilize only a small fraction of publicly available data. UniAug leverages a diffusion model to learn graph data distributions across domains and subsequently performs structure augmentations on downstream datasets. Our experimental results demonstrate that UniAug not only achieves cross-domain data scaling but also delivers benefits across diverse downstream tasks, regardless of dataset scale or domain.

Q2. The authors use node degrees as features for self-supervised baselines, which can significantly impair model performance. Although the authors elaborate on this issue in Appendix D, the experimental descriptions still confuse me, and I cannot find strong evidence that the proposed UniAug outperforms existing SSL methods under comparable settings

In our manuscript, we pre-train all self-supervised baselines on the same pre-training set as UniAug to benchmark performance in a multi-domain pre-training setting. As the pre-training data coverage increases, we encounter diverse feature heterogeneity challenges, including missing features and mismatched semantics. While existing baselines are not well-suited to handle missing feature issues, we address this limitation by calculating node degrees as proxy node features for these baselines.

We acknowledge that this workaround may contribute to performance drops for the baselines. To provide a more comprehensive evaluation, we include both semi-supervised and self-supervised results for graph classification in Appendix D. In the semi-supervised setting, for each data split, the baseline is pre-trained on the train and test sets in a self-supervised manner and then fine-tuned on the training set with labels. In the self-supervised setting, the baselines are pre-trained on the entire dataset, and the learned graph embeddings are subsequently classified using a downstream SVM classifier.

---

To be continued in the next reply

Thank you for your thoughtful and constructive review. We sincerely appreciate the time and effort you’ve invested in providing such valuable feedback. If any questions or concerns remain, we would be happy to continue the discussion and provide further clarification. We look forward to your reevaluation of our work.

Best regards, The Authors

Q1. The authors should clearly illustrate the scale and domain variety of the pre-training data collection in the paper and make it public for the community to conduct further research on the issue. Meanwhile, providing code examples is also essential.

We thank the reviewer for their suggestions regarding data and code. Below, we provide more details about our pre-training data collection process.

Our pre-training dataset is primarily based on the Network Repository, which contains thousands of graphs spanning 33 categories, including biological networks, social networks, and others. We collected all available graphs from the Network Repository, removed duplicates, and filtered out outliers as described in Section 3.1. This process resulted in a collection of 3000 graphs. However, as noted in Section 3.1, the distribution coverage of these graphs was incomplete. To address this, we incorporated an additional 1000 graphs from the Github Star dataset, forming our complete pre-training collection. Both the Network Repository and Github Star datasets are publicly accessible, and we will make the data processing code available upon acceptance of the manuscript.

Regarding the source code for the experiments, we have included it in the supplementary files. The submission also contains running scripts to reproduce the experiments. Upon acceptance, we will provide detailed tutorials to ensure reproducibility and facilitate the use of our methods.

Q2. The authors should consider whether excessively introducing additional datasets for pre-training is appropriate and whether it might produce negative effects.

We thank the reviewer for their valuable suggestion regarding data. We would like to emphasize the importance of data quality in our approach. As discussed in Section 3.1, we filtered out certain outliers from the Network Repository. Specifically, there are extreme cases such as fully connected graphs and empty graphs (i.e., graphs without edges), which differ significantly from the majority of graphs. To ensure the model learns meaningful distributions, we excluded these extreme cases from our pre-training dataset.

Another critical factor in data scaling is the self-labeling process described in Section 3.2. Our pre-training dataset comprises graphs with diverse patterns. To enable the diffusion model to effectively distinguish these patterns, we incorporated a self-supervised labeling process and integrated the self-assigned labels into the model. This step is crucial for achieving positive transfer, as evidenced by the significant performance drops observed in Table 7 when the self-labeling process is omitted.

Finally, we refer to the experiments conducted with three pre-training datasets: SMALL, FULL, and EXTRA collections. As we increase both the scale and diversity of the data from SMALL to EXTRA, we observe a clear trend of performance improvement, as shown in Fig. 3. These findings highlight that, with proper outlier removal and the inclusion of the self-labeling process, our diffusion model and data augmentation paradigm can effectively scale to larger and more diverse datasets.

Q3. Related work about data scaling on graphs should be included in the paper.

We thank the reviewer for suggesting the related works. We will include the mentioned references in the revised manuscript.

---

We greatly appreciate the valuable feedback and suggestions provided, which have helped us strengthen the manuscript. We sincerely appreciate your thoughtful feedback and look forward to your further input and support for the paper.

Thank you for the authors' rebuttal. However, I still have the following concerns.

First, for the datasets and codes released after the acceptance of the paper, it seems to be a less convincing promise, which significantly reduces the reproducibility of the paper. I will not buy the clarification.

Second, I am curious about how to ensure the accuracy of labels output by the self-supervised labeling process.

Third, the authors do not reply to our second question: whether excessively introducing additional datasets for pre-training is appropriate?

We thank the reviewer for the follow-up questions and concerns. Below, we address each point in detail:

Q1. For the datasets and codes released after the acceptance of the paper, it seems to be a less convincing promise, which significantly reduces the reproducibility of the paper.

As mentioned in our previous response, we have included all necessary resources for reproducibility in the supplementary material (a zip file) of the submission. This includes:

• Code for data downloading and preprocessing,
• Processed pre-training data,
• Pre-trained model weights, and
• Running scripts.

The provided bash scripts in the run directory enable full reproduction of the results. We believe this ensures a high level of reproducibility and addresses concerns about code and dataset availability.

Q2. How to ensure the accuracy of labels output by the self-supervised labeling process.

The self-supervised labeling process functions as a clustering mechanism based on graph properties, and as such, does not have a notion of "accuracy" in the traditional supervised learning sense. The primary purpose of this process is to identify and group distribution patterns of the graphs, which the diffusion model leverages to enhance its understanding of diverse graph patterns.

Q3. Whether excessively introducing additional datasets for pre-training is appropriate?

As noted in our earlier response, increasing both the scale and diversity of datasets for pre-training has consistently resulted in performance improvements across various tasks and datasets. From a performance perspective, this demonstrates that the inclusion of additional datasets for pre-training is not only appropriate but also beneficial.

Thank you for your insightful review and for dedicating your time to providing valuable feedback. Based on your comments, we have clarified our statements to best resolve your concerns about our manuscript. We would greatly appreciate it if you could share any further questions or comments.

Best regards, The Authors

Q1. When evaluated in their original semi-supervised or self-supervised settings, these baselines achieve results similar to UniAug. A more accurate comparison would keep the baselines in their original configurations, and it would also help to see similar comparisons for link prediction and node classification tasks.

In our manuscript, we pre-train all self-supervised baselines on the same pre-training set as UniAug to benchmark performance in a multi-domain pre-training setting. As the pre-training data coverage increases, we encounter diverse feature heterogeneity challenges, including missing features and mismatched semantics. While existing baselines are not well-equipped to handle missing feature issues, we address this limitation by calculating node degrees as node features for these baselines.

We acknowledge that this workaround may lead to performance drops for the baselines. To provide a more comprehensive evaluation, we include both semi-supervised and self-supervised results for graph classification in Appendix D. Additionally, we present self-supervised results for link prediction and node classification in the following tables. The best results are highlighted in bold, with the last column showing the average ranking. Our results demonstrate that UniAug consistently outperforms the baselines in terms of average ranking, particularly on heterophilic node classification datasets.

In summary, UniAug stands out over the baselines in two key aspects:

1. Versatility: The structure-only diffusion model of UniAug enables pre-training across graphs from different domains, even in cases where most graphs lack corresponding node features. Furthermore, UniAug’s data augmentation paradigm makes it adaptable to various graph-related tasks, including but not limited to node classification, link prediction, and graph classification.
2. Performance: As demonstrated in the manuscript and supported by the tables above, UniAug achieves superior performance compared to the baselines in both the multi-domain pre-training and self-supervised settings.

[1] Hassani, Kaveh, and Amir Hosein Khasahmadi. "Contrastive multi-view representation learning on graphs." ICML, 2020.

[2] Zhu, Yanqiao, et al. "Deep graph contrastive representation learning." arXiv, 2020.

[3] Thakoor, Shantanu, et al. "Large-Scale Representation Learning on Graphs via Bootstrapping." ICLR, 2022

[4] Hou, Zhenyu, et al. "Graphmae: Self-supervised masked graph autoencoders." KDD, 2022.

---

To be continued in the next reply

Thank you for your thoughtful and constructive review. We sincerely appreciate the time and effort you’ve invested in providing such valuable feedback. We would greatly appreciate it if you could share any further questions or comments. We look forward to your reevaluation of our work.

Best regards, The Authors