Thanks for your valuable comments. Our responses are as follows.

R1 for W1.

While there have been some LLM4graphs works which seems achieve feature diversity problem, as highlighted in our paper, they encounter many challenges which is correctly our focus.

• Restricted Feature Input. LLMs require node features to be in text form, whereas FUG can handle node features in various formats, including an amount of floating-point data. Moreover, arbitrarily converting graphs or nodes into text can lead to serious domain biases.
• Structure Neglect. LLMs are unable to process structured data, often resulting in the separate encoding of features and structures in LLM4Graphs. In contrast, FUG can simultaneously encode features and structures.
• Model Complexity. Due to their large number of parameters, LLMs struggle to be optimized by graph tasks in pre-training. FUG, being lightweight, is easy to train and applicable to various scenarios.

Therefore, using LLMs to solve the feature diversity problem is often limited and sub-optimal. As shown in Table 2, OFA is a recent SOTA LLM4Graph approaches, yet its performances on the text-rich graph datasets Cora and PubMed are significantly inferior to that of FUG. On the PubMed dataset, OFA's performance is even worse than simply using the raw bag-of-words node features as input for downstream tasks.

Furthermore, FUG strategy may have the potential to facilitate the development of large graph models, which is more suitable for graph data.

R2 for W2.

Thank you very much for the provided the references.

First, please kindly note that, the main focus of our work is to develop a graph pre-training model which is generalizable across different datasets, while the four cited methods cannot be applied to cross-domain scenarios with varying feature shapes. Besides, the results in Table 2 rather than Table 3 validates our main contribution. As shown in Table 2, FUG demonstrates significant performance advantages compared to the SOTA cross-domain graph pre-training methods, which demonstrate the effectiveness of our method.

In the intra-domain scenario, follow your suggestions, we compare FUG with these works, as shown in the table below. GraphACL and ROSEN results are obtained using the same experimental settings as in our paper. POT, being a plug-in method with experimental settings similar to ours, is directly reported with its best performances from its paper. Due to the unavailability of Sp2GCL code and the considerable differences in the datasets used in their paper, we do not compare with this method.

As shown in the table above, even in intra-domain scenarios (which are not our main focus), FUG demonstrates competitive performance, achieving the best results on three out of five datasets. This proves the effectiveness of FUG and is consistent with the arguments in our paper. We will add these results and discussions in the future versions.

R3 for W3.

In fact, all the three losses are crucial and effective, which is why FUG outperforms the other three methods across all datasets in Table 4. We explain W3 from two aspects.

• In fact, these three losses are not independent; the two RT-FUG losses rely on $L_{DE}$, as it ensures that the input $H$ of GE retains sufficient information from $X$. This is evident in datasets with high-quality features like Photo and Computers, where without $L_{DE}$, the other two losses do not function properly.
• Notably, the two RT-FUG losses have a greater impact on performance in datasets with high-quality structures, such as CiteSeer. This proves the effectiveness of the RT-FUG losses, as they guide GE to encode structures effectively with a low-quality $H$.

Thus, as a pre-training model suitable for a wide range of datasets, all three losses are indispensable for FUG.

R4 for Q1.

In fact, we have already conducted the sensitivity analysis experiments. But due to limited space in the main text, we included the results in Appendix F, Figure 4. We will add a guide in the main text in future versions to help readers find these results.

R5 for Q2.

We apologize for any misunderstanding. In fact, in Equation 9, $z$ is not made to approach $0$. Instead, we enforce $||z||_2 = 1$. Equation 9 ensures that the mean of the normalized representations approaches $0$, which allows the representations to be uniformly distributed on a hyper-sphere centered at the origin with a radius of 1. This guarantees the diversity of the representations.

R6 for L1.

Thank you very much for pointing out the limitation we had overlooked. In our understanding, the Gaussian function you mentioned refers to the "Gaussian potential kernel" provided in "Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere". (If this is not the case, we will be grateful if you could provide new references.)

Using the Gaussian function to achieve uniformity has many advantages. The uniformity metric is asymptotically correct, and the uniformity metric is empirically reasonable with finite number of points. However, it has two problems. a) It relies on correct negative sampling, and b) it has a high time complexity of $O(n^2)$. In contrast, $L_{RT-FUG^-}$ avoids these drawbacks. $L_{RT-FUG^-}$ eliminates the need for manual negative sampling, thereby avoiding sampling bias. Additionally, the time complexity of $L_{RT-FUG^-}$ is $O(n)$. We will include relevant explanations in future versions.

Lastly, thanks for your comments. We would be very grateful if you could engage in further discussions with us. This would be highly beneficial for our work.

Thanks you very much for your comments. The following are our response.

R1 for W1.

The works related to Graph Domain Adaptation (GDA) primarily focus on knowledge transfer between domains. They differ significantly from FUG in several ways.

• First, GDAs are typically supervised in the source domain and often use data from the target domain during training. In contrast, graph pre-training models are mostly fully self-supervised or unsupervised.
• Second, Existing GDAs use datasets with fixed feature shapes. For example, in the work you mentioned, the feature dimensions of the datasets are fixed at 7537. In reality, a trained GDA cannot be universally applied to diverse graph data.

Thank you for your suggestion, we will supplement our discussion with GDA-related works in future versions.

R2 for W2.

This is mainly due to two factors:

• First, as described in Section 1, PCA, as a classic machine learning algorithm, can only serve as a data preprocessing method and cannot participate in graph pre-training. Consequently, during data preprocessing, PCA loses a substantial amount of important information that cannot be recovered through pre-training.
• Second, another disadvantage of PCA methods is their inability to transform features to arbitrary shapes. PCA can only perform dimension reduction, not dimension increase. The dimension of node features varies greatly, as shown in Table 7 (Appendix), where the feature dimension of Physics is 8415, while PubMed is only 500. Using PCA preprocessing can only reduce the dimension of Physics to 500 to match PubMed, resulting in severe information loss.

We will add these discussions in the future versions.

R3 for W3.

Admittedly, using similarity can also align the relative semantics between positives. However, using the l2 distance has advantages, mainly in that.

• First, using the l2 distance can better balance the gradients. The other two losses are defined based on the l2 distance, so defining $L_{RT-FUG+}$ using the l2 distance can make the values of the three losses more comparable, thereby balancing the gradients.

• Second, the l2 distance and similarity are equivalent. As mentioned in Section 3 and Appendix A, reducing the l2 distance and increasing similarity are equivalent under the assumption that the length of the representation vectors is the same. Additionally, both $L_{DE}$ and $L_{RT-FUG-}$ include normalization operations, which essentially act as flexible similarity constraints between representations. The l2 distance constraint in $L_{RT-FUG+}$ can indirectly ensure that the representations satisfy the assumption of equal lengths.

We will add relevant discussions in future versions.

R4 for W4.

FUG does not require a projector, demonstrating the effectiveness of the proxy tasks designed for FUG. We believe that the existing of projectors prevents overfitting to the proxy tasks, thereby increasing the quality of representations. Essentially, the overfitting problem stems from the proxy tasks, as existing CL losses bring bias from negative sampling. However, $L_{RT-FUG-}$ avoids negative sampling by utilizing the concept of global uniformity, thus avoiding artificially defined negative samples. Consequently, our proxy task is more effective and can directly constrain representations, leading to better model performance.

R5 for Q1.

This is a very valuable question. As you mentioned, like GCC, multi-datasets collaborative training only yields slight improvements for FUG. As stated in Section 5.2, we believe this is mainly because the proposed RT task is easy to learn and can be generalized across datasets. Therefore, a small amount of data training can achieve performance close to that in intra-domain scenarios. Additionally, this might be because existing tasks like node classification and link prediction are relatively easy, and the information quality provided by the datasets is high, facilitating knowledge transfer. We will attempt to challenge more complex graph scenarios in future works to explore more effective graph pre-training models.

Thanks for your comments, which are incredibly helpful for improving our paper.

R1 for W1.

We apologize for not explaining this clearly. Here, we provide a formal definition of $Dim(\cdot)$ and $Fea(\cdot)$. $Dim(\cdot)$ is defined as a dimension-specific encoder, formalized as $DimEmb_i = Encode(X_{:,i}^T)$, and $Fea(\cdot)$ is defined as a sample-specific encoder, formalized as $FeaEmb_j = Encode(X_j, \epsilon)$, where $Encode$ represents a linear or nonlinear mapping, $\epsilon$ represents additional inputs. With these definitions, PCA can be decomposed as follows: $DimEmb = CovEncode(\widetilde{X}^T) =\widetilde{X}^T *\widetilde{X}= \widetilde{X}^TW_1$ $FeaEmb = Encode(\widetilde{X})=W_2 X, W_2 = \varphi(DimEmb)$ where $\widetilde{X}$ is the standardized feature matrix, $W_2$ is the parameter matrix for $Fea(\cdot)$, and $\varphi(DimEmb)$ represents the process of solving $W_2$, with $\varphi$ denoting matrix decomposition, sorting and concatenating based on eigenvalues. This clearly demonstrates how the PCA method can be decomposed into the $Dim(\cdot)$ and $Fea(\cdot)$ components.

More detailed definitions and prove can be found in global rebuttal. We will incorporate this into the theoretical section to improve our paper.

R2 for W2.

As mentioned in Sections 1 and 3, our work is primarily inspired by [1]. In [1], the InfoNCE loss is briefly explored as a generalized nonlinear PCA objective function. We further propose a relative semantic relationship encoding objective to better unify the contrastive loss and the PCA objective, and we provide additional analysis of the positive and negative parts. Additionally, as research for the CL and PCA frameworks, we extra-analyze aspects such as data augmentation, encoder, and sampling strategy. We will add these discussions in future versions.

R3 for W3.

Thank you very much for your suggestions. We will correct them in future versions.

[1] The trade-off between universality and label efficiency of representations from contrastive learning, ICLR 2023.

Thanks for your thorough review of our paper and your comments. Here are our responses.

R1 for W1.

(a) We first give the definition of $Dim(\cdot)$ and $Fea(\cdot)$.

Given a data matrix $X\in\mathbb{R}^{N\times D}$, $N$ represents the number of samples and $D$ represents the number of dimensions.

$Dim(\cdot)$ refers to a serious of dimension-specific encoders that directly accept multi-samples input with a specific dimension $X_{:,i}^T$ and output the embedding of that dimension, denoted as $DimEmb_i = Encode(X_{:,i}^T)$, $DimEmb\in \mathbb{R}^{D\times N’}$. In contrast, $Fea(\cdot)$ refers to a serious of sample-specific encoders that accept all dimensions of a specific sample as input and output the embedding of that sample, denoted as $FeaEmb_j = Encode(X_j, \epsilon)$, $FeaEmb\in\mathbb{R}^{N\times D’}$. Here, $Encode$ represent linear or non-linear functions, $\epsilon$ denotes additional inputs.

With the clear definitions, PCA can be decomposed into $Dim(\cdot)$ and $Fea(\cdot)$, with a rigorous proof, which can be found in Global Rebuttal.

(b) Secondly, although Ref [1] you provided decomposes InfoNCE into the tightness part and the contrastive part, our work is theoretically entirely different from it.

• Different Contributions. In [1], the authors aim to analyzing the relationships between several CE-based losses, whereas we aim to unify the PCAs and CL frameworks. Work [1] might be somewhat similar to a small part (task analysis) of our theory. Apart from that, we extra-analyze the augmentation, encoder and sampling strategy.
• Relative Semantic Perspective. In task analysis, we provided a new relative semantic perspective, whereas work [1] tried to connect InfoNCE with labels $y$ (as shown in Equations 3 and Table 2 of work [1]). Given that labels are unknown in SSL, our provided perspective, which only focus on the relationship between data, is more applicable.
• Positive Part Analyzing. Work [1] only demonstrated that positive pairs in InfoNCE are aligned. In contrast, our work reveals that aligning augmented views of the same data inherently aligns representations between different data.

We will increase these in future versions to refine the theory. We will also greatly hope to discuss the theoretical part with the reviewer to further improve this work.

R2 for W2 and Q2.

We guess here you may refer to $L_{DE}$ (in Table 4) rather than CE since FUG does not include a CE component.

The main reason of its significance is that, $L_{RT-FUG^+}$ and $L_{RT-FUG^-}$ are effective when based on $L_{DE}$, especially for datasets with high quality features. As described in Section 5.4, when $L_{DE}$ is absent, DE is not sufficiently trained, which leads to DE embedding $X$ almost randomly to generate $H$, which is an important input for GE. Furthermore, without the guidance of $L_{DE}$, GE only needs to extract relationships in $H$ to reduce the loss, ignoring whether $H$ contains sufficient information from $X$. Therefore, in FUG w/o $L_{DE}$, the $H$ input to GE is of poor quality, resulting in worst performances on some datasets. We will include more explanations in future versions to facilitate understanding.

R3 for W3 and Q1.

Consider that Ref [2] you provided in W3 is also about Graph Prompt Learning, so we combine our responses to your W3 and Q1.

• First, it seems that Prompt Learning (PL) and Pre-Training (PT) are two distinct directions for improving graph learning. To be specific, PT (that you provided) focuses on making models learn knowledge applicable to various data and generate representations, whereas PL (which our FUG belongs to) aims to generalize pre-trained models to different downstream tasks, including few-shot and zero-shot scenarios. Thus, as a universal graph pre-training model, our FUG is primarily compared to some advanced PT models in Tables 2 and 3. Notably, in Table 2, our FUG significantly outperforms some latest SOTA universal PT models, which demonstrates the effectiveness of FUG as a universal graph pre-training approach.
• Second, PL and PT are not mutually exclusive while may work together to achieve better performance. As a flexible PT strategy, FUG can accept various graph inputs and can be integrated with different PLs, thereby extending their generalizability across diverse graph datasets. Additionally, integrating PLs also enhances FUG's performance in few-shot and zero-shot scenarios. Exploring the combination of FUG with various PLs will be our future research direction.

R4 for W4.

In fact, the FUG model has only one more hyper-parameter compared to classical GCLs (or even fewer, as FUG does not require tuning multiple parameters related to augmentation).

• First, in Equation 11, $\lambda_1$ is fixed at 1, requiring only two hyper-parameters to be tuned, while classic GCLs require tuning the temperature parameter $\tau$. Therefore, FUG has just one more hyper-parameter to be tuned.
• Second, as shown in Table 9 (in the appendix), the search spaces for $\lambda_2$ and $\lambda_3$ are small, specifically [0.1, 0.3, 0.5] and [20, 50, 100, 200, 300], respectively. They exhibit clear unimodality in sensitive analysis in Figure 4, making the tuning process straightforward.
• Furthermore, as shown in Table 2 and Figure 4, the model can achieve good results even without meticulous tuning, demonstrating the robustness of FUG.

We will add these discussions in the future versions.

R5 for Q3.

Yes, they can be replaced by other contrastive losses such as InfoNCE since FUG is a highly inclusive theory. However, compared to our losses, InfoNCE have some drawbacks. For example, as described in Section 3 and Appendix B, InfoNCE introduce negative sampling bias and has high computational complexity. These issues are critical for universal graph pre-training models.

Finally, thank you again for your comments. We look forward to further discussing with you to address your concerns and to continually improve our paper.

Thank you very much for recognizing our work. Our responses are blow.

R1 for W1.

The robustness improvement brought by data augmentation mainly stems from its ability to encode perturbed data into the same representation as clean data. Although FUG does not use data augmentation, it retains this characteristic through positive neighbor sampling. Specifically, in homogeneous graphs, linked nodes are considered to have the same semantics, even if their features and topologies are not entirely identical. Therefore, the differences between linked nodes can be seen as a form of perturbation. Compared to data augmentation, this is a method that does not require hyper-parameter tuning and adaptively adds noise, allowing it to ignore noise without encoding inter-class nodes into the same representation.

R2 for W2.

Thanks for your suggestions. We will add them in future versions.

R3 for W3.

Indeed, as a classic and effective machine learning algorithm, it is challenging to completely separate PCA into two unrelated parts. We have added a more precise definition of $Dim(\cdot)$ and $Fea(\cdot)$, where $Dim(\cdot)$ is formalized as $DimEmb_i = Encode(X_{:,i}^T)$, and $Fea(\cdot)$ is formalized as $FeaEmb_j = Encode(X_j, \epsilon)$, where $Encode$ represents a linear or nonlinear mapping and $\epsilon$ represents additional inputs (details can be found in global rebuttal). Under this definition, PCA can be decomposed into $Dim(\cdot)$ and $Fea(\cdot)$ as described in Section 3, because the ordering of eigenvalues can be seen as the process of solving the weights of the $Encode$ in $Fea(\cdot)$. We will include this explanation in future versions.