Uncovering the Redundancy in Graph Self-supervised Learning Models

We sincerely thank you for all the comments and it is a great honor for us for your enjoying our paper. We have addressed all your questions below and hope they have clarified all confusion you had about our work.

[W#1] I in particular wonder why “Although SLIDE significantly reduces the number of parameters in self-supervised GNNs, SLIDE still achieves better performance than FT".

Response:

Indeed, reducing the number of parameters can lead to a slight performance decrease in self-supervised GNNs, specifically when applying FT to Slim GNNs compared to Original GNNs. However, as highlighted in Section 2, we identify strong correlations among different dimensions of embeddings, emphasizing the necessity of introducing de-correlation techniques to enhance the informativeness of graph self-supervised learning models. Therefore, the improved performance of SLIDE despite having fewer parameters is attributed to the incorporation of decorrelation methods. This underscores the effectiveness of our approach in mitigating model redundancy and enhancing model performance. Moreover, in Section 4.2 of our paper we conduct ablation experiments on the de-correlation method, validating our findings in Section 2 and demonstrating its critical importance within SLIDE.

[W#2] Is there any (mathematical) definition on the model redundancy? It is better to verify that the proposed model does reduce the model redundancy.

Response: We appreciate and thank you for your insightful question.

The redundancy of the model currently lacks a mathematical definition, but can be roughly measured using some alternative indicators. Typically, we evaluate the model redundancy by assessing the performance of the models and the correlations within the representations. In our paper, we highlight the beneficial performance of the SLIDE models, which suggests a reduction in model redundancy. As for correlations, we conduct layer-level and neuron-level correlation analyses on embeddings from both Original GNN and the SLIDE models, and these results are presented in the table below. To simplify the evaluation process, we analyze representations of randomly sampled nodes and we use MaskGAE as an example. At the layer level, we assess the correlations between representations of adjacent layers. At the neuron level, we calculate correlations between representations represented by different subsets of neurons within a single layer. All these results are computed using CKA scores. Furthermore, as mentioned in the Conclusion, we recognize the importance of supplementing theoretical analyses for model redundancy, which is a focus of our future work. We aim to provide a detailed mathematical definition of model redundancy in our future endeavors. Once again, thank you for your insightful questions and your appreciation of our work!

Layer Level:

Neuron Level:

We are deeply grateful for your insightful feedback and constructive suggestions. Your thorough review has significantly strengthened the quality of our manuscript.

[W#1] About RFF

Response: Thanks for your good question about Random Fourier Features (RFF).

Deep neural networks exhibit complex dependencies between their features, where simply removing linear correlations is insufficient. A direct approach to address this challenge involves mapping features to a high-dimensional space using kernel methods. However, kernel mapping expands the original feature map to an infinite dimension, rendering it impractical to compute correlations between dimensions. Recognizing the advantageous properties of Random Fourier Features (RFF) in approximating kernel functions and measuring feature independence, SLIDE employs RFF to project original features into the function space of Random Fourier Features. Then, we eliminate the linear correlations among new features, thereby removing both linear and non-linear correlations among the original features.

Without RFF, as shown in Equation 4, the regularization mechanism would degrade, focusing only on linear correlations. This highlights the critical role of RFF in enabling the model to effectively capture and model non-linear relationships. This capability significantly enhances the model's capacity to learn intricate data patterns, which is particularly advantageous in tasks such as node classification.

[W#2] About Equation 2

Response: $\hat{C}\_{H\_{\*,i},H\_{\*,j}}^W=\frac{1}{N_{tr}-1}\sum_{n=1}^{N_{tr}}[(w_nu(H_{n,i})-\frac{1}{N_{tr}}\sum_{m=1}^{N_{tr}}w_mu(H_{m,i}))^T\\ \cdot (w_nv(H_{n,j})-\frac{1}{N_{tr}}\sum_{m=1}^{N_{tr}}w_mv(H_{m,j}))]$

For the meaning of Equation 2, $H_{\*, i}$ and $H_{\*, j}$ mean the i-th and j-th dimension of node embeddings. $W$ means the learnable weights of nodes in the input graph data, and $w_n$ means the weight of the n-th nodes in the training set which has $N_{tr}$ nodes. $\hat{C}\_{H\_{\*,i},H\_{\*,j}}^{W}$ is the partial cross-covariance matrix. What's more, $u$ and $v$ are the random Fourier features function space. Overall, Equation 2 aims to obtain the partial cross-covariance matrix between different dimensions of embeddings through node weighting. And for the significance of Equation 2, it facilitates the minimization of correlations across different dimensions of embeddings by learning the weights W associated with nodes. This approach enables the framework to effectively manage and optimize relationships among different dimensions of embeddings, thereby reducing the model redundancy and enhancing the model's performance.

[W#3] About Weights W

Response: Thanks for your valuable suggestions.

Using MaskGAE on the Cora dataset, we analyze the learned weights W with respect to node degrees. We observe that nodes with smaller degrees tend to have larger weights. Across multiple epochs and runs, we identify the top five nodes with the highest weights for each epoch (referred to as "big answer"). The 8th node consistently appears prominently, representing about 10% of occurrences, while over half of the nodes rarely appear in the "big answer". These findings highlight important nodes in the dataset, suggesting further investigation into their significance.

Our hypothesis links these observations to the graph's underlying structure. We examine the relationship between node degrees and weights by comparing the average degree of nodes in the training set with those of the top ten nodes most and least frequent in the "big answer". Nodes with degrees ≤ 4 average 125.0222 occurrences, while those ≥ 10 average 37.8723 occurrences, reinforcing our hypothesis. We sincerely thank for this insightful question, which has prompted further exploration into an intriguing aspect of our analysis.

[W#4] About the consistency and generalizability of the proposed approach

Response: Thank you for your valuable feedback and suggestion.

In our study, we conduct experiments to investigate model redundancy in graph self-supervised learning models on the classical node classification tasks. These preliminary experiments lay the foundation for our findings. We acknowledge the importance of expanding our analysis to include more diverse graph learning tasks, and we have conducted initial experiments about model redundancy in graph self-supervised learning models on graph classification and link prediction tasks. The results are shown below, illustrating that model redundancy in graph self-supervised learning extends across a broad spectrum of graph learning tasks. This will allow us to further validate and generalize the effectiveness of our approach across different scenarios and datasets. We appreciate your insights and look forward to incorporating them into our future work.

Link prediction:

Graph classification:

We extend our sincere gratitude for the thoughtful review and constructive feedback. Your thorough evaluation has undoubtedly contributed to enhancing the robustness of our findings.

[W#1] The task evaluated in this paper is only node classification. I’m curious what about other tasks? It might be important to check the correlation between the model redundancy problem and node classification, or the problem is task-independent?

Response: Thanks for your question.

We choose node classification as a representative task for evaluating model redundancy in graph self-supervised learning due to its classic and widely studied nature in graph learning. However, the issue of model redundancy is not limited to node classification. We further conduct similar experiments on other graph learning tasks, i.e., graph classification and link prediction, using GraphMAE and MaskGAE, respectively. The results are consistent with those observed in node classification, demonstrating that the problem of model redundancy is indeed pervasive across different graph self-supervised learning tasks.

We appreciate your interest in this aspect and hope that these additional results provide a comprehensive understanding of the issue across various tasks.

Link prediction:

Graph classification:

[W#2] The authors point out that the proposed model is orthogonal to other fine tuning methods. A detailed discussion or more experimental results would be a plus.

Response: Thanks for your thoughtful feedback and interest in our research.

In our study, we have explored how SLIDE differs from previous fine-tuning approaches. Specifically, to demonstrate the orthogonality of our method with traditional fine-tuning approaches, we employ the following approach: we started with the classic fine-tuning method LoRA on GraphMAE, where our SLIDE model randomly prunes a subset of neurons to create frozen Slim GNNs. Subsequently, we introduce an additional LoRA module specifically designed for fine-tuning. To further validate our approach, we apply the de-correlation methods to adjust the LoRA module, aiming to reduce the correlation between embeddings within the Slim GNNs integrated with LoRA modules. The results are shown in the table, where "Slim-LoRA" denotes direct fine-tuning of Slim GNNs using LoRA and "SLIDE-LoRA" is our method.

SLIDE's reduction of model parameters enables fine-tuning of the entire model, highlighting a distinct advantage. However, in SLIDE-LoRA which applies SLIDE based on LoRA, the parameters of Slim GNNs cannot be adjusted; only the parameters of LoRA modules can be modified. This limitation impacts SLIDE-LoRA's performance. Nevertheless, LoRA serves as a coarse method for adjusting model parameters, enabling SLIDE-LoRA to enhance model performance by reducing correlations among final representations. We achieve slightly superior results compared to using LoRA directly on Original GNNs, underscoring the efficacy of SLIDE-LoRA in enhancing model capabilities. The experiment provides empirical evidence supporting the feasibility and efficacy of our method in enhancing model performance and the orthogonality of our method with traditional fine-tuning approaches.

Once again, we sincerely appreciate your valuable feedback and hope that these insights clarify the distinctiveness and effectiveness of SLIDE in the context of fine-tuning methodologies.

We sincerely thank you for valuable opinions and concerns about our work. It is our obligation to describe more details and give more explanations. We really hope that these further efforts can alleviate your confusion.

[W#1] About the issue of redundancy

Response: To demonstrate that redundancy exists across a broader spectrum of graph learning tasks, we conduct experiments on GraphMAE, effective for graph classification, and MaskGAE which excels in link prediction. The results indicate that model redundancy exists across a wide range of tasks.

Link prediction:

Graph classification:

[W#2] About SLIDE

Response: Thanks. We appreciate the opportunity to clarify the contributions and the motivations behind our approach.

Our research has demonstrated the existence of model redundancy in graph self-supervised learning models. Even though we remove a portion of parameters, model redundancy may still occur during fine-tuning. Therefore, based on our findings, we believe that it is necessary to introduce a de-correlation module during fine-tuning. Our experiments have also shown that integrating such a module improves the performance. We observe that models without de-correlation show a noticeable decrease in performance compared to models with de-correlation by conducting statistical significance tests. Specifically, we present the p-values from experiments conducted across 6 datasets using MaskGAE. As can be seen, p-values are less than 0.01 on most of them. These results underscore the statistical significance of the performance differences between models with and without de-correlation.

Regarding computational complexity, the de-correlation method we introduced maintains linear complexity relative to the number of nodes. This ensures that the additional computational cost is minimal compared to fine-tuning Slim GNNs without de-correlation, preserving the efficiency of the models.

[W#3] About comparison

Response: Thanks.

The motivation behind Fig. 6 is to show that despite reducing the parameters, SLIDE maintains comparable performance. We show that our approach effectively reduces the parameter tuning required compared to full fine-tuning of Original GNNs.

Actually, in Section 2 we have already compared the performance and parameter reduction achieved by "linear probing" between Original GNNs and Slim GNNs. Only classifiers are fine-tuned while GNNs are frozen. This results of the comparisons are explicitly detailed in Table 1-2, which also implicitly indicate the proportion of parameters attributed to linear layers (50% and 25%). Furthermore, our experimental results indicate that despite significant parameter reduction, the performance of Slim GNNs does not exhibit a noticeable degradation.

[Q#1] About datasets

Response: As mentioned in Section 2, we pretrain Original GNNs on six datasets (Cora, CiteSeer, PubMed, Amazon-Photo, Amazon-Computers and Ogbn-arxiv), and obtain several kinds of Slim GNNs.

[Q#2] About theoretical analysis

Response: Thanks.

As highlighted in the conclusion, we agree with you and recognize the importance of theoretical analysis in enhancing our understanding. Theoretical analysis of model redundancy represents a significant avenue for future research. Enhancing the theoretical foundations of model redundancy will be crucial for advancing our methodologies.

We could potentially explore theoretical models from several angles. For instance, we could theoretically analyze model redundancy from the perspectives of feature decorrelation. Investigating causal mechanisms within GNN architectures could also offer formal insights into how redundant features affect model predictions. Additionally, developing mathematical formulations to quantify redundant features could offer intuitive insights and further optimize GNN efficiency.

We appreciate your insightful question and recognize theoretical analysis as a critical next step in advancing our understanding and methodologies for addressing redundancy within graph self-supervised learning models.

[Q#3] About statistical tests

Response: We have performed significance tests and calculated p-values to evaluate the performance differences between our method and previous approaches. The p-values for most of comparisons are less than 0.05, indicating that the performance differences are statistically significant. As an example, we present significance test results for MaskGAE in the table below.

Observing the results, out of 12 comparisons, 10 pass the significance test with p-values less than 0.05. One exception is observed in the comparison between SLIDE and FT on Ogbn-arxiv, where SLIDE achieves comparable performance to FT despite using parameters that are 67% fewer.