ML$^2$-GCL: Manifold Learning Inspired Lightweight Graph Contrastive Learning

We sincerely appreciate the reviewer’s positive feedback and careful reading. Below, we will provide a point-by-point response.

W1: See Q of Reviewer m6R5.

W2: This may be caused by software incompatibility. You can try Adobe Acrobat 10.0 to open the PDF file.

W3: We will modify this in the final version.

C: We did consider using other types of graph encoders, such as Graph Transformers (GT). However, our decision to adopt GCN as the encoder was ultimately driven by the following factors:

1. Computational Efficiency and Model Lightweight

One goal of this work is to develop a lightweight graph contrastive learning framework. Compared with Graph Transformers, GCN offers lower computational complexity and fewer parameters, making it easier to train on large-scale graph data while maintaining robust performance.

2. Compatibility with Manifold Learning

Our method is rooted in manifold learning, emphasizing the preservation of local linear relations. GCN’s hierarchical neighborhood aggregation mechanism inherently aligns with this design philosophy, enabling the learned embeddings to better retain local geometric information. In contrast, Graph Transformers primarily rely on global attention, which, while effective for capturing long-range dependencies, may dilute the fidelity of local geometric structures.

3. Experimental Results and Comparability

To ensure fair comparisons with mainstream GCL methods, we adopted the same encoder setup. Since most baseline models use GCN, maintaining consistency ensures experimental comparability and highlights the advantages of our framework itself.

Even so, we acknowledge the potential of Graph Transformers in modeling long-range dependencies. Future work could explore their integration into the ML²-GCL framework to further enhance global structural modeling capabilities.

Q: The lightweight modeling in this paper primarily relies on the following core technologies and solutions, which reduce computational complexity while ensuring model effectiveness and generalization capability:

1. Manifold Learning-Driven Positive Pair Weight Calculation

We compute the positive pairs weight matrix W_p with locally linear embedding, enabling direct learning of global nonlinear representations on the original graph structure without pairwise distance computation.

This strategy applies to other contrastive learning tasks requiring structural preservation, such as text and protein interaction networks. Additionally, it can be used for dimensionality reduction tasks, e.g., manifold-based image feature extraction.

2. Closed-Form Solution Optimization

We solve the positive pairs weight matrix W_p through a closed-form solution, which significantly reduces computational complexity compared with gradient descent-based optimization methods, making the approach more lightweight. This closed-form solution can be generalized to unsupervised dimensionality reduction, spectral clustering, and other tasks, particularly suitable for large-scale datasets such as graph neural network pretraining.

3. Single-View Modeling Avoiding Data Augmentation

Traditional graph contrastive learning relies on data augmentation to generate multiple views. This study implements contrastive learning directly on a single view, reducing storage requirements while eliminating semantic bias introduced by augmentation.

This method applies to scenarios where data augmentation is difficult to define, such as social network analysis and biological graph applications, where complex data structures pose challenges in generating high-quality augmented views.

The lightweight design in this study is not task-specific but based on rational improvements in optimization strategies, sampling methods, and modeling approaches. It can thus be generalized to other graph learning tasks, visual contrastive learning, and embedding learning in natural language processing. We believe these strategies can inspire the development of more efficient and scalable unsupervised learning methods.

We sincerely appreciate the reviewer’s positive feedback and attention. Here, we will provide a point-by-point response.

W1: Thank you for your attention. In fact, the proposed novel contrastive loss can be regarded as personalized graph embedding, where positive pairs with larger weights should be more similar, while negative pairs with smaller weights should be more different.

W2: In future, we will investigate deep integration mechanisms between manifold learning and dynamic graph structures to address nonlinear evolution patterns in temporal data or dynamic networks. We can also extend such unsupervised contrastive paradigms to multimodal data for efficient computation in large-scale graph scenarios. The content will be supplemented upon acceptance of the paper.

C: We will modify "While existing works follow the basic principle that pulling positive pairs closer while pushing negative pairs far away..." to "While existing works follow the basic principle of pulling positive pairs closer and pushing negative pairs far away...".

Q: Personalized graph embedding is a more generalized framework which assigns each positive pair or negative pair its own weight. Traditional graph embedding can be regarded as a special form of personalized graph embedding. In Appendix A.3, we have briefly discussed the differences.

Thanks for the reviewer’s careful reading and valuable suggestion. In the following, we will provide a point-by-point response.

Wa: In future, we will investigate deep integration mechanisms between manifold learning and dynamic graph structures to address nonlinear evolution patterns in temporal data or dynamic networks. We can also extend such unsupervised contrastive paradigms to multimodal data for efficient computation in large-scale graph scenarios. The content will be supplemented upon acceptance of the paper.

Wb: The code will be made publicly available upon acceptance of the paper. For experimental details, please refer to Appendix C.

Ca: Thanks for the careful reading. We will modify this in the final version.

Cb: This may be caused by software incompatibility. You can try Adobe Acrobat 10.0 to open the PDF file.

Q: Thank you for the question. Our method, ML²-GCL, intentionally excludes the graph encoder when calculating the reconstruction weights for positive pairs. This is a deliberate design choice, primarily motivated by the following reasons:

1. Avoiding Encoder Interference to Ensure Geometric Consistency

Traditional GCL methods typically rely on embeddings from graph encoders to measure node similarity and construct positive/negative pairs. However, this approach is susceptible to instability caused by encoder initialization, training states, and parameter updates, which can lead to inconsistent positive pairs selection.

In ML²-GCL, the calculation of positive pair weights is based on locally linear embedding-based geometric optimization, directly constructing the positive pairs weight matrix W_p in the raw feature space. This avoids biases introduced by graph encoders, ensuring that W_p is solely determined by the inherent geometric structure of the data, aligning with the principles of manifold learning.

2. Enhancing Robustness of Contrastive Learning

Computing W_p in the raw feature space can serve as prior information for the encoder’s input data, providing stable supervisory signals. In contrast, computing W_p after GCN processing would risk instability, as updates to GCN parameters could disrupt the construction of positive pairs. Furthermore, positive pair selection should depend on the graph structure and node features themselves, rather than the encoder’s initial state.

3. Reducing Computational Complexity and Improving Efficiency

Traditional GCL methods require similarity calculations in the encoder’s embedding space, whereas ML²-GCL computes W_p via a closed-form solution, significantly lowering computational costs and enabling a lightweight framework. Our method also avoids exhaustive pairwise similarity computations by focusing on local geometric structures, further enhancing efficiency.

4. Ensuring Generalizability Across Graph Encoders

Since W_p is encoder-independent, ML²-GCL can flexibly integrate with various graph neural networks, such as GCN, GAT, and GraphSAGE, without altering the positive pair construction strategy. This design ensures broad applicability, allowing ML²-GCL to adapt to diverse GNN architectures seamlessly.

In summary, our design is grounded in manifold learning theory, ensuring stable positive pair construction, computational efficiency, and method generalizability. The experimental results validate the effectiveness of this approach, demonstrating its theoretical and practical soundness.

We sincerely appreciate the reviewer’s positive feedback and valuable comments. Below, we will provide a point-by-point response to each comment.

W1: We have briefly discussed this in Introduction. Here, we will give a detailed discussion.

Similarities

1. Consistency in Core Objectives

Both aim to extract effective low-dimensional representations from complex data structures. Manifold learning reveals low-dimensional manifold structures embedded in high-dimensional data through dimensionality reduction, while graph contrastive learning learns discriminative embeddings through contrasting different graph structures or node relations.

2. Focus on Local Structures

Manifold learning algorithms preserve local geometric properties through constructing neighborhood relations. Graph contrastive learning enhances sensitivity to local structures through contrasting node or subgraph neighborhoods.

3. Advantages in Handling Nonlinear Relations

Manifold learning captures nonlinear low-dimensional structures of high-dimensional data. Graph contrastive learning excels at modeling non-Euclidean relations of graph data.

Differences

1. Theoretical Foundations and Input Data Types

Manifold learning is based on topological manifold theory, assuming high-dimensional data lies on low-dimensional manifolds. Its inputs are high-dimensional vectors, such as images and text. Graph contrastive learning is rooted in graph theory and contrastive learning frameworks. Its inputs are graph-structured data, such as nodes and edges.

2. Methodological Approaches

Manifold learning reduces dimensionality via constructing local neighborhoods or global constraints, while graph contrastive learning generates positive/negative pairs and optimizes models using contrastive loss functions to distinguish similar/dissimilar structures.

3. Application Scenarios

Manifold learning is primarily used for data visualization, denoising and feature extraction. Graph contrastive learning is applied to graph classification, node classification and graph generation, particularly in scenarios with missing or improvable graph structures .

4. Mathematical Tools and Optimization Goals

Manifold learning relies on graph Laplacian matrices, geodesic distance calculations, and minimizes reconstruction errors or preserves local geometry. Graph contrastive learning uses contrastive loss to maximize similarity of positive pairs and minimize similarity of negative pairs, focusing on discriminability of representation spaces.

W2: We will modify "While existing works follow the basic principle that pulling positive pairs closer while pushing negative pairs far away..." to "While existing works follow the basic principle of pulling positive pairs closer and pushing negative pairs far away...".

W3: We will modify this in the final version.

Q1: Yes, both of them are square matrices with dimension N.

Q2: We observe that the optimal range of k values is generally small across all datasets, which can be primarily explained from the following two perspectives:

1. Theoretical Importance of Local Neighborhoods

In ML²-GCL, k determines the scope of the positive pair set P_i, where the positive samples of an anchor node are derived from its k-hop neighbors. Theoretically, a smaller k makes positive pairs more localized, enhancing the preservation of local geometric structures while reducing noise interference from distant neighbors. Our optimization objective, based on manifold learning, constructs the positive pairs weight matrix W_p through locally linear embedding, making it more suitable for information propagation within small-scale neighborhoods. When k becomes excessively large, distant positive pairs may introduce excessive noise, compromising the fidelity of manifold structures and resulting in overly homogeneous representations that degrade contrastive learning performance.

2. Empirical Observations on k-Sensitivity

Our experiments reveal that larger k values lead to performance degradation and reduced model discriminability. This may occur because a larger k introduces excessive positive pairs, potentially amplifying noise and destabilizing local structures. When k increases, more nodes are included in the positive pair set. However, information from many of these nodes may already have been propagated through GCN layers, leading to information redundancy and diminished model discriminative power.

Based on the above analysis, we conclude that smaller k values allow ML²-GCL to better preserve manifold-localized structures while avoiding over-smoothing issues, thereby delivering superior experimental performance.