Thank you for your detailed comments and constructive suggestions. To address your concerns, we provide the following responses.

Q1. In this paper, the structure-aware ability of SSL-GM is supported by the objective $L_{cont}$. However, its success depends on the quality of the learned representations from the GNN, rather than solely on the capabilities of the student MLP. In other words, $L_{cont}$ enforces the student MLP to output representations similar to the aggregated representations from the GNN but does not inherently empower the student with structure-aware abilities. For unseen nodes during training, the student MLP may encounter challenges in learning structure-aware representations without the supervision provided by the GNN.

Thank you for your insightful comment regarding the structure-aware capabilities of our SSL-GM model. We appreciate the opportunity to clarify how our model addresses the limitations of traditional GNN-MLP methods and its independence from GNN supervision.

Addressing Limitations of GNN-MLP Methods. Our primary goal with SSL-GM is to overcome the challenges faced by existing GNN-MLP models that depend heavily on knowledge distillation from teacher GNNs to student MLPs. We found that the performance of student MLPs in these models is tightly bound to the quality of pre-trained GNNs, leading to limited generalizability, especially for unseen nodes. traditional methods, SSL-GM injects high-order information into MLPs through self-supervised learning (SSL), negating the need for a well pre-trained GNN. This approach ensures that the GNN representations serve as high-order approximations of the MLP representations.

Inherent High-Order Information in MLP Representations. By maximizing the mutual information between MLP representations and the approximated GNN representations, we enable the MLP to inherently contain high-order structural information. This mechanism is key to ensuring the structure-aware capabilities of the student MLP independently of the GNN. To validate this capability, we conducted experiments in inductive and cold-start settings, assessing the model's effectiveness on unseen nodes. The results, as presented in Tables 2 and 3 of our paper, demonstrate the robustness of SSL-GM in these challenging scenarios.

Inductive and Cold-Start Settings. The inductive and cold-start experiments provide concrete evidence of SSL-GM's ability to generalize well to unseen nodes, a critical measure of its structure-aware capabilities. These settings were specifically chosen to test the model performance in scenarios where traditional GNN-MLP methods typically struggle.

Q2. It's worth considering whether addressing the representation shift issue is necessary. The absence of the reconstruction strategy only leads to a slight decrease in performance (as shown in Table 1, "w/o Rec." column), indicating that this strategy may not be indispensable in some cases.

Thank you for your observation regarding the impact of the reconstruction strategy on our SSL-GM model's performance. We appreciate this opportunity to discuss the conditions under which this strategy is particularly impactful.

We concur that the significance of the representation shift issue is closely tied to the size and complexity of the graph. In large-scale graphs with extensive and complex structures, addressing representation shift becomes increasingly important. As illustrated in our experiments, particularly on datasets like Arxiv, the absence of the reconstruction regularizer led to a noticeable performance drop (nearly 1 percentage point). This indicates that for large-scale graphs, the reconstruction strategy plays a crucial role in maintaining the integrity and quality of representations.

Our findings highlight the need for a strategic use of the reconstruction strategy in SSL-GM, applying it judiciously based on the characteristics of the graph at hand. For larger graphs, the strategy is indispensable for maintaining high-quality representations and overall performance. We plan to continue exploring the dynamics of representation shift across different graph sizes and types, further refining our understanding of when and how to effectively implement reconstruction strategies in our model.

Q3. To enhance reproducibility and ensure accurate results, it would be helpful if the authors could provide detailed information about the experimental environment used for SSL-GM.

Thank you for emphasizing the importance of reproducibility in our research. We agree that providing detailed information about our experimental setup is crucial for validating and replicating our results. Below is a comprehensive outline of our experimental environment:

python==3.8.0
torch==2.10
torch_geometric==2.3
torch_scatter==2.1.2
torch_cluster==1.6.3
torch_sparse==0.6.18
ogb==1.3.6

Code Availability and Bug Mitigation. We have meticulously reviewed and updated our code to address potential bugs and ensure consistent performance. The updated code is readily available for download and can be executed in environments such as Google Colab.

Step-by-Step Guide for Replication in Colab.

• Code Setup. Download and upload the updated code to Google Colab.
• Environment Installation. Execute the following commands in Colab to set up the necessary environment.

!pip3 install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
!pip install torch_geometric
!pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-2.1.0+cu118.html
!pip install ogb

• Execution of Experiments. Run the code using the command.

!python main.py --setting trans --dataset cora --use_params --seed 1

For comprehensive results, average the outcomes over ten different seeds (1 to 10), and make 10 train/validation/test splits for each seed to minimize the impact of randomness. In our experiments on Colab, the accuracy for Cora over 10 seeds typically peaks at epochs 200 or 300. The averaged accuracy is $84.63±0.36$.

Ensuring Reproducibility.

We encourage researchers to follow these guidelines for replicating our experiments. Our approach ensures a high degree of reproducibility and accuracy, allowing others to validate and build upon our work confidently.

We hope this detailed information assists in the replication and verification of our experimental results. Our commitment to open and reproducible research is paramount, and we are grateful for the opportunity to share these details.

In light of these responses, we hope we have addressed your concerns, and hope you will consider increasing your score. If we have left any notable points of concern unaddressed, please do share and we will attend to these points.

Thank you for the authors' response. I appreciate their clarification, which has addressed my concerns regarding the experimental settings.

I would like to further discuss the implications of SSL-GM in practical applications. While SSL-GM offers improvements in addressing the limitations of GNN-MLP methods, it introduces an additional training process using a Logistic regression function for test nodes or graphs based on downstream tasks. This approach may hinder the inherited advantage of GNN-MLP techniques, particularly in latency-constrained applications. For example, in node classification tasks where the deployed model needs to handle an arbitrary number of test nodes at any given time, traditional GNN-MLP methods can perform inference on all the nodes simultaneously, while SSL-GM requires additional training time, resulting in increased latency. This trade-off between performance and speed could potentially reduce the primary contribution of SSL-GM in addressing the limitations of GNN-MLP methods.

Additionally, I have noticed the extension of SSL-GM to supervised learning. It is worth mentioning that the performance of SSL-GM drops significantly on the Cora dataset in this extension. Furthermore, the reported results are limited to the semi-supervised setting. I am particularly interested in the evaluation of SSL-GM in the inductive and cold-start settings, which would help validate its value in addressing the limitations of GNN-MLP methods while preserving their practical advantages.

Thank you for your detailed comments and constructive suggestions. To address your concerns, we provide the following responses.

Q1. The authors seem that minimizing each term in (17) also minimizes the corresponding term in (13). Pls prove these conclusions rigorously instead of only using intuitively explanations.

Thank you for highlighting the need for a more rigorous explanation of the connections between the terms in equations (17) and (13). We appreciate the opportunity to delve deeper into the mathematical underpinnings of our model. First of all, we want to emphasize that we consider Eq. (13) only as an instantiation of Eq. (17). Here we will reinterpret the proof process.

Information Bottleneck Perspective. We start by reinterpreting equation (17) from the perspective of the information bottleneck principle. The equation aims to jointly optimize four terms, each representing a different aspect of the information bottleneck:

where $\mathbf{T} = (\mathbf{H}, \mathbf{Z})$, $\mathbf{H}$ and $\mathbf{Z}$ are the representations learned by MLPs and GNNs, respectively.

Modeling the Terms. We model the first term by minimizing the distance between $\mathbf{H}$ and $\mathbf{F}$, represented as $\| \mathbf{H} - \mathbf{F}^* \|$. For the second term, we minimize the distance between $\mathbf{F}$ and $\mathbf{Z}$, corresponding to $\| \mathbf{Z} - \mathbf{F}^* \|$. The third term, $H(\mathbf{H} | \mathbf{Z})$, is approached by optimizing the covariance between $\mathbf{Z}$ and $\mathbf{H}$, expressed as $\sum_i \text{Cov}(\mathbf{H}_i, \mathbf{Z}_i)$. To minimize the last term, $H(\mathbf{Z})$, we make $\mathbf{Z}$ invariant to a specific variable, using $\mathbf{Z}$ to reconstruct $\mathbf{X}$.

Combining these four terms, with some transformation, we derive our final optimization objective:

Clarification on the Connection Between Eq. (13) and Eq. (17). It is important to note that a direct, rigorous connection between the terms in equations (17) and (13) is not necessary. Our final objective serves as a practical implementation of the information bottleneck concept, tailored to the specific context of our model.

In summary, our approach to optimizing equation (17) is grounded in the principles of the information bottleneck, with each term contributing to the overarching goal of learning robust and informative representations. We believe this explanation provides a more rigorous understanding of our method's mathematical foundation.

Q2. Even if the first point holds, (13) minimizes the summarization of 4 terms, and thus the minimizer may not be the minimizers of the 4 terms. Thus, there is no guarantee that (13) and (17) will have the same minimizer. Nowadays, many machine learning papers have theorems but my opinion is that theorems should be rigorous.

Thank you for your critical observation regarding the relationship between the minimizers of equations (13) and (17). We appreciate this opportunity to clarify our approach and address the need for theoretical rigor.

Differentiation Between Eq. (13) and (17). As discussed, equation (13) represents an implementation of the broader conceptual framework outlined in equation (17). Our approach to minimizing equation (17) through equation (13) is a practical instantiation rather than a direct equivalence. We acknowledge that the minimizer of equation (13) may not necessarily be the same as the individual minimizers of the four terms in equation (17). The composite nature of equation (13) means that its minimization involves a balance among these terms, which may not align perfectly with their individual minimization.

Future Research Directions. This realization forms the basis for our future research. We aim to develop more refined implementations that bring the minimization of equation (13) closer to the ideal minimization of equation (17). This will involve exploring methods with lower upper bounds and potentially tighter convergence to the theoretical optimum. We also recognize the importance of theoretical rigor in our assertions and theorems. Moving forward, we will focus on strengthening the mathematical foundations of our model, ensuring that our theoretical claims are robustly supported by rigorous analysis.

Q7. Pls run the experiments on large practical datasets, e.g., Papers100M, MAG240M, and IGB.

Thank you for your valuable suggestion to extend our experiments to include large-scale practical datasets like Papers100M, MAG240M, and IGB. Although these datasets are seldomly used in graph learning experiments, we appreciate your emphasis on the importance of testing our model in diverse and challenging real-world scenarios.

We fully acknowledge the significance of evaluating our model on these large datasets. Such an expansion in our experimental scope would undoubtedly provide deeper insights into the scalability and robustness of our model. We are committed to incorporating this suggestion into our future research plans. Conducting experiments on these larger datasets will be a key focus in our subsequent work, allowing us to further validate and refine our model.

In light of these responses, we hope we have addressed your concerns, and hope you will consider increasing your score. If we have left any notable points of concern unaddressed, please do share and we will attend to these points.

In our SSL-GM approach, the second term of mutual information maximization is directly optimized. We posit that when this second term is maximized, it effectively transforms the first term into $\sum_{i \in \mathcal{V}} I(y_i; x_i | S_i) = \sum_{i \in \mathcal{V}} I(y_i; S_i)$. This transformation aligns our objective with that of traditional GNNs, focusing on maximizing the mutual information between labels and subgraphs, thereby capturing both semantic and structural elements of the graph.

In summary, each model adopts a unique approach to maximizing mutual information, progressively incorporating more complex layers of graph information. Our SSL-GM method synthesizes these approaches, aiming to capture a comprehensive representation of both node features and graph structure.

Q4. It is unclear what are the challenges of using self-supervised learning (SSL) to train MLP and what are the new designs of the paper. SSL is widely used for graph learning, and the author should be very specific in the challenges of using it to train MLP. Currently, descriptions of the weak points of existing works, e.g., “insufficient exploration of structural information when inferring unseen nodes”, “cannot fully model the structural information” are rather vague. In section 3, the authors propose several techniques and loss terms, e.g., alignment loss, data augmentation, and reconstruction loss. These are not new for SSL, which are also evidenced by the citations provided by the authors. The question is that what are the new things proposed by the authors. The paper will be stronger if the authors can connect the challenges and the proposed new techniques.

Thank you for your insightful comment. We appreciate the opportunity to clarify the innovative aspects of our work, particularly in addressing the limitations of existing GNN-MLP methods in inductive, cold-start, and label sparsity settings.

• Insight on Generalization Issues. Our core contribution does not lie in new model architectures and new objectives. Instead, our work identifies the lack of generalization in current methods as a key issue, stemming from the limitations of knowledge distillation (KD) that aligns GNN and MLP outputs only at the label level. We propose that aligning these models in the representation space using self-supervised learning (SSL) can yield more generalizable representations. This insight represents a significant shift from existing practices.

• Challenges in SSL Alignment. Different from KD methods, aligning GNNs and MLPs under SSL is challenging due to the absence of a pre-trained GNN. To this end, we convert the objective to jointly train GNN and MLP by aligning the 0-hop representations (MLP output) with high-order representations (GNN output). Our approach is applicable with various objectives like Bootstrap, InfoNCE, or MSE. This flexibility in alignment strategy is a novel contribution to the field.

• Addressing Representation Inconsistency and Model Collapse. We observed inconsistency in representations between GNNs and MLPs, potentially leading to model collapse (Appendix D.3). Our solution involves using MLP outputs to approximate GNN outputs, a strategy akin to SGC but applied to a different problem context.

• Graph Augmentation and Reconstruction Regularizer. We further incorporate data augmentation to enhance the quality of the learned representations. However, given that structural permutation in augmentation can significantly shift GNN representations, we introduce a reconstruction regularizer to mitigate representation shift. This addition is critical in ensuring the stability and reliability of the learned representations.

Q5. Pls provide the time for model training, which is also an important practical consideration.

Thank you for your valuable feedback. In response to your comment, we have presented additional experimental results in Appendix C of our paper to further validate the efficacy of our SSL-GM model in inference acceleration.

• Accuracy vs Efficiency. We present a comprehensive comparison between our method and other acceleration methods. The results showcase the inference time and accuracy for various methods, including SGC, APPNP, Quantization (QSAGE), Pruning (PSAGE), and Neighbor Sample on Flickr and OGB-Arxiv. We have intentionally omitted GNN-MLP methods that were compared in Figure 3 for brevity. Please note that the time consumption of SAGE and BGRL are the same since the learning process of SSL does not affect inference.

• Different Settings. To ensure a thorough evaluation, we chose three datasets across all node classification settings, including transductive, inductive, and cold-start. Our results, as shown in the additional tables, clearly indicate that SSL-GM outperforms existing methods in terms of efficiency and accuracy.

[Experimental results are following]

Remark. The current implementation, as outlined in equation (13), represents the first step in an ongoing process to optimize equation (17). We are committed to continuous improvement and refinement of our methods. Your feedback is invaluable in guiding our efforts towards more rigorous and effective model development. It helps us identify areas for theoretical and practical enhancement in our research.

Q3. Moreover, I failed to follow the mutual information part of Section 5, especially how the authors transform different models into mutual information forms. If this can be done, pls conduct the mathematical transformations in rigorous ways.

Thank you for pointing out the need for a more rigorous and detailed mathematical explanation of the transformations into mutual information forms for various models. We appreciate this opportunity to delve into the specifics of these transformations.

Transformation Process. Based on the findings in [1], optimizing the cross-entropy $\text{CE}(\mathbf{X}, \mathbf{Y})$ between $\mathbf{X}$ and $\mathbf{Y}$ corresponds to maximizing the mutual information $\text{MI}(\mathbf{X}, \mathbf{Y}) = \sum_{i\in \mathcal{V}} I(\mathbf{y}_i, \mathbf{x}_i)$ between $\mathbf{X}$ and $\mathbf{Y}$.

[1] Malik Boudiaf, Jerome Rony, Imtiaz Masud Ziko, Eric Granger, Marco Pedersoli, Pablo Piantanida, and Ismail Ben Ayed. A unifying mutual information view of metric learning: cross-entropy vs. pairwise losses. In ECCV, 2020.

Model-Specific Transformation. The specific transformation is summarized in the following table.

• MLPs. For MLPs optimizing the cross-entropy between node features and labels, the objective transforms directly into $\sum_{i\in \mathcal{V}} I(\mathbf{y}_i, \mathbf{x}_i)$.

• GNNs. GNNs consider both structural information and node features. We define the computational graph surrounding node $i$ as $S_i = (\mathbf{X}^{[i]}, \mathbf{A}^{[i]})$, where $\mathbf{X}^{[i]}$ is the node features of neighborhoods of target node $i$ and $\mathbf{A}^{[i]}$ is the corresponding subgraph structure, transforming the GNN learning objective into $\sum_{i \in \mathcal{V}} I(\mathbf{y}_i; \mathcal{S}_i)$.

• GLNN. For GLNN that uses MLPs to mimic the predictions of GNNs, we denote the outputs of GNNs as $y_i | S_i$ and thus convert the learning objective $\sum_{i \in \mathcal{V}} I(\mathbf{x}_i; \mathbf{y}_i | \mathcal{S}_i)$.

• NOSMOG and GENN. The following works of GLNN further add structural information in the learning objective, denoted as $\sum_{i \in \mathcal{V}} I(\mathbf{x}_i; \mathbf{y}_i | \mathcal{S}_i) + I(\mathbf{y}_i; \mathbf{A}^{[i]})$.

• SSL-GM (Ours). Our proposed SSL-GM applies contrastive learning to maximize the mutual information between the node features and the high-order representations first and then use the learned representations to do classification, denoted as $\sum_{i \in \mathcal{V}} I(\mathbf{y}_i; \mathbf{x}_i | \mathcal{S}_i) + I(\mathbf{x}_i; \mathcal{S}_i)$.

Detailed Explanation. In the context of MLPs, the focus is on maximizing the mutual information solely between the labels and node features. This approach, however, does not account for structural information inherent in the graph data, potentially limiting the depth of understanding that can be derived from the features alone. GNNs extend this concept by maximizing the mutual information between the labels and the corresponding subgraphs. This approach captures both semantic (feature-based) and structural (graph topology) information, offering a more holistic representation of the graph data. GLNNs further enhance this framework by maximizing the mutual information between node features and the soft labels generated by pre-trained GNNs. This approach leverages high-level abstract information retained in GNN outputs, enriching the feature representation with insights gleaned from the graph structure. NOSMOG and GENN incorporate an additional layer of complexity by maximizing mutual information between the subgraph of the target node and its labels. This addition emphasizes fine-grained structural information, further enriching the model understanding of the graph's intricacies.

Thank you for your detailed comments and constructive suggestions. To address your concerns, we provide the following responses.

Q1. The experimental results only contain the performance over node classification and graph classification. Is it possible to evaluate the proposed method over link prediction?

Thank you for your insightful question regarding the application of our model to link prediction tasks. We appreciate this opportunity to discuss the adaptability of our method to different graph analysis tasks.

Applying the Model to Link Prediction. Our model adopts self-supervised learning (SSL) approach that is inherently flexible and can be applied to link prediction without requiring significant modifications to the graph training process. To represent links, we suggest concatenating the corresponding node representations. This concatenated representation will then serve as the input to the logistic regression model for predicting the presence or absence of a link. For the inference phase in link prediction, we propose adopting a linear protocol where the parameters of the encoder are frozen. This means that during inference, only the logistic regression model, which is used for predicting links, will be trained. The link prediction task can be approached in a standard manner, involving the sampling of some edges from the original graph as positive examples and performing negative sampling for negative edges.

Future Work. While our current experimental results do not include link prediction, we acknowledge the importance of this task in graph analysis. Therefore, we are considering extending our experimental evaluation to include link prediction, providing a comprehensive view of the applicability across various graph-related tasks.

Q2. It seems that this paper only borrows some ideas from contrastive learning. Based on contrastive learning, this paper develops a new objective function that can be used to solve the model generalization problem. Therefore, could the authors highlight some contributions here? I think it is a good paper but the contribution of this paper is a little bit marginal. If the authors are able to emphasize their contributions here, I am willing to improve my rate.

Thank you for your insightful comment. We appreciate the opportunity to clarify the innovative aspects of our work, particularly in addressing the limitations of existing GNN-MLP methods in inductive, cold-start, and label sparsity settings.

• Insight on Generalization Issues. Our core contribution does not lie in new model architectures and new objectives. Instead, our work identifies the lack of generalization in current methods as a key issue, stemming from the limitations of knowledge distillation (KD) that aligns GNN and MLP outputs only at the label level. We propose that aligning these models in the representation space using self-supervised learning (SSL) can yield more generalizable representations. This insight represents a significant shift from existing practices.

• Challenges in SSL Alignment. Different from KD methods, aligning GNNs and MLPs under SSL is challenging due to the absence of a pre-trained GNN. To this end, we convert the objective to jointly train GNN and MLP by aligning the 0-hop representations (MLP output) with high-order representations (GNN output). Our approach is applicable with various objectives like Bootstrap, InfoNCE, or MSE. This flexibility in alignment strategy is a novel contribution to the field.

• Addressing Representation Inconsistency and Model Collapse. We observed inconsistency in representations between GNNs and MLPs, potentially leading to model collapse (Appendix D.3). Our solution involves using MLP outputs to approximate GNN outputs, a strategy akin to SGC but applied to a different problem context.

• Graph Augmentation and Reconstruction Regularizer. We further incorporate data augmentation to enhance the quality of the learned representations. However, given that structural permutation in augmentation can significantly shift GNN representations, we introduce a reconstruction regularizer to mitigate representation shift. This addition is critical in ensuring the stability and reliability of the learned representations.

In light of these responses, we hope we have addressed your concerns, and hope you will consider increasing your score. If we have left any notable points of concern unaddressed, please do share and we will attend to these points.

Q4. Although this work is a combination of existing methods, it is fascinating to introduce this model from a higher level rather than loss function level.

Thank you for your insightful suggestion to present our SSL-GM model from a more holistic perspective. We appreciate this opportunity to highlight the integrative approach and overarching objectives of our model, beyond the specifics of the loss function.

Holistic Approach. In developing SSL-GM, our focus extends beyond the choice of loss functions to a comprehensive consideration of the model architecture and its ability to generalize effectively. This broader view is essential for understanding the innovative aspects of our approach. A key component of our model is the use of contrastive learning between GNNs and MLPs. This strategy is designed to infuse MLPs with generalizable high-order representations, enhancing their capability to capture complex graph structures and features.

Strategic Components. To avoid model collapse, a common challenge when directly minimizing the distance between GNNs and MLPs, we introduce non-parametric aggregation. This component ensures consistency and stability in the relationship between GNNs and MLPs. Further strengthening our model, we apply data augmentation techniques and leverage a reconstruction term. These additions are geared towards improving the model capability and preventing potential representation shifts. Importantly, the choices of augmentations and loss functions are flexible, allowing for adaptability to different graph scenarios.

Emphasizing Generalization and Adaptability. The central theme of SSL-GM is to enhance the model generalization capabilities, enabling it to perform effectively across a variety of graph-related tasks and settings. The flexible nature of key components, including the choice of loss functions and augmentations, reflects our commitment to creating a versatile and adaptable model that can address diverse challenges in graph model acceleration.

Q5. B.3, learning rates is misspelled where ‘5e4’ appears twice.

Thank you for bringing this typographical error to our attention. We sincerely apologize for the oversight and appreciate your meticulous reading of our manuscript. We have reviewed Section B.3 and corrected the error where the learning rate '5e4' is incorrectly mentioned twice.

In light of these responses, we hope we have addressed your concerns, and hope you will consider increasing your score. If we have left any notable points of concern unaddressed, please do share and we will attend to these points.

Q3. In section 3.3, it would be better to introduce representation shift in detail and explain how reconstruction helps in mitigating representation shift.

Thank you for pointing out the need for a more detailed discussion on representation shift and its mitigation. We have revised Section 3.3 to address this.

Defining Representation Shift. We define representation shift as the alteration in learned representations due to data augmentations. While feature augmentation does not significantly alter the distribution of learned representations, structural augmentation can have a profound impact. This shift, particularly pronounced in structural augmentation, can significantly alter the local structure of the target node. For instance, as depicted in Fig. 2, simple edge permutation can dramatically change the 2-hop neighborhoods of a target node, leading to a substantial shift in the representations.

Mitigation Strategy - Reconstruction Regularizer. To counter this problem, we hypothesize that if GNN representations can preserve more localized information, the impact of representation shift can be minimized. Based on this, we introduce a reconstruction regularizer. This regularizer aims to reconstruct the raw node features based on the GNN representations, thereby mitigating the representation shift. The underlying idea is to anchor the learned representations closer to the original feature space, reducing the impact of structural changes.

Effectiveness of Our Approach. The reconstruction regularizer thus serves as a stabilizing factor, ensuring that the GNN representations remain informative and reliable despite augmentations. This approach is innovative in how it leverages the reconstruction concept to maintain the integrity of learned representations in the face of potential shifts.

Reviewer 1

Thank you for your detailed comments and constructive suggestions. To address your concerns, we provide the following responses.

Q1. The article lacks sufficient innovation and is merely a combination and application of existing methods, such as Bootstrap loss, SGC, graph augmentation, and reconstruction.

Thank you for your insightful comment. We appreciate the opportunity to clarify the innovative aspects of our work, particularly in addressing the limitations of existing GNN-MLP methods in inductive, cold-start, and label sparsity settings.

• Insight on Generalization Issues. Our core contribution does not lie in new model architectures and new objectives. Instead, our work identifies the lack of generalization in current methods as a key issue, stemming from the limitations of knowledge distillation (KD) that aligns GNN and MLP outputs only at the label level. We propose that aligning these models in the representation space using self-supervised learning (SSL) can yield more generalizable representations. This insight represents a significant shift from existing practices.

• Challenges in SSL Alignment. Different from KD methods, aligning GNNs and MLPs under SSL is challenging due to the absence of a pre-trained GNN. To this end, we convert the objective to jointly train GNN and MLP by aligning the 0-hop representations (MLP output) with high-order representations (GNN output). Our approach is applicable with various objectives like Bootstrap, InfoNCE, or MSE. This flexibility in alignment strategy is a novel contribution to the field.

• Addressing Representation Inconsistency and Model Collapse. We observed inconsistency in representations between GNNs and MLPs, potentially leading to model collapse (Appendix D.3). Our solution involves using MLP outputs to approximate GNN outputs, a strategy akin to SGC but applied to a different problem context.

• Graph Augmentation and Reconstruction Regularizer. We further incorporate data augmentation to enhance the quality of the learned representations. However, given that structural permutation in augmentation can significantly shift GNN representations, we introduce a reconstruction regularizer to mitigate representation shift. This addition is critical in ensuring the stability and reliability of the learned representations.

Q2. In section 4.3, there is only 'Figure 3' to demonstrate the capability of SSL-GM for inference acceleration. However, it is necessary to provide detailed experimental results regarding accuracy and inference time. These results should be obtained from a wider range of datasets and classification settings.

Thank you for your valuable feedback. In response to your comment, we have presented additional experimental results in Appendix C of our paper to further validate the efficacy of our SSL-GM model in inference acceleration.

• Accuracy vs Efficiency. We present a comprehensive comparison between our method and other acceleration methods. The results showcase the inference time and accuracy for various methods, including SGC, APPNP, Quantization (QSAGE), Pruning (PSAGE), and Neighbor Sample on Flickr and OGB-Arxiv. We have intentionally omitted GNN-MLP methods that were compared in Figure 3 for brevity. Please note that the time consumption of SAGE and BGRL are the same since the learning process of SSL does not affect inference.

• Different Settings. To ensure a thorough evaluation, we chose three datasets across all node classification settings, including transductive, inductive, and cold-start. Our results, as shown in the additional tables, clearly indicate that SSL-GM outperforms existing methods in terms of efficiency and accuracy.

[We list the experimental results in the next part. ]

• Distinct Update Process in SSL-GM. While SGC uses an identity function and APPNP employs a random walk-based message passing for the update process, SSL-GM uniquely utilizes a non-linear function as its update function. This non-linear approach is more akin to the workings of a GCN (Graph Convolutional Network), differentiating our method from both SGC and APPNP.

• Contribution of Non-Linear Aggregation. The use of non-linear adjacency matrix aggregation in SSL-GM is a deliberate choice to enhance the model capability in graph inference acceleration. This non-linear approach facilitates more complex and nuanced representation learning, contributing to the overall effectiveness and generalization of our model, especially in scenarios where linear approaches might be limited.

We believe this detailed explanation, along with the included table, effectively clarifies the unique aspects and contributions of our non-parametric aggregator in SSL-GM. We appreciate your feedback, which has helped us improve the clarity of our methodology.

Q3. So, from the perspective of acceleration effectiveness and improvement in generalization, what is the contribution of non-linear adjacency matrix aggregation to accelerating graph inference?

Thank you for your insightful question. We recognize the importance of elucidating the role of non-linear aggregation in our model, particularly regarding its impact on generalization and inference acceleration.

Improvement in Generalization. Our non-linear aggregation method, in contrast to linear approaches like SGC, incorporates a non-linear function between each layer. This addition is designed to capture more complex patterns in the dataset, potentially leading to the discovery of more generalizable patterns. Moreover, the non-linear function can act as a form of regularization, helping to prevent overfitting and thereby enhancing the model capability to generalize to new datasets.

To substantiate our claims, we have conducted experiments comparing model performance with and without non-linearity. The results, presented in the included table, demonstrate the effect of non-linearity on various datasets such as Cora, Citeseer, Pubmed, and Amazon datasets. These results provide empirical evidence of the non-linear approach impact on model performance.

Acceleration Effectiveness Unaffected by Non-Linearity. Regarding inference acceleration, it is crucial to note that we utilize the trained MLP for inference. Hence, the non-linearity introduced in the GNNs does not affect the inference acceleration. This is because the non-linear characteristics of the GNNs do not impact the MLPs during the inference phase.

Q4. I cannot understand Formula 4. The author injects randomness by perturbing the structure and features of the original graph, with the expectation that the MLP encoder can capture invariant key features. However, Formula 4 is perplexing. The first part of the formula computes the mutual information between $G_1$ and $G_2$, but is it merely a distinction of whether the perturbed structure is included?

We apologize for any confusion caused by Equation 4 and appreciate the opportunity to clarify its meaning and significance in our paper.

Recap. The augmentation process is aligned with the objective of optimal contrastive learning, which aims to train an augmentation-invariant encoder. This is encapsulated in Equation 4, where $\mathcal{E}^* = argmin_{\mathcal{E}} I(\mathcal{G}_1, \mathcal{G}_2) - I(\hat{\mathbf{H}}, \hat{\mathbf{Z}})$. In this equation, $\mathcal{G}_1 = (\emptyset, \hat{\mathbf{X}})$ and $\mathcal{G}_2 = (\hat{\mathbf{A}}, \hat{\mathbf{X}})$ represent two augmented views of the graph - $\mathcal{G}_1$ as the augmented node-set without edges and $\mathcal{G}_2$ as the augmented graph. $\hat{\mathbf{H}}$ and $\hat{\mathbf{Z}}$ are the representations encoded by MLPs and GNNs on these augmented graphs, respectively.

Detailed Explanation:

Random Augmentation. The random augmentation is applied in each epoch to create $\mathcal{G}_1$ and $\mathcal{G}_2$. The first part of the equation computes the mutual information between these two augmented views.

Learning Objective. The equation aims to identify an encoder that can capture all information in the augmented views, regardless of the specific augmentations applied. While achieving this ideal is challenging, the goal is for the encoder to learn the most essential information common to both views, a principle known as mutual information maximization.

Thank you for your detailed comments and constructive suggestions. To address your concerns, we provide the following responses.

Q1. Although a significant number of experiments were conducted, the novelty and contribution of the proposed method remain limited.

Thank you for your insightful comment. We appreciate the opportunity to clarify the innovative aspects of our work, particularly in addressing the limitations of existing GNN-MLP methods in inductive, cold-start, and label sparsity settings.

• Insight on Generalization Issues. Our core contribution does not lie in new model architectures and new objectives. Instead, our work identifies the lack of generalization in current methods as a key issue, stemming from the limitations of knowledge distillation (KD) that aligns GNN and MLP outputs only at the label level. We propose that aligning these models in the representation space using self-supervised learning (SSL) can yield more generalizable representations. This insight represents a significant shift from existing practices.

• Challenges in SSL Alignment. Different from KD methods, aligning GNNs and MLPs under SSL is challenging due to the absence of a pre-trained GNN. To this end, we convert the objective to jointly train GNN and MLP by aligning the 0-hop representations (MLP output) with high-order representations (GNN output). Our approach is applicable with various objectives like Bootstrap, InfoNCE, or MSE. This flexibility in alignment strategy is a novel contribution to the field.

• Addressing Representation Inconsistency and Model Collapse. We observed inconsistency in representations between GNNs and MLPs, potentially leading to model collapse (Appendix D.3). Our solution involves using MLP outputs to approximate GNN outputs, a strategy akin to SGC but applied to a different problem context.

• Graph Augmentation and Reconstruction Regularizer. We further incorporate data augmentation to enhance the quality of the learned representations. However, given that structural permutation in augmentation can significantly shift GNN representations, we introduce a reconstruction regularizer to mitigate representation shift. This addition is critical in ensuring the stability and reliability of the learned representations.

Q2. In Section 3.1, the author introduces the Non-Parametric Aggregator to help align the representations of GNN and MLP. While the author explains the differences in the appendix, the aggregation method given in Equation 2 still resemble the form of APPNP. I did not find an explanation for this issue in the experimental section and other where of the paper. The author claims that, in contrast to SGC and APPNP, SSL-GM uses non-linear adjacency matrix aggregation instead of high-order adjacency matrices.

Thank you for your comment. We acknowledge the need for a clearer explanation regarding our non-parametric aggregator and its distinction from SGC and APPNP. We aim to elucidate this in the following points:

• Objective of Non-Parametric Aggregator. The primary goal of our non-parametric aggregator is to approximate the outputs of GNNs. Our approach does not focus on the detailed architecture of the aggregator but rather on its functional outcome. This perspective allows flexibility in either applying existing methods like SGC or APPNP as non-parametric aggregators or adapting GNNs into a non-parametric form by parameter removal.

• Detailed Differences. To illustrate the differences more clearly, we have included a table in our revised manuscript. This table compares the pre-transformation, aggregation, and update processes of SGC, APPNP, and our SSL-GM.