We sincerely thank the reviewer for the thoughtful comments and constructive suggestions. We provide our feedback as follows.

Q1: The authors use the PPR algorithm, which involves matrix inversion. The sampled subgraph has quadratic complexity.

AQ1: Thank you for your questions. In practice, we do not perform matrix inversion on the entire large graph. Instead, during subgraph sampling, we iteratively compute the local neighborhood, which greatly reduces the algorithm’s complexity. The actual computational cost mainly depends on the number of sampled nodes and the number of iterations, rather than the square of the total number of nodes in the graph.

Additionally, we compared the training and inference times of SSTAG with other methods on large-scale datasets. As shown in Table 5, our model uses only a structure-aware MLP for inference, which significantly reduces computational overhead and ensures scalability in real-world applications.

Q2: What kind of interpretable advantages can the graph + LLM framework provide?

AQ2: The integration of GNNs and LLMs in SSTAG offers several interpretability benefits:

• Semantic-Structural Fusion: GNNs encode local graph topology, while LLMs embed high-level semantic features from textual attributes. Our framework enables interpretability both in terms of which neighboring nodes contribute to predictions (via structure), and what semantic features are being captured (via language).
• Modality Alignment via Memory Anchors: The proposed memory bank mechanism aligns subgraph embeddings with prototypical semantic-structural anchors, facilitating a clearer understanding of what type of substructure or semantic pattern the model relies on.
• Unified Embedding Space: Our unified architecture supports representation sharing across node-, edge-, and graph-level tasks, making it easier to track embedding behaviors across different prediction granularities.

Q3: The authors use a consistency loss instead of a contrastive loss. What is the motivation behind this choice, and are there experiments to verify its advantages?

AQ3: We chose to use consistency loss instead of traditional contrastive loss for several reasons:

• Teacher-Student Distillation: The training paradigm uses a cascaded LM+GNN teacher and a structure-aware MLP student. In this setting, directly aligning the embeddings between the teacher and the student via consistency loss is simpler and more stable than generating positive and negative pairs required by contrastive loss. Moreover, most knowledge distillation works also employ consistency-based objectives for student-teacher alignment [1, 2].

• Memory Consistency: In addition to aligning the student with the teacher, we further encourage consistency between the student and the prototype memory anchors. This extends beyond typical contrastive learning frameworks by leveraging memory-based regularization.

• Empirical Evidence: As shown in Table 3, removing either the student-teacher or memory-based consistency loss leads to a significant performance drop, highlighting the importance of both components.

References:

• [1] Hinton, G., Vinyals, O., & Dean, J. Distilling the Knowledge in a Neural Network. NeurIPS 2015.
• [2] Sun, S., Cheng, Y., Gan, Z., & Liu, J. Patient Knowledge Distillation for BERT Model Compression. EMNLP 2020.

Q4: The authors have demonstrated the superiority of their framework on various tasks, but it seems that they haven't tried the most common graph clustering task.

AQ4: Due to space limitations, our experiments primarily focus on common graph learning tasks such as classification, link prediction, and regression, which demonstrate the effectiveness of the proposed method.

Additionally, we conduct k-means clustering experiments using the embeddings generated by our method on benchmark datasets such as Cora and PubMed. We report standard clustering metrics, including normalized mutual information (NMI) and adjusted Rand index (ARI), and compare our results with other self-supervised baselines. The results show that our method consistently achieves strong performance on clustering tasks, outperforming other competing methods.

Q5: While LLM-based ideas are well motivated, the mechanism could be further explained in terms of semantic alignment between textual and graph domains.

AQ5: In SSTAG, the alignment between LLM-generated semantic features and GNN-based structural features is achieved through the following mechanisms:

• Unified Semantic Space: All node and edge texts are first encoded using a large language model or text encoder (e.g., SentenceTransformer) to obtain high-dimensional, unified semantic embeddings.
• Structural Alignment: These semantic embeddings are then aggregated with their local neighborhoods via a GNN, effectively incorporating structural context. The resulting embeddings are fused and aligned through an MLP, enabling them to retain original semantic meaning while becoming structure-aware.
• Consistency Training + Memory Anchor: To ensure alignment between semantic and structural representations, we employ teacher-student consistency training. Additionally, a memory bank provides stable prototype anchors, which constrain representations across different graph domains to cluster in a shared semantic-structural space. This enables effective cross-domain semantic alignment.

Q6: There are a few minor grammatical errors in the paper. Please check them carefully.

AQ6: We appreciate the reviewer pointing this out. We will conduct a thorough proofreading of the paper to correct all grammatical and typographic errors in the final submission.

We thank the reviewer for the constructive comments. We provide our feedback as follows.

W1: The paper lacks a discussion on GIANT. The authors should clarify distinctions and compare with GIANT in terms of methodology, performance, or application scenarios.

AW1: Thank you for your suggestion. We provide a thorough analysis to clarify the distinctions between GIANT and our SSTAG framework.

1. Methodological Differences:

• GIANT mainly focuses on feature extraction, providing these embeddings to downstream GNN or MLP models. In contrast, SSTAG is designed as an end-to-end structure-aware learning and knowledge distillation pipeline, going beyond feature extraction to support unified node-, edge-, and graph-level tasks. Moreover, SSTAG can also incorporate graph-aware features from GIANT as additional inputs, further enriching the model’s representations.

• While GIANT only fine-tunes language models using graph information, SSTAG aligns and distills the representations of both LLMs and GNNs into a compact student MLP, further enhanced by memory-based consistency. This design improves flexibility and transferability.

2. Application Scenarios: GIANT is primarily designed for node classification tasks. In contrast, SSTAG extends beyond node-level tasks to a broader range of graph applications—including node classification, graph classification, link prediction, and regression, even in cross-domain settings.

3. Experiments: As reported in the GIANT paper, GIANT achieves 73.08% accuracy on the shared benchmark ogbn-Arxiv dataset, and SSTAG achieves a comparable result. In the cross-domain transfer setting, SSTAG is directly compared with strong language-only models, graph-only models, and state-of-the-art multimodal baselines, and consistently demonstrates robust performance even when faced with limited or noisy features.

W2: It is recommended to compare with language-model-only approaches (i.e., by removing Equation 7) to show the performance gains from incorporating graph structures.

AW2: As shown in Table 3, we have reported the results after removing the graph encoder (Eq. (7)), where the student model only extracts information from the language model pathway. The experimental results demonstrate that removing the GNN-based structural component leads to a consistent drop in performance (e.g., WikiCS: 68.76 → 64.34; ogbn-Arxiv: 72.85 → 69.53). This confirms that incorporating explicit graph structure provides significant improvements over variants that rely solely on the language model.

W3: The authors conducted pre-training on ogbn-paper100M for all self-supervised methods, but GraphMAE and GraphCL were not designed for cross-domain tasks. Evaluating them in this manner may not be fair, as they could perform better with target dataset-specific training.

AW3: Thank you for your question. All methods are evaluated under a strict and unified cross-domain transfer protocol, which reflects real-world scenarios where large-scale labeled data on the target graph is typically unavailable. Using methods such as GraphMAE and GraphCL as baselines is a standard practice in the cross-domain text-attributed graph literature [1,2].

As shown in the table below, under the same setting, we train and test on the Product and Pubmed datasets with a 20%/80% train/test split. Even within-domain evaluations, our method consistently outperforms the baselines.

Reference:

• [1] He Y, Sui Y, He X, et al. Unigraph: Learning a unified cross-domain foundation model for text-attributed graphs. KDD 2025.
• [2] Hao Liu, Jiarui Feng, Lecheng Kong, Ningyue Liang, Dacheng Tao, Yixin Chen, and Muhan Zhang. One for All: Towards Training One Graph Model for All Classification Tasks. ICLR 2024.

Q1: The standard split should have been used for fine-tuning on PubMed. Is the poor GCN performance on PubMed due to the use of ST language models for node representations?

AQ1: Thank you for your question. We followed the standard train/validation/test split as specified in the OGB and original PubMed settings. To ensure a fair benchmark on text-attributed graphs (TAG), we use raw text as node features and encode them with the Sentence Transformer (ST) model, instead of traditional TF-IDF or bag-of-words vectors commonly used in the standard GCN pipeline. As a result, the performance of GCN may appear lower than some previously reported results, which often rely on heavily pre-processed features. Our proposed method is specifically designed to effectively handle raw textual attributes, enabling stronger generalization and better performance under this more realistic and challenging setting.

Q2: For molecular graph datasets, did the authors use SMILES strings for encoding?

AQ2: For molecular graph datasets, SMILES strings are used only to construct the molecular graph structure (i.e., the adjacency matrix) and are not used as textual input for the language model. We encode descriptive attributes related to molecular properties—such as the number of aromatic rings, logD, or solubility—as natural language prompts (e.g., “How many aromatic rings does the molecule contain?”). Details and sources of datasets (HIV, BBBP, BACE, MUV, ESOL, LIPO, CEP) are provided in Appendix A.1 to ensure transparency and reproducibility.

We thank the reviewer for the helpful comments. We discuss all the questions raised.

W1: The proposed framework largely relies on existing self-supervised learning and knowledge distillation techniques, with limited novelty in its core ideas.

AW1: The contributions of SSTAG are not simply incremental but represent a novel synthesis and extension for graph learning.

• Unified Task Module: Unlike prior works that target only specific tasks or domains, we propose a unified graph task module capable of handling node, edge, and graph-level predictions within a single architecture. This flexibility enables cross-domain transfer (see Sections 1 and 5).
• Dual-Modality Distillation: To our knowledge, we are the first to co-distill both LLM and GNN knowledge into a structure-aware MLP, combining semantic reasoning and structural understanding (see Sections 4.2 and 4.3).
• Structure-Aware Lightweight Model: We further introduce an in-memory mechanism to capture and align invariant knowledge via memory anchors, which is novel in the context of graph representation learning and critical for generalization (see Section 4.3).
• Experimental Validation: Extensive experiments on multiple benchmark datasets demonstrate the superiority of our proposed SSTAG framework: (a) it consistently outperforms state-of-the-art baselines in cross-domain transfer learning tasks; (b) it exhibits excellent scalability on large-scale graphs compared to existing GNN- and LLM-based methods; and (c) it significantly reduces inference costs while maintaining competitive performance (see Section 5).

The integration of semantic (text), structural (graph), and memory mechanisms into a single, self-supervised, cross-domain pipeline represents a substantial advance over existing methods.

W2: The paper focuses on experiments but lacks in-depth theoretical analysis of the model's performance and generalization, particularly regarding the dual knowledge distillation framework's benefits.

AW2: Our work primarily focuses on proposing a novel structure-aware self-supervised learning framework for text-attributed graphs. The main innovation is the dual knowledge distillation framework, which simultaneously transfers complementary knowledge from both large language models and graph neural networks into a lightweight, structure-aware MLP, thus enhancing scalability and efficiency. Thank you for your suggestion. In future research, we plan to provide a deeper theoretical analysis.

W3 and Q3: While the results are promising, they are based on specific datasets, and the method's generalizability to other graph datasets is unverified. Can SSTAG be applied to graphs with non-text attributes, and what modifications would be needed for such cases?

AW3 and AQ3: This paper focuses on text-attributed graphs, and thus the datasets used are primarily of this type. We demonstrate the generalizability of our method across diverse data domains, task types, and cross-domain settings. Specifically, our experiments cover 12 datasets spanning five domains, with tasks including node classification, link prediction, graph classification, and regression. As shown in Tables 1 and 2, all models are pre-trained on ogbn-Paper100M and evaluated on completely independent graphs, highlighting the strong cross-domain adaptability of our approach.

SSTAG can be extended to non-text-attributed graphs for the following reasons: (1) Framework generality: Its modular design allows the language model encoder to be replaced with any encoder suitable for the given data modality. (2) Component generality: The memory bank and distillation components are modality-agnostic, focusing on aligning invariant graph representations regardless of input type.

With minimal modifications, SSTAG can be adapted for non-text graphs. The unified task template and structure-aware student (MLP) remain unchanged; only the teacher encoder needs to be adjusted to handle the available attribute types. For example, for graphs with numerical or categorical features, the language model encoder can be replaced with an MLP, CNN, or other suitable networks.

Q1: The paper does not detail the individual contributions of each framework component (graph task module, knowledge extraction, and distillation). Ablation studies would clarify the impact of each component on performance.

AQ1: As shown in Section 5.3 (Table 3), we have conducted comprehensive ablation studies. The results demonstrate that removing any component (such as the graph task module, knowledge extraction, distillation, or memory bank) leads to a decrease in performance, sometimes significantly. This confirms the necessity of each part of the framework. We provide detailed explanations below.

Q2: The paper lacks details on hyperparameter selection and sensitivity analyses.

AQ2: The hyperparameter tuning process in this work is divided into three categories. First, some hyperparameters (such as the number of epochs, learning rate, optimizer, and batch size) are selected empirically based on standard practice. Second, certain hyperparameters (such as the masking rate and the number of memory anchors) are optimized using grid search on the validation split, with metrics like accuracy or AUC depending on the specific task. Third, for parameters with complex interactions, we select the best values based on cross-validation across multiple settings. The selection criteria and the actual values used in our experiments are detailed in the table below. It is worth noting that optimal values may vary slightly across different datasets.

Limitations: The discussion of limitations, including scalability, dynamic graphs, and resource efficiency, is brief. More detailed analysis and evaluation of these aspects—especially for dynamic graphs and optimization strategies—would strengthen the paper. Future work should address these points.

Thank you for your suggestion. This work primarily focuses on text-attributed graphs, and further exploration is needed for dynamic and large-scale graph scenarios. In future work, we plan to conduct a more in-depth analysis of the challenges posed by dynamic and large-scale graphs. We will provide a more detailed discussion of these aspects in the revised version of the paper.

We thank the reviewer for the constructive comments. We provide our feedback as follows.

W1: The paper lacks code and hyperparameters, limiting reproducibility and affecting the overall score.

AW1: Due to the limitations of the rebuttal process, we are unable to provide a link to the code at this stage. We will release the code publicly upon acceptance of the paper. In the meantime, we have provided detailed hyperparameter settings in the table below to facilitate reproduction of our results.

W2: Eq. 2 seems incorrect as it outputs a vector instead of a scalar, based on Eq. 3. Additionally, no citation is provided for the PPR algorithm, despite references to unrelated papers.

AW2: In Eq. (2), $\pi_{v}$ denotes the Personalized PageRank vector (dimension $N \times d_{e_v}$), which encodes the structural importance of all nodes relative to the target node $v$. In Eq. (3), $\pi_u$ refers to the $u$-th scalar entry of that vector, i.e., the importance score of the specific node $u$.To remove any ambiguity, we will rename $\pi_u$ in Eq. (3) to $\pi_{vu}$ in the revised manuscript.

The PPR algorithm is a classical algorithm from the reference listed below. We will add the corresponding citation in the main text.

• Page, L. et al. The PageRank Citation Ranking: Bringing Order to the Web. Stanford InfoLab, 1999.

W3: The reported accuracies of GCN and other supervised methods are notably lower compared to existing literature.

AW3: To ensure a fair comparison under the TAG setting, we utilize raw textual features rather than hand-crafted or pre-processed features in our experiments. Consequently, the reported accuracies of GCN and other supervised methods are lower than those typically observed in the literature, where higher results often rely on pre-processed features such as bag-of-words or TF-IDF. With raw textual attributes, GCN’s performance is lower, as GCNs generally struggle with high-dimensional and noisy text features. Our proposed SSTAG method overcomes this challenge by effectively integrating semantic and structural information, as evidenced by its strong performance (e.g., 72.85% on ogbn-Arxiv).

W4: The integration of components, especially the memory bank in the pipeline, is unclear. A schematic would help clarify the method.

AW4: Thank you for your valuable suggestion. As described in Section 4.3, we have explained the memory bank. For further clarification, we provide a detailed overview of the entire pipeline here:

• Our pipeline takes raw text as input and processes it sequentially through the following modules. In the teacher model, the text is first encoded by a large language model (LLM) encoder to extract semantic representations. These are further processed by a GNN encoder to incorporate structural information from the graph. In the student model, the semantic representations are refined by a structure-aware MLP to enhance the combined features. The resulting embeddings are stored and updated in the memory bank, which plays a key role in preserving and enhancing the learned semantic-structural information. Consistency loss is applied to encourage stable and robust feature learning. Finally, the pre-trained models are used in downstream tasks.

We will include a diagram of the entire pipeline in the revised version to visually illustrate the workflow and the specific role of each component.

W5: There are no comparisons regarding the accuracy impact of using a distilled MLP model instead of the GNN during inference.

AW5: Thank you for your question. First, we would like to point out that most existing distillation methods do not report the inference performance of the teacher model, but rather focus on the results of the student model [1，2]. Additionally, we have conducted further experiments (see the table below) comparing the inference accuracy and efficiency of the distilled MLP model with the GNN teacher model. The results show that the distilled MLP achieves a 35.1% improvement in inference efficiency, a 99.7% reduction in parameters, and only a slight drop in accuracy (1.06%) compared to the GNN. This demonstrates the effectiveness and efficiency of our approach. We will include these results in the revised manuscript.

Reference：

• [1] Hinton, G., Vinyals, O., & Dean, J. Distilling the Knowledge in a Neural Network. NeurIPS 2015.

• [2] Sun, S., Cheng, Y., Gan, Z., & Liu, J. Patient Knowledge Distillation for BERT Model Compression. EMNLP 2020.

Q1: In Table 5, the training/pretraining times would make a lot more sense if the computational hardware used was also provided. Can you please specify the hardware used in the paper?

AQ1: Thank you for your suggestion. All experiments were conducted on a Linux server equipped with 945 GB of RAM and eight NVIDIA A100 GPUs, each with 40 GB of memory. The software stack comprised Python 3.11, PyTorch 2.0.1, DGL 1.1.2, Transformers 4.32.1, and CUDA 11.8.

Q2: Are the language models trained from scratch during SSTAG pretraining? Or is a checkpoint used? If a checkpoint is used, is it finetuned during the SSTAG training or is it kept frozen?

AQ2: Thank you for your question. We do not train the language models from scratch; instead, we use publicly available pre-trained checkpoints for all LLM components. During the SSTAG pretraining stage, the language model is kept frozen—only the GNN, the distilled MLP, and the memory bank parameters are updated. This design makes the SSTAG framework both efficient and scalable, and avoids substantial computational and data requirements.

I thank the authors for addressing my concerns. Overall, I am inclined to increase my score. However I have two more things I would like to discuss further:

1. It is a little curious that SSTAG reaches the best performance on agbn-arxiv in the last row of Table 1 of the last response. This seems like a very hard setting - only 10% labels AND the features used are not ST-encoded embeddings (something SSTAG was designed with). How should I understand this result?

2. I am curious as to why the authors insist on only providing the accuracies of the distilled MLP in the paper, when it seems like the final trained GNN has better performance (based on their rebuttal). Including the GNN performance numbers would atleast aid researchers in the future when they want to compare to SSTAG, but are not operating in the distilled setting (i.e. when they want to maximize performance and do not care about inference time compute).