Many thanks for the valuable comments and suggestions. We have accordingly updated our manuscript and provided responses as follows. All grammar errors and typos are corrected in the updated manuscripts.

Comment #1: What is the performance comparison between SimTeG, GraphFormer and Patton?

Thanks for the question. GraphFormer and Patton are valuable and related works. We have also included them in our related works. We report the results of GraphFormer on our three datasets as follows. For Patton, their implementation only supports their customized datasets and is not trivial to be adopted in our benchmarks. However, since patton and GraphFormer follow similar GNN-nested transformer architecture, it would be valuable to compare the performance of GraphFormer first.

As shown in the following table, SimTeG performs consistently better than GraphFormer. It is possibly because 1) the GNN-nested architecture of GraphFormer solely allows 1-hop message passing, which limits the express ability of GNN models; 2) GraphFormer’s implementation modifies the architecture code of Bert and cannot be easily extended to other SOTA embedding models nowadays.

We have added the GraphFormer results to Table A8 in the paper.

Comment #2: The finding is quite straightforward and not exciting enough to me. It is intuitive that the initial node features are important and if a language model is trained on the downstream task first to generate the node features, it will contribute to a very good performance.

We provide a general response section above to clarify our motivation and contributions. We sincerely appreciate the reviewer checking it. Besides, we also moderately adjust our introduction section to make our motivation and argument clearer.

We partially agree that “if a language model is trained on the downstream task first to generate the node features, it will contribute to a very good performance.” However several crucial questions remain open: (1) what is the extent of “very good”, especially with a simple GNN architecture? (2) Are advanced text embeddings good alternatives to complicated GNN architecture?

We kindly disagree that “the findings are quite straightforward and not exciting enough”. We have summarized our contributions in the general response and hope it would address the concerns of the reviewer. Basically, the central argument is that advanced text features are good alternatives for complex GNNs. Validating this argument is beneficial for the application of GNNs in industry because industry applications of GNN architectures are limited by computational efficiency. To date, a notable exception is Pinsage [1], a GraphSAGE-based recommender system for Pinterest. If incorporating advanced text features could bypass the necessity of using complex GNN models, it would significantly boost the application of GNNs in industry.

Comment #3: The technical novelty is not enough. This method can be seen as a single step for GLEM. The performance comparison with GLEM is also very marginal regarding SOTA GNN.

As the statement of the general response and our abstract, we acknowledge that the proposed method is not technically novel, and this two-stage training has been explicitly or implicitly applied in various applications, e.g., GLEM [2]. Besides, related works [3][4] also utilize a similar mechanism. However, we state two key differences that distinguish our work.

First, the motivation of our work is not to propose a novel framework that could generally outperform the other methods. Instead, we would like to utilize this simple two-stage training to demonstrate our key argument: by incorporating advanced text features, one can bypass the necessity of using complex features. This is mainly demonstrated by the performance improvement of a simple two-layer graphsage.

Besides the key findings and motivation, other unique empirical findings in our work (e.g. The impact of PEFT in Section 5.3 and The impact of LM variants in Section 5.4) could also be generalized to any similar frameworks.

[1] Ying, Rex, et al. "Graph convolutional neural networks for web-scale recommender systems." Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining. 2018. [2] Zhao, Jianan, et al. "Learning on large-scale text-attributed graphs via variational inference." arXiv preprint arXiv:2210.14709 (2022). [3] He, Xiaoxin, et al. "Explanations as Features: LLM-Based Features for Text-Attributed Graphs." arXiv preprint arXiv:2305.19523 (2023). [4] Chen, Zhikai, et al. "Exploring the potential of large language models (llms) in learning on graphs." arXiv preprint arXiv:2307.03393 (2023).

We are thankful for the reviewer’s constructive comments and valuable questions. We respond as follows and hope this could address the concerns. In addition to what follows, we also provide a general response above that may address the common concerns about our motivation and technical contributions to this work.

Comment #1: The technical contribution of the paper appears to be limited, especially when considering GLEM.

Thanks for the question. As mentioned in the general response and our abstract, the primary objective of this paper is not proposing a brand new and novel technique to achieve SOTA results on the leaderboard. Instead, we would like to use the proposed simple framework to validate our key argument: incorporating advanced text features could bypass the necessity of using SOTA GNNs. We refer the reviewer to look through our general responses above for a detailed introduction of our motivation and contributions.

Comment #2: The authors argue that prior methods have crafted intricate tasks and structures to bolster the performance of textual graph learning with LMs. However, existing methods like [1,2,3,4,5] are conceptually simple and their frameworks are straightforward. Moreover, their training processes do not necessitate significant modifications to the prevalent model architectures.

Thanks for the comments. In comparison with SimTeG, prior methods [1,2] and concurrent works [3,4,5] are relatively intricated in terms of their designed tasks or frameworks. For examples, GIANT [1] designs a graph-structure-related self-supervised task to generate text embeddings; GLEM [2] utilizes a joint training framework for GNNs and LMs to iteratively generate representative text embeddings; while the concurrent works [3,4,5] try to utilize LLMs to help generate more powerful text embeddings. Though showing promising results, the introduced tasks or techniques prevent us from directly validating our hypothesis from these works – whether incorporating advanced text features could bypass the necessity of using SOTA GNNs. As a result, we propose a simple framework to assess this argument.

Comment #3: The paper’s motivation is somewhat ambiguous. Given that a rudimentary Graph Neural Network (GNN) can already yield satisfactory results for these tasks, the rationale for introducing a language model, which may be slower in inference and parameter-inefficient, is unclear.

We kindly disagree that “rudimentary GNN can already yield satisfactory for these tasks” – as shown in Table 1, rudimentary GNNs do not work well on text features generated by traditional NLP techniques such as Skip-gram, which is the official feature of OGB. These results motivate us to explore the effect of advanced text features from language models.

In addition, as shown in different benchmarks, the SOTA GNN models are specialists for certain cases. For example, RevGAT is SOTA on OGBN-Arxiv; while SAGN + SLE dominates OGBN-Products; and SEAL-related works are the mainstream for some link prediction benchmarks. However, our results in Tables 1 and 2 show that with advanced text features, a two-layer GraphSAGE could generally perform well on various datasets and two different tasks. This may further ease the application of GNN models on various domains and tasks. We had a discussion about this on Observation 2 in Section 5.1 of the paper.

Comment #4: While the authors have undertaken link prediction experiments, there is a noticeable absence of comparisons with some of the state-of-the-art (SOTA) methods that incorporate LMs.

Thanks for the comments. We did not include the link prediction results of some SOTA methods as few previous works test their methods on link prediction tasks, and their provided code only implemented node classification and is non-trivial to be extended to link prediction tasks. To mitigate the reviewer’s concerns, we tried several related works and included the results of graphformer on our link prediction datasets. We provide it as follows and have added this baseline to Table A8 of the paper.

Following the general response, we note that competing for SOTA results is not the focus of our paper. Instead, we aim to compare the performance of SOTA GNN and GraphSAGE with and without SimTeG, which could validate the central argument of our paper – incorporating advanced text features could bypass the necessity of using complex GNN architectures.

Comment #5: It is suggested to compare the proposed method with GNNs which utilize the bag-of-words feature.

Thanks for the suggestions. As for our used datasets, the OGB does not provide the bag-of-words features, and we did not find GNN models and papers that utilize the bag-of-words feature on OGB datasets. Could you kindly provide more information or reference on this?

Intuitively, we generally believe that the dense word embedding methods should work better than bag-of-words features because the former is recognized to be more representative on summarizing sentences and paragraphs.

Comment #6: On OGBN-Arixv, SimTeG outperforms GLEM. GLEM can adaptively optimize the input embedding for GNNs, which will show better performance compared with SimTeG. Can the authors provide more discussions about this?

Thanks for the question. SimTeG is similar to the first stage of GLEM, except for two key differences.

SimTeG utilizes LoRA during the LM finetuning stage to alleviate the overfitting problem to generate more representative features. Second, SimTeG does not utilize pseudo labels during the GNN training stage. The pseudo labeling is crucial for GLEM and helps it iteratively optimize the embeddings. However, pseudo labeling does not help much on OGBN-Arxiv. As shown in the Figure 2 of GLEM paper, GNN already achieves the best performance on the first stage on OGBN-Arxiv dataset because of the ‘good prediction’ of LM. Instead, GLEM turns out to be more effective on more noisy datasets (e.g., OGBN-Products), where SimTeG fails to achieve on-par performance. We provided a related discussion in Observation 4 of Section 5.1 in the paper.

As stated in the previous response, we believe that our empirical findings like utilizing LoRA in Section 5.3, and the study about impact of LM variants in Section 5.4 would also be beneficial to improve the performance of methods like GLEM.

Comment #7: Could the authors provide further insights into the specific LM variants that can significantly enhance GNN models? For instance, it would be valuable to understand whether larger LM parameters or other factors play a substantial role in this improvement.

Thanks for the question. We have discussed the influence of LM variants in section 5.4 and Table A7 of the original submission. We further elaborate on it and add new empirical results to make the discussion more complete. Basically, we summarize our findings below: the number of LM parameters is not substantial for the improvement. Including the results shown in our paper, we further test 2 more LMs that have different numbers of parameters, instructor-xl (1.5B) and deberta-v2-xxlarge (2.8B). The results on OGBN-Arxiv and statistics are shown in the following table and included in Figure 4 (right subfigure) of our updated manuscript. When we try to increase the number of parameters, the performance does not improve further. This is possibly because even though models with larger parameters could have better fine tuning node classification performance, the resulted text embedding is not correspondingly improved. Thus, we conjecture that the key factor for improving the performance of GNN is the quality of text embeddings, which, as shown as MTEB (https://huggingface.co/spaces/mteb/leaderboard), is not positively related to the number of parameters.

We performed an ablation study regarding the quality of text embeddings on Table A7 in our original paper, where we systematically compared two models with the exactly the same architecture, roberta-large and all-roberta-large-v1. The second one is the finetuned version of the first one using contrastive learning. The results show that for both LM finetuning stage and GNN training stage, all-roberta-large-v1 consistently outperforms roberta-large, indicating that language models specifically finetuned for text embedding tasks are more suitable for our cases.

[1] Learning on Large-scale Text-attributed Graphs via Variational Inference. ICLR 2023 [2] Node Feature Extraction by Self-supervised Multi-scale Neighborhood Prediction. ICLR 2022 [3] Explanations as Features: LLM-based Features for Text-Attributed Graphs. Arxiv [4] Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs. Arxiv [5] Natural Language is All a Graph Needs. Arxiv.

We are thankful for the reviewer’s constructive comments. We address the concerns both in the revised manuscript and in what follows.

Comment #1: Statistical significance tests are missing. In fact, some gaps in Tables 1-3 are subtle and unlikely significant given the reported standard deviation.

Thanks for the suggestions. We compute the p values for two comparisons, SimTeG v.s. baseline (GIANT/OGB) and GraphSAGE v.s. SOTA GNN on three datasets. As shown in the following second table where we compare between GraphSAGE and SOTA GNN, the p values of SimTeG are significantly smaller than the baseline embeddings. Specifically, the p values of SimTeG on OGBN-Arxiv and OGBL-Citation2 are close or larger than 0.05. This further supports our key findings: in cooperation with advanced text embeddings, one can bypass the necessity of using complex GNN models. We have also added the p-value results to the paper (Section A1.2).

It is worth noting that as the results of GLEM are reported by the original paper and we do not have the results for each individual experiment, we are not able to compute the corresponding p values. We do acknowledge that there is a subtle difference between SimTeG and GLEM and GLEM outperforms SimTeG on OGBN-Products. This phenomenon is discussed in our observation 4 in Section 5.1 of the paper

Comment #2: Could you report the performance of GraphFormers?

Thanks for the suggestions. GraphFormer is closely related to our research line and we report the performance of GraphFormers as follows. The results of GraphFormer on OGBN-Arxiv and OGBN-Products are directly borrowed from the GLEM paper [1] since the datasets and split are exactly the same. We run GraphFormer on ogbl-citation2 for 10 times and report the mean with std. For the hyperparameter setting, we use the default parameters, and the batch size is set to 100 to make it consistent with the reported results in GLEM. As shown in the table, SimTeG performs consistently better than GraphFormer. It is possibly because 1) the GNN-nested architecture of GraphFormer solely allows 1-hop message passing, which limits the express ability of GNN models; 2) GraphFormer’s implementation modifies the architecture code of Bert and cannot be easily extended to other SOTA embedding models nowadays.

We have added the GraphFormer results to Table A8 in the paper.

Comment #3: Could you explore other PEFT strategies to check their effect on the overfitting problem?

Many thanks for the suggestion. It would be beneficial to understand other PEFT strategies to check the effects. Due to the time limitation, we conducted the experiments for PEFT with IA3 [2] on OGBN-Arxiv. The results are as follows. As the case study showed, different PEFT may result in similar effects since all of them restrict the learnable parameters of LMs.

[1] Zhao, Jianan, et al. "Learning on large-scale text-attributed graphs via variational inference." arXiv preprint arXiv:2210.14709 (2022). [2] Liu, Haokun, et al. "Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning." Advances in Neural Information Processing Systems 35 (2022): 1950-1965.