Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation

Thank you for your insightful and positive feedback! Below we respond to your questions and concerns.

Q1: As mentioned in Section 3.2, the authors mentioned that the designed prompt template is shown in Figure 2. In addition, this reviewer is interested to know some more details about the design of “in-context demonstrations”. a) Does SubgraphRAG utilize the same “in-context demonstration” for all the questions? b) How were these in-context demonstrations designed? Specifically, what are the criteria for the selection of these demonstrations? How do you generate the explanations?

Response: a) Yes, all questions use the same in-context demonstration.

b) The in-context demonstration is designed with the goal of encouraging LLMs to produce reasoning that is grounded in evidence. To achieve this, we crafted a template that includes a reasoning chain as part of the demonstration. We want this template to serve to guide the LLMs in generating explainable QA results by illustrating how reasoning with evidence can lead to the correct answer. For a concrete example, you may refer to Figure 7 in Appendix.

Q2: The authors raise concerns about the reproducibility issue of the ToG baseline. Since ToG relies on closed-source LLMs such as GPT4, this reviewer believes that the performance can be greatly affected by the specific versions of LLMs. SubgraphRAG also uses closed-source LLMs for evaluation. Which specific version of GPT4o is being used in this work? (Please also include this in the implementation part in the paper.)

Response: We thank the reviewer for raising this point. We used gpt-3.5-turbo-1106, gpt-4o-mini-2024-07-18, and gpt-4o-2024-08-06 in our experiments. We have added this statement in Sec. 4.2 in the revised manuscript.

Thank you for the valuable constructive feedback! Below we respond to your questions and concerns.

Q1: The retrieval evaluation, as presented in Table 1, relies solely on the recall metric. However, SubgraphRAG retrieves a larger number of triplets (e.g., top-100), which should naturally achieve higher recall than the baselines. To ensure a fair comparison, additional metrics such as precision, F1 score, or recall @ k should also be included.

Response: We appreciate this important observation about evaluation metrics. In Figures 3 and 5, we plot retrieval recall values against varying numbers of retrieved triples (recall @ k). These results demonstrate that SubgraphRAG consistently outperforms baseline retrievers when operating under identical retrieval budgets. Since we maintain controlled retrieval sizes, superior recall directly implies better precision and F1. Furthermore, our analysis reveals that SubgraphRAG exhibits favorable scaling properties: as more top-ranked triples are incorporated, the recall improvements remain substantial. This scaling characteristic is particularly valuable for accommodating the varying long-context reasoning capabilities of different LLMs.

Q2: Is SubgraphRAG still superior if we control the LLM used for reasoning? In particular, it does not outperform RoG with Llama 2-7B on CWQ when combined with Llama 3.1-8B.

Response: We appreciate the reviewer’s astute observation. While the observation made is correct, it’s crucial to understand the root cause.

To rigorously address this concern, we conducted comprehensive ablation studies that isolate the retriever’s impact by controlling the LLM reasoner. Our experiments employ Llama 3.1-8B and more powerful GPT-4o-mini as fixed LLM reasoners while varying only the retrieval method. The results below demonstrate that SubgraphRAG consistently outperforms RoG across both LLM reasoners and datasets.

Llama 3.1-8B + WebQSP

Llama 3.1-8B + CWQ

GPT-4o-mini + WebQSP

GPT-4o-mini + CWQ

Regarding the observation made in the question for comparing against RoG + a 7B LLM, it’s important to note that the 7B LLM employed by RoG has been directly fine-tuned on CWQ. In contrast, our approach only prompts pre-trained LLMs, unlocking the potential in adopting advanced closed-source LLMs while maintaining generalizability. The controlled experiments provide strong evidence that SubgraphRAG's performance improvements stem substantially from its enhanced retrieval capabilities.

Q3: How well does SubgraphRAG perform when compared to alternative lightweight GNN-based approaches like EtD and GNN-RAG?

Response: We’ve performed additional empirical studies and updated the manuscript accordingly. The studies demonstrate the superior performance when compared to GNN-based approaches.

For retrieval evaluation, we are only able to compare against GNN-RAG as the official implementation of EtD remains unavailable. GNN-RAG employs two GNNs to predict answer entities and then extracts shortest paths between topic entities and predicted answer entities. Our retriever consistently outperforms GNN-RAG (Table 1, Figure 3, and Figure 5). Notably, while our retriever is not explicitly trained to predict answer entities, it outperforms GNN-RAG in answer entity recall using equivalent retrieval budgets (Figure 3 and 5). In terms of efficiency, our retriever is an order of magnitude faster than GNN-RAG (Table 1).

For end-to-end KGQA evaluation, SubgraphRAG consistently outperforms EtD and performs better or comparable with respect to GNN-RAG (Table 3). As GNN-RAG adopts the strategy of RoG in fine-tuning LLM reasoners, we again perform the studies controlling the LLM reasoner as in the response to Q2. In this more controlled setting, our approach consistently outperforms GNN-RAG.

Llama 3.1-8B + WebQSP

Llama 3.1-8B + CWQ

GPT-4o-mini + WebQSP

GPT-4o-mini + CWQ

Thank you to the authors for their detailed response and for the effort put into addressing the reviewers' comments. I also noticed that the manuscript has been updated (e.g., Figure 3 appears to be new). To help reviewers track changes, I would recommend that the authors document the modifications between the two versions. That said, I still have several concerns regarding the following aspects:

Evaluation metrics (W2): The authors propose three retrieval metrics: path recall, GPT-4 triplet recall, and answer recall. However, there seems to be a mismatch in how these metrics are applied to the baseline methods (e.g., RoG in Figure 3). For example, RoG and GNN-RAG achieve good answer recall, but their GPT-4 triplet recall is worse than GraphSAGE/cosine similarity. These inconsistencies may suggest that certain metrics favor specific methods and I think the authors may need to evaluation retrieval precision (i.e., how many retrieved triplets are irrelevant).

Moreover, based on my review of Tables 1, 2, and Figure 3, it appears that Tables 1 and 2 report top-100 performance for SubgraphRAG, while the competing baselines (GNN-RAG, RoG) are evaluated at top-50. This discrepancy makes SubgraphRAG's performance seem inflated. If this observation is correct, I recommend aligning the evaluation, e.g., by reporting top-10 performance across all methods for a fairer comparison.

Regarding Table 1, I also recommend clarifying how time overhead is reported. The authors mention precomputing node/relation embeddings (Lines 361-364), which could result in lower time overhead of SubgraphRAG, but with higher memory overhead. Moreover, the downstream LLM time response is not reported, and SubgraphRAG seems to use a larger input context of top-100. I suggest discussing the trade-offs between time and memory usage for SubgraphRAG, and how SubgraphRAG's retrieved context impacts LLM response times.

Additionally, for the new downstream results, I would appreciate clarification on whether SubgraphRAG uses top-100 triplets and what the average/median input size for RoG and GNN-RAG is.

Scientific soundness (W1): It remains unclear why the proposed triplet scoring method outperforms more advanced methods like GNNs (the explanation in Line 215 is vague). For example, as I understand it, GNNs are capable of traversing deeper into the graph to retrieve multi-hop answers, even when considering only the top-1 retrieved nodes. In contrast, SubgraphRAG is limited to retrieving only 1-hop triplets at the top position, which may restrict its performance on multi-hop questions. I would recommend that the authors clarify in which specific scenarios SubgraphRAG is more effective or advantageous compared to competing methods, such as GNNs.

Once again, I appreciate the authors' thorough work. However, I am unable to increase my score at this time due to the concerns outlined above. I recommend that the authors refine the evaluation to ensure fairness and identify specific use cases where SubgraphRAG excels over more advanced baselines.

Q3: Figure 3 appears to be new. Based on my review of Tables 1, 2, and Figure 3, it appears that Tables 1 and 2 report top-100 performance for SubgraphRAG, while the competing baselines (GNN-RAG, RoG) are evaluated at top-50. This discrepancy makes SubgraphRAG's performance seem inflated. If this observation is correct, I recommend aligning the evaluation, e.g., by reporting top-10 performance across all methods for a fairer comparison.

Response: We appreciate the reviewer's attention to evaluation details. However, the different retrieval sizes do not inflate SubgraphRAG's performance - they reflect the optimal settings determined in the original papers. The baseline results for RoG and GNN-RAG are reported using their officially recommended configurations. These recommended settings are often chosen based on optimal downstream evaluation performance. See for example Figure 3 in the RoG paper. Similarly, we determined top-100 as optimal for SubgraphRAG through validation with Llama 3.1-8B, the smallest language model in our evaluation suite.

Figures 3 and 5 originally only study the retrieval performance (recall @ K) of various SubgraphRAG variants across various retrieval sizes. We've updated it to include the baseline results. The results show that SubgraphRAG consistently outperforms the baseline retrievers when evaluated under identical retrieval size. Artificially constraining well-performing methods to use suboptimal settings would not provide meaningful insights about their actual effectiveness for downstream LLM reasoning tasks.

Q4: Regarding Table 1, I also recommend clarifying how time overhead is reported. The authors mention precomputing node/relation embeddings (Lines 361-364), which could result in lower time overhead of SubgraphRAG, but with higher memory overhead. Moreover, the downstream LLM time response is not reported, and SubgraphRAG seems to use a larger input context of top-100. I suggest discussing the trade-offs between time and memory usage for SubgraphRAG, and how SubgraphRAG's retrieved context impacts LLM response times.

Response: Thank you for carefully checking the evaluation settings. We agree that time and memory trade-offs are critical considerations for deploying KG-based RAG systems in real-world scenarios. However, we would like to clarify that embedding pre-computation is not unique to SubgraphRAG – it is a common requirement shared across most baselines in our evaluation, including cosine similarity, G-Retriever, and GNN-RAG. To ensure fair comparison, we consistently excluded embedding pre-computation time across all these approaches. We’ve updated Lines 357-361 to provide a more precise description of wall clock time measurements and include a discussion of the time-memory trade-offs.

Regarding the impact of context size on LLM response time, empirical studies demonstrate that the overall question-answering latency is hierarchically influenced by: (1) the number of LLM calls (primary factor), (2) output token count (secondary factor), and (3) input token count (tertiary factor). For LLMs like GPT-4 and Claude-3.5, an additional input token typically adds about $10^{-2}$ the latency of an extra output token and $10^{-4}$ that of an additional LLM call [1]. While SubgraphRAG may have varying input token counts, it maintains efficiency by requiring only a single LLM call per question, in contrast to baselines that necessitate multiple calls like RoG and ToG.

[1] Vivek. Understanding input/output tokens vs latency tradeoff.

Q5: For the new downstream results, I would appreciate clarification on whether SubgraphRAG uses top-100 triplets and what the average/median input size for RoG and GNN-RAG is.

Response: The average input size for RoG and GNN-RAG are visualized in Figures 3 and 5 while SubgraphRAG employs top-100. To thoroughly investigate retrieval size effects, we conducted experiments by varying the beam search count for RoG, which directly influences the number of retrieved paths and triples. Our controlled comparisons between RoG's retriever and our approach, using identical LLM reasoners (Llama 3.1-8B and GPT-4-mini) and equivalent numbers of retrieved triples, demonstrate that SubgraphRAG's superior retrieval mechanism translates directly into enhanced downstream reasoning performance, as evidenced in Figure 6.

Thank you for reading our responses and providing detailed feedback!

Q1: It remains unclear why the proposed triple scoring method outperforms more advanced methods like GNNs (the explanation in Line 215 is vague). For example, as I understand it, GNNs are capable of traversing deeper into the graph to retrieve multi-hop answers, even when considering only the top-1 retrieved nodes. In contrast, SubgraphRAG is limited to retrieving only 1-hop triples at the top position, which may restrict its performance on multi-hop questions. I would recommend that the authors clarify in which specific scenarios SubgraphRAG is more effective or advantageous compared to competing methods, such as GNNs.

Response: We appreciate this important question comparing SubgraphRAG's multi-hop capabilities to GNNs. SubgraphRAG’s effectiveness in multi-hop retrieval is fundamentally enabled by our Directional Distance Encoding (DDE) mechanism detailed in Section 3.1. Retrieval for multi-hop reasoning fundamentally requires capturing both distance relative to variable-sized topic entities and semantic information. While GNNs are effective for various scenarios, they face theoretical limitations in capturing distance information, as shown in seminal GNN theory studies [1, 2]. Distance encoding has emerged as an effective solution to mitigate this issue by labeling nodes with binary features and performing multiple rounds of training-free message passing. Our DDE mechanism extends this idea specifically for KGQA by labeling topic entities, enabling efficient approximate computation of relative distances both from and towards variable-sized topic entities. When combined with semantic text embeddings, this approach proves particularly effective for triple scoring. Our ablation studies with respect to retrieval size (# triples) and question hop in Table 2, Figure 3 and Figure 5 demonstrate the effectiveness of DDE.

We acknowledge that GNN research has a rich history spanning at least eight years since the introduction of GCNs [3], yet their application to KG-based RAG remains relatively unexplored. We position SubgraphRAG as a valuable GNN-inspired baseline to catalyze further research in both GNN-based and GNN-inspired approaches.

[1] Li et al. Distance Encoding: Design Provably More Powerful Neural Networks for Graph Representation Learning

[2] Zhang et al. Labeling Trick: A Theory of Using Graph Neural Networks for Multi-Node Representation Learning

[3] Kipf & Welling. Semi-Supervised Classification with Graph Convolutional Networks

Q2: Why do RoG and GNN-RAG fall short for GPT-4o triple recall despite their good performance for path and answer entity recall (Figures 3 and 5)? Notably, their performances are worse than the GraphSAGE variant of SubgraphRAG and cosine similarity.

Response: Thank you for this insightful question. The performance gap you’ve identified reveals a fundamental limitation in path-based retrieval approaches like RoG and GNN-RAG while highlighting the advantages of SubgraphRAG’s flexible subgraph retrieval. Specifically, RoG fine-tunes an LLM to extract shortest paths between topic entities and answer entities, while GNN-RAG trains GNNs to predict answer entities and extracts shortest paths from topic entities to predicted answer entities.

The core challenge in KG-based RAG lies in retrieving sufficient subgraph evidence for LLM reasoning, which requires going beyond retrieving candidate answer entities or specialized subgraphs like shortest paths. However, ground truth subgraphs are rarely available in practice. As a workaround, existing approaches like RoG use shortest paths as weak supervision signals for training retrievers. Recognizing the limitations of shortest paths in representing complete evidence subgraphs, we leverage GPT-4o to label relevant subgraphs for more comprehensive retrieval evaluation. While this labeling approach may not be perfect, it is currently the most viable option.

The observed performance discrepancies between shortest path/answer entity retrieval and GPT-4o triple retrieval for RoG and GNN-RAG empirically validate the inherent limitations of path-based approaches. In contrast, SubgraphRAG variants and the cosine similarity baseline perform general triple retrieval and are not restricted to paths. For fair comparison, we train SubgraphRAG variants with shortest path triples like RoG, but our retriever mechanism enables them to generalize beyond the specialized paths, resulting in superior performance in retrieving GPT-4o triples.

Thank you for carefully reading our manuscript and providing detailed constructive feedback. While we address your main questions below, we are actively working on the additional baseline comparisons you suggested.

Q1: How is the evaluation of SubgraphRAG with respect to the baselines affected by the retrieval size (# triples)?

Response: Thank you for the insightful question. While SubgraphRAG may utilize more triples in certain settings, our extensive experiments demonstrate that the performance improvements stem from substantial improvement in retrieval.

Our analysis in Figures 3 and 5 presents retrieval recall values across varying numbers of retrieved triples. The results show that SubgraphRAG consistently outperforms baseline retrievers when operating under identical retrieval budgets. Importantly, SubgraphRAG demonstrates robust scaling properties: as we incorporate more top-ranked triples, the recall improvements remain substantial. This characteristic is particularly valuable for accommodating the varying long-context reasoning capabilities of different LLMs.

For reasoning evaluation, we investigate the effects of adjusting beam search count for RoG [1], a competitive and reproducible baseline. The original RoG paper has already examined this aspect with its fine-tuned LLM, demonstrating optimal reasoning performance at their chosen beam search count (Figure 3, RoG paper). Our additional comparisons between the RoG retriever and our retriever, using identical LLM reasoners (Llama 3.1-8B and GPT-4o-mini) and the same numbers of retrived triples, reveal that SubgraphRAG’s retriever maintains superior performance even when controlling the retrieval size (Figure 6).

Beyond the empirical studies, a fundamental design principle of this work is selecting a suitable number of triples that the LLM reasoner can process effectively, where suitability here is defined based on the reasoning capabilities of the specific LLMs. Notably, prior graph-based RAG methods, to the best of our knowledge, all overlook this crucial dimension.

In our study, LLMs handle the slightly larger numbers of triples in Table 3 without difficulty. In fact, the larger numbers of triples reported in Table 3 are still below the maximum that certain LLMs can process. As shown in Figure 4, LLama-3.1 8B performs optimally with around 100–200 triples, while GPT-4o-mini continues to improve with up to 500 triples.

This insight reflects the key contribution of our work: emphasizing the retriver efficiency which enables the size-adjustable retrieval of a suitable (potentially larger) number of relevant triples via a lightweight model while leveraging the strong reasoning capabilities of LLMs to digest all those retrieved triples. This principle is supported by the results presented in Table 3, Figure 4, and the retriever latency results in Table 1.

Q2: How well does SubgraphRAG perform when compared to GNN-RAG?

Response: For retrieval evaluation, our retriever consistently outperforms GNN-RAG (Table 1, Figure 3, and Figure 5). Notably, while our retriever is not explicitly trained to predict answer entities as in GNN-RAG, it outperforms GNN-RAG in answer entity recall using equivalent retrieval budgets (Figure 3 and 5). In terms of efficiency, our retriever is an order of magnitude faster than GNN-RAG (Table 1). For end-to-end KGQA evaluation, SubgraphRAG performs better or comparable with respect to GNN-RAG (Table 3).

We have conducted additional ablation studies that isolate the retriever’s impact by controlling the LLM reasoner. Our experiments employ Llama 3.1-8B and more powerful GPT-4o-mini as fixed LLM reasoners while varying only the retrieval method. The results below demonstrate that SubgraphRAG consistently outperforms GNN-RAG across both LLM reasoners and datasets.

Llama 3.1-8B + WebQSP

Llama 3.1-8B + CWQ

GPT-4o-mini + WebQSP

GPT-4o-mini + CWQ

Overall, the empirical studies demonstrate the surprising effectiveness and efficiency of our approach when compared to dedicated GNN-based approaches. The performance gains may be attributed to fundamental limitations of GNNs, including limited representation power for structural information [2] and degradation of pre-trained text embeddings due to over-smoothing [3] and over-squashing [4].

We’ve updated our manuscript to properly include all the above results and properly cited ReaRev, the GNN architecture employed by GNN-RAG.

Q4: How well does SubgraphRAG perform when compared to SR [1] and UniKGQA [2]?

Response: We conducted a thorough comparison with SR + NSM w E2E, which was identified in [1] as one of the strongest performing variants across both datasets. We have updated our manuscript to reflect these new evaluation results.

Regarding reproducibility, we successfully executed comparisons using the official codebase for the WebQSP dataset. However, we were not able to run the codebase for the CWQ dataset, an issue that has been reported and remained open in SR's official GitHub repo after more than two years (https://github.com/RUCKBReasoning/SubgraphRetrievalKBQA/issues/6). As a result, for CWQ, we directly referenced the KGQA performance reported in the original paper.

For retrieval evaluation, SubgraphRAG achieves superior performance across both effectiveness and efficiency, as evidenced in Tables 1 and 2. This performance advantage persists when controlling for retrieval size, as illustrated in Figure 5. Furthermore, in terms of end-to-end KGQA performance, SubgraphRAG consistently outperforms SR + NSM w E2E across all evaluated metrics.

For UniKGQA, we were not able to run the official codebase due to various issues. For example, the link to the preprocessed data and KG dump in the README file is simply empty: https://github.com/JBoRu/UniKGQA. Multiple unresolved GitHub issues further document these accessibility problems. Given these constraints, we have included UniKGQA's originally reported KGQA performance metrics in Table 3, which still demonstrate SubgraphRAG's superior performance.

[1] Zhang, Jing, et al. "Subgraph Retrieval Enhanced Model for Multi-hop Knowledge Base Question Answering." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

[2] Jiang, Jinhao et al. "UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph." ICLR 2023.

Dear Reviewer Yzun,

As the discussion period ends soon, we would like to check whether our responses answer your questions. Following your suggestions, we have clarified the effects of retrieval size in our evaluations, incorporated three new baselines (GNN-RAG, SR, UniKGQA), and highlighted our contributions when compared to a previous work. Thank you again for your comments and suggestions to improve our paper, and we look forward to your reply.

Best, Authors

Thank you for carefully reading our manuscript and providing detailed and valuable feedback. Below we respond to your questions and concerns.

Q1: How well does the SubgraphRAG retriever perform when compared to retrievers in ChatKBQA [1] and Interactive-KBQA [2]?

Response: Thank you for suggesting these additional retrieval baselines to help us strengthen our work. We have already updated our manuscript to cite these works. We fully agree that experimental comparisons with these approaches would provide valuable insights. We are actively conducting these additional experiments and commit to updating our results with these comparisons before the end of the rebuttal period.

Q2: Are the experiment results of the baselines reproduced a second time or referenced from the original text? In particular, it seems that the referenced StructGPT missed many tests, while the results of baseline methods like RoG should have been reproduced by the authors themselves.

Response: We appreciate the reviewer's attention to our experiment settings. Our baseline results were obtained through a combination of reproduction and citation of original results. For retrieval evaluations, we provide comprehensive details in Appendix B.1. For reasoning evaluations, we include discussions in the “Experiment Settings” in Sec. 4.2. For baseline methods such as RoG, we managed to fully reproduce their results using the official pre-trained models and codebase.

Regarding StructGPT, the selective inclusion was based on several practical considerations. First, WebQSP was the only dataset in our study for which StructGPT reported results. Second, the authors provided processed files specifically for WebQSP, facilitating accurate comparison. Third, reproducing StructGPT's results for other datasets proved challenging, as evidenced by numerous unresolved reproduction issues in their official GitHub repository. Finally, given that StructGPT no longer represents the state-of-the-art performance in this domain, we strategically focus our reproduction efforts on more recent and competitive baseline methods.

Q3: Can you clarify why the results of ToG are not reproducible? The calculation of ToG's Hits@1 is actually based on ToG only finding the first match in the answer text. However, this is not the reason for irreproducibility. Interactive-KBQA has reproduced results.

Response: We attempted to run the official ToG codebase by following their README instructions, but we were unable to reproduce the results. While we have full confidence in the authors' scientific integrity, additional documentation and guidance would enhance the reproducibility of their work.

Regarding the ToG results cited in the Interactive-KBQA paper [2], actually, their reproduced ToG results also differ substantially from those reported in the ToG paper. Specifically, ToG claims to be better than StructGPT by 10%, while the reproduced results from Interactive-KBQA show that ToG is worse than StructGPT by 8%. Therefore, the results of ToG from Interactive-KBQA essentially align with our observation that ToG’s reported results may not be easily reproducible given its released codebase.

Thank you for pointing out that our original footnote caused confusion between metric choice and irreproducibility. We have updated the footnote to enhance clarity and will appreciate your guidance for further refinement.

Q4: There are some ambiguous expressions that lack sufficient mathematical definitions or explanations. 1) What is “scalable” and how to prove whether the method is “scalable” or not? 2) What constitutes suitable prompting?

Response: Thank you for these points of clarification. Regarding scalability, we consider both algorithmic time complexity and empirical wall clock time efficiency. Our method exhibits linear complexity in relation to the graph size/number of candidate triples. Empirically, it achieves improvements in wall clock time by orders of magnitude compared to the baselines (Table 1). Regarding suitable prompting, we investigated various prompting strategies and selected the optimal one based on empirical performance, with detailed results presented in Figure 2 and Appendix D.

Q5: There are some typos, formatting issues, and grammar errors.

Response: Thank you for the great efforts in proofreading our manuscript. We have revised the manuscript to fix the issues you explicitly pointed out. We will make a more careful pass of the manuscript after the rebuttal period for possible other issues.

[1] Luo et al. ChatKBQA: A Generate-then-Retrieve Framework for Knowledge Base Question Answering with Fine-tuned Large Language Models. ACL (Findings) 2024.

[2] Xiong et al. Interactive-KBQA: Multi-Turn Interactions for Knowledge Base Question Answering with Large Language Models. ACL 2024.

Thank you for reading our previous responses and recognizing the breadth of our experiments! We are glad that we’ve addressed your previous questions (e.g., the non-reproducibility of the ToG codebase) and have the opportunity to address your new questions.

First, we have to say that whether a methodology is considered 'captivating' can be subjective. We find greater appeal in methods that are simple yet effective, rather than overly complex. That said, we prefer to prioritize objective evaluation results as a measure of a method's merit.

Q6: With lightweight subgraph retrieval, does SubgraphRAG sacrifice explainable reasoning when compared to approaches like RoG and ToG. In particular, achieving explainability often introduces additional reasoning costs for LLMs.

Response: Thank you for raising this important consideration. Actually, our manuscript has already demonstrated SubgraphRAG’s explainability and reasoning capabilities in Section 4.2. Following the evaluation in the RoG paper, we have provided nine detailed explainable reasoning examples from WebQSP and CWQ in Appendix E, which demonstrates the effectiveness of SubgraphRAG in providing explanations along with answer predictions. We understand that these explainability studies might have been overlooked given our broad experiment results, and we appreciate the opportunity to highlight them further.

In contrast, RoG’s LLM fine-tuning approach potentially compromises the inherent explainable reasoning capabilities of LLMs, as explained in page 18, A.4 of the RoG paper (https://arxiv.org/abs/2310.01061). To compensate for this lost capability, RoG employs ChatGPT to label explanations for 2000 samples as supplementary fine-tuning data.

Once again, our above comparisons are based on objective evidence rather than subjective impressions 'This paper appears to sacrifice that aspect'

Q4: SubgraphRAG presents triples to LLMs in structured form, i.e., (h, r, t), rather than transforming them into natural language descriptions, which misses an opportunity to leverage LLMs’ strengths in natural language understanding and reasoning. Converting triples to natural language may allow LLMs to more comprehensively exploit the triple information and achieve more accurate reasoning. Should triples be directly converted to natural language for LLMs to better understand and utilize them?

Response: Thank you for the highly insightful suggestions. We have conducted empirical investigations comparing both approaches - structured triples versus natural language descriptions. Our experiments revealed no clear performance differences between the two representations. We suspect that structured triple representations post less severe challenges for long-context reasoning because they are more compact in terms of the number of occupied tokens. The compact nature of structured triples allows more information to be processed within the same context window compared to their natural language counterparts. Moreover, converting triples to natural language descriptions introduces additional computational overhead during LLM inference. These considerations led us to adopt structured representations in our approach, aligning with several contemporary graph-based RAG approaches [5, 6, 7], despite that some prior work has explored converting triples into natural language descriptions [8].

Q5: Experiments are conducted solely on WebQSP and CWQ datasets. How does SubgraphRAG perform on other datasets like HotpotQA?

Response: Thank you for this important question. We selected WebQSP and CWQ as our primary evaluation datasets because they represent the current standard benchmarks in KG-based RAG research, allowing direct comparisons with existing methods. While we acknowledge that evaluating on additional datasets would strengthen our findings, our approach is fundamentally dataset-agnostic and can be applied to various knowledge graph question answering scenarios. For instance, CommonsenseQA [9] and OpenBookQA [10], which utilize the ConceptNet knowledge graph [11] and also allow extracting topic entities from questions, present natural extension opportunities for SubgraphRAG. Regarding HotpotQA [12], though it is not inherently a KGQA dataset, efforts like [13] have demonstrated successful transformation of text knowledge base (Wikipedia) into structured graph representations. Such graph-based transformations of Wikipedia articles make HotpotQA compatible with our approach, where paragraphs relevant to a question are essentially topic entities. We agree that expanding our evaluation to these datasets would be valuable for future work.

[1] Luo et al. Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning.

[2] Sun et al. Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph.

[3] Leng et al. Long Context RAG Performance of Large Language Models.

[4] Liu et al. Lost in the Middle: How Language Models Use Long Contexts.

[5] Wu et al. Retrieve-Rewrite-Answer: A KG-to-Text Enhanced LLMs Framework for Knowledge Graph Question Answering.

[6] Luo et al. Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning.

[7] He et al. G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering.

[8] Li et al. Graph Reasoning for Question Answering with Triplet Retrieval.

[9] Talmor et al. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge.

[10] Mihaylov et al. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering.

[11] Speer et al. ConceptNet 5.5: An Open Multilingual Graph of General Knowledge.

[12] Yang et al. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering.

[13] Asai et al. Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering.

Thank you for your insightful and positive feedback! Below we respond to your questions and concerns.

Q1: Although the paper details the algorithmic flow of SubgraphRAG, it lacks a full real example from query to retrieval and reasoning to answer generation. Could you provide a complete example for a typical concrete query from WebQSP or CWQ?

Response: Thank you for helping us further improve the presentation. We’ve included examples for multiple real samples from WebQSP and CWQ in Appendix E, covering the corresponding question, retrieved triples, LLM responses, and ground truth answers.

Q2: SubgraphRAG employs an MLP with pre-trained text embeddings and Directional Distance Encoding (DDE) for subgraph retrieval. As LLMs may inherently excel in processing complex semantic and relational information, could LLM-driven retrieval better identify information required for multi-hop reasoning?

Response: The direct use of LLMs for retrieval, while theoretically promising, faces significant practical limitations. As demonstrated in Table 1, a single call of even a small 7B LLM (RoG [1]) is two orders of magnitude more costly than our retriever in terms of wall clock time efficiency. This efficiency gap widens further when considering the limited context window sizes of LLMs, which necessitate multiple calls – for instance, ToG [2] requires three separate calls for its iterative graph exploration approach.

The challenges extend beyond mere computational efficiency. Even in cases where the context window size is sufficient, LLMs can still struggle to reason over a long context [3, 4]. Our ablation studies (Figure 4) provide empirical evidence that including more than 100 triples actually deteriorates the reasoning performance of Llama 3.1-8B.

We acknowledge that complex reasoning scenarios might benefit from leveraging LLM capabilities through multiple rounds of retrieval and reasoning. This could be achieved by decomposing queries into smaller subqueries and applying our retriever to each component separately, thereby maintaining the efficiency advantages of our approach while potentially accessing the sophisticated reasoning capabilities of LLMs in a more controlled manner. This is beyond the scope of this work and we are actually currently working on this.

Q3: The mechanism of DDE heavily relies on the assumption of high-quality topic entities. In practice, the extracted topic entities can be noisy or simply missing for ambiguous and polysemous queries. Under such circumstances, will the topic entity bias lead to misleading retrieval results? Is there a more generalized retrieval strategy available?

Response: We greatly appreciate your insightful observation regarding DDE’s reliance on topic entities. While our current implementation demonstrates strong performance with high-quality topic entities, we acknowledge the challenges posed by ambiguous and polysemous queries. To address these challenges, we propose extending our approach through two complementary strategies.

First, we can employ LLMs as a preprocessing step for query refinement and disambiguation prior to topic entity extraction. For instance, given an ambiguous query like "Tell me about Python's applications", the LLM can generate distinct interpretations: "What are the applications of Python, the programming language?" and "What are the characteristics of pythons, the snake species?". This disambiguation step allows our system to extract precise topic entities (Python [programming language] and Python [snake genus]) and provide accurate retrieval results for both interpretations.

Second, when direct topic entity extraction proves challenging, we propose leveraging LLMs to generate multiple candidate topic entities with associated confidence scores. The retrieval results can then be aggregated across these candidates, weighted by their confidence scores, to provide more robust results. This approach maintains the benefits of entity-guided retrieval while being more resilient to noise in entity extraction.

It's worth noting that this challenge is not unique to DDE – to the best of our knowledge most existing KGQA and KG-based RAG approaches, including our baselines Retrieve-Rewrite-Answer [5], RoG [6], and G-Retriever [7], similarly rely on extracted topic entities. Overall, the proposed extensions advance toward more robust entity-guided retrieval while preserving DDE's core advantages.