Knowledge Graph Retrieval-Augmented Generation via GNN-Guided Prompting

Thank you for your valuable feedback and for highlighting both the strengths and weaknesses of our work. Your comments have been instrumental in guiding us to refine our approach and clarify key points in the paper. We have provided detailed responses below to address each of your concerns and ensure the clarity and robustness of our contributions.

The authors omit discussion and comparison with many relevant works (e.g., RoG [1])

We thank you for pointing out RoG as an important related work. In our experimental setup, we included Think-on-Graph (ToG), which shares a similar design philosophy with RoG by relying on LLMs to make retrieval decisions over KG paths without external scoring mechanisms. We chose ToG as a competitive representative of LLM-only retrieval-based reasoning frameworks, and GGR consistently outperforms ToG across all evaluated datasets and LLM variants. Additionally, we have reviewed the results reported in the RoG paper and find that GGR also achieves overall higher accuracy. This further supports the effectiveness of integrating GNN-based global scoring into the retrieval loop. While RoG explores task decomposition and relation prediction strategies, our work focuses on enhancing retrieval quality via graph-aware soft relevance signals. We will revise the paper to explicitly discuss RoG and clarify the methodological differences and performance comparisons.

GNN-based approaches have been widely adopted in KGQA tasks and recently extended within RAG-based inference frameworks (GNN-RAG [4]). However, the paper does not compare against these approaches or elaborate on how GCR captures information that previous GNN models cannot. I recommend expanding both the experimental section and related work sections to include these important comparisons. I speculate that the step-wise LLM reasoning might be more significant, and this approach could potentially be extended to work with existing frameworks in the literature (such as NSM, GNN-RAG). For example, when comparing with GNN performance numbers reported in NSM [3] (Table 2), those GNN implementations achieve better performance than the proposed GNN-Score. This suggests the possibility of directly combining those more powerful GNNs with the step-wise LLM reasoning approach presented here. Could we replace the proposed GNN-scoring with established KGQA-GNNs to further improve the performance in KGQA?

We thank you for this valuable suggestion. GGR is fundamentally different from traditional GNN-based KGQA models such as GNN-RAG, where the GNN is the main reasoning engine used for direct answer prediction or symbolic path traversal. In contrast, GGR treats the GNN as an auxiliary component. Its sole purpose is to assign soft, question-aware relevance scores to nodes and relations within the subgraph. These scores guide the selection of candidate triplets for step-wise reasoning, but do not determine the final answer. The actual reasoning and decision-making are entirely handled by the LLM, which integrates both retrieved facts and question semantics. This design makes the GNN a lightweight and interpretable guide, not a standalone predictor, and enables flexible integration with different LLMs without modifying their architecture or training. In response to your suggestion, we experimented with a stronger GNN architecture given by GNN-RAG, which includes better attention modules. While this variant did improve retrieval quality and downstream QA performance, the gains were moderate. This confirms that although a stronger GNN can offer additional benefits, its role in GGR remains supportive. It helps identify plausible paths, but the LLM is responsible for evaluating and composing reasoning chains. We will expand the experimental section to include these results and clarify the auxiliary role of the GNN throughout the paper.

We sincerely thank you for your thoughtful and thorough review of our submission. Your comments and suggestions have provided us with a fresh perspective on the paper and its contributions. We have carefully considered your feedback and made revisions to address your concerns, as outlined in our responses below.

One concern is about the quality and robustness of GNN training. According to ablation studies, the performance of GGR is highly dependent on the ability of GNN to generate accurate relevance scores. However, GNNs are trained based on path supervision, and the training data are simply the paths connecting query entities and answer entities. However, the generated paths might not be the golden reasoning paths, especially for those hub entities with large in&out degrees, which can hurts the confidence of the GNN scores. There's no discussions on the GNN training and evaluation details in the paper. The sensitivity of GNNs to the quality/completeness of training paths needs to be further explored.

We appreciate your concern about the quality of GNN training. Our supervision uses all paths (within a length limit) connecting query and answer entities, which may include noisy or non-optimal paths. However, this design is intentional: the GNN acts as a soft, structure-aware scorer rather than a final decision-maker. Its scores guide candidate selection, while the LLM performs the actual reasoning. This setup reduces reliance on perfect supervision and allows the LLM to override misleading signals. We agree that exploring path quality, filtering, and potential feedback loops from LLM outputs to GNN training are promising directions, and we will discuss these limitations and future extensions in the final version.

Scalability of subgraph extraction: For each query, the GGR framework needs to extract an N-hop subgraph. Although N-hop limits the scale of the subgraph, it may still consists of a huge amount of entities for some queries. Dynamically generating subgraphs and do GNN inference for every query introduces scalability challenges for extremely large or rapidly evolving KGs.

We thank you for raising this important scalability concern. To address the potential overhead of inference, our framework employs a GNN Score Filtering mechanism designed for efficiency. Specifically, we group the candidate triplets by the GNN scores to perform downstream reasoning and retrieval. This design avoids redundant computation and allows us to retain only high-relevance triplets for LLM prompting, significantly reducing the scale of processed subgraphs. We will clarify this optimization in the paper to better highlight how GGR supports scalable retrieval over large or dynamic KGs.

Limited evaluation datasets: The authors only conducts evaluations on WebQuestionsSP and ComplexWebQuestions, which are both based on Freebase. Evaluation on more diverse KGs with different schemas, densities, domains can better demonstrate the generalizability of the GRR framework. In addition, the questions in these 2 datasets can usually be answered by Freebase. This cannot effectively test the system's ability to identify unanswerable queries when the knowledge graph lacks the necessary reasoning evidence.

We thank you for the insightful suggestion. We selected WebQuestionsSP and ComplexWebQuestions because they are widely used benchmarks in KGQA, supporting evaluation of both simple and multi-hop reasoning over a well-defined KG (Freebase). We agree that broader evaluation on diverse KGs, such as those with different schemas (e.g., biomedical), varying densities, and domain-specific relation patterns, would better demonstrate the generalizability of GGR. We consider this an important direction for future work and will discuss it in the paper accordingly.

GNNs are trained using path supervision between question and answer entities. Could you elaborate on how to address this issue when the true path for certain types of questions is noisy, sparse, or unavailable? How much carefully curated path data is typically required to train a performant GNN in your framework?

We appreciate your concern about the reliability of path supervision. In our framework, the GNN is trained using all paths connecting the query and answer entities within a fixed hop limit, without manual curation or filtering. As a result, the supervision naturally includes noisy or suboptimal paths, which reflect real-world KG conditions. This is a deliberate design: the GNN provides soft structural relevance signals based on observed graph patterns, while the final reasoning is handled by the LLM. Our method does not require curated or golden paths, and remains robust even when path data is sparse or unavailable, falling back on LLM-only reasoning in such cases. We will clarify this point and highlight that GGR operates effectively without manually labeled reasoning paths.

Thank you for your detailed and constructive feedback on our paper. We appreciate the time and effort you have taken to review our work, and your insights have been invaluable in helping us identify areas for improvement. Below, we address your specific concerns.

Too narrow scope of related works. KGQA has been extensively studied in the literature. Beyond prompt augmentation as proposed in this paper, many studies have explored altering the behavior of LLMs by leveraging knowledge from KGs. Also, methods augmenting attention with extracted graphs and semantic parsing has already studied how to effectively capture multi-hop relation. This is addressed by more recent and advanced methods such as neural tree search, and generative subgraph retrieval. The authors should broaden the coverage of related works and compare the proposed method with these relevant works. Also, unified transformers have been studied to incorporate the extracted subgraphs or paths of knowledge graphs into the inference of LLMs.

Thank you for the suggestion to expand our coverage of related works. While prior efforts have explored enhancing KGQA through semantic parsing, attention-based reasoning, neural tree search, generative subgraph retrieval, and unified transformer architectures, our work addresses a distinct challenge: improving the assessment in the retrieval process itself in KG-RAG. Unlike methods that alter the LLM architecture or generate symbolic programs, our method focuses on stepwise triplet selection by integrating GNN-based global relevance scoring with LLM-based semantic alignment. This hybrid approach enables more accurate multi-hop reasoning by retaining globally important yet locally weak signals, while avoiding architectural modifications and maintaining compatibility with existing LLMs. We will revise the Related Work section to acknowledge these complementary directions while clarifying that our key contribution lies in enhancing retrieval quality through GNN-guided prompting, rather than modifying inference mechanisms or generating full reasoning paths.

Out-dated techniques. KG retrieval and GNN-based processing adopted somewhat outdated techniques and models. Many advanced GNNs have been adopted and Transformer-based architectures have been proposed specifically for KGQA benchmarks to more effectively handle multi-hop relations. These advanced methods have neither been compared as baselines nor adequately discussed in the related work section.

Thank you for pointing out the concern regarding the GNN architecture. While our initial implementation adopts a lightweight GNN to emphasize the role of graph-based retrieval in guiding LLM reasoning, our framework is modular and supports the integration of more advanced GNNs. To demonstrate this, we replaced the original GNN with the architecture from GNN-RAG, which incorporates better attention mechanisms. The updated results are shown below. As the table indicates, while the advanced GNN provides additional gains, the improvement is modest, reinforcing our design choice: the GNN serves as a retrieval assistant, and our key contribution lies in the retrieval framework that combines GNN scores and LLM-based semantic reasoning, rather than in the specific GNN variant. We will clarify this point and include these results in the revised version.

In the path supervision training step, how many paths are considered as the GT paths? Shorter paths are preferred?

Thank you for the question. In the path supervision step, we consider all valid paths in the retrieved subgraph that connect a query entity to a ground-truth answer entity, constrained by a maximum path length. All such paths below this threshold are included as ground-truth paths. While we do not explicitly prioritize shorter paths, the length limit implicitly favors them by excluding overly long or noisy paths that may be less useful for reasoning. This approach balances coverage and relevance, providing the GNN with diverse yet meaningful supervision signals for learning question-aware relevance.