GIVE: Structured Reasoning of Large Language Models with Knowledge Graph Inspired Veracity Extrapolation

We understand the reviewer's concerns about the role of expert knowledge in the reasoning process, as well as in the answer-generation process, and we appreciate your suggestion on our literature discussion and experiments. We hope the following clarifications addressed all the questions and we will make sure to include these discussions in our revised manuscript.

For Claims And Evidence paragraph 1:

We apologize for the confusion. To clarify, GIVE enriches the expert knowledge in the KG by extrapolating expert knowledge towards relevant concepts akin to the queried ones, as detailed in Section 3.4.3. Direct injection of expert knowledge into the question-answering is $**not**$ assumed, due to the uncertain quality of the accessible knowledge base. Figure 4 shows that the performance of GIVE improves with an increased ratio of expert knowledge to total knowledge (expert plus extrapolated). Higher expert knowledge ratio ensures that the neighborhood related to the query is well-connected in the KG, offering substantial evidence for veracity extrapolation. Lower ratio indicates reliance on internal knowledge, as discussed in Section 3.3 and Open Relations in Section 3.4.3. GIVE performs well with ample expert evidence guiding reasoning, quantitatively when expert knowledge ratio reaches about 20%, $**not**$ when injecting solely 20% of this knowledge. All ablation studies in Section 4.7 utilize GPT3.5-turbo, with other details consistent with Section 4.2.

For Claims and Evidence Paragraph 2:

Commonsense reasoning experiments aim to test GIVE's generalizability in three ways: (1) on noisy, large-scale KG, (2) varying sparsity levels, and (3) tasks where pre-trained knowledge is extensive. This setting presents challenges, as the basic LLM already achieves about 70% accuracy; pre-training on commonsense knowledge is inherently easier than on scientific information. In such cases, misinformation can lead to hallucination. GIVE improves performance by up to 3.4% and 4.9% over RAG and ToG, and consistently outperforms base LLM, indicating its reasoning process avoids hallucinations. ConceptNet is indeed important, the challenge is how to wisely use its information to further enhance the rich pre-training knowledge without causing hallucination, especially with its sparse versions.

Including a 0% scenario, as the reviewer suggested, offers more insight: GIVE_a and GIVE_a+c achieve 69.84% and 69.36%, respectively, due to the exclusion of expert information. Without expert information, the knowledge provided by GIVE is derived solely from the LLM's internal knowledge (Sections 3.3 and Open Relations in 3.4.3). This aligns with our observation in Figure 4, where the expert knowledge ratio is 0. The better performance in this case is attributable to richer pre-training in commonsense compared to biomedical knowledge.

For Methods and Evaluation Criteria:

We understand the reviewer's concern about the role of expert knowledge. GIVE is designed for situations where a high-quality KG is inaccessible in domain-specific tasks. In such scenarios, GIVE does $**not**$ intend to utilize expert knowledge directly for question answering but rather as an "inspiration," illustrated by the transition from solid to dashed lines in Figure 2. The KG links some entities unrelated to the query; GIVE encourages the model to assess whether similar connections exist among other related entity pairs. In Tables 3 and 4, GIVE_a+c+e generally demonstrates improved or equivalent performance. The minor margin does not imply that expert information is trivial, but rather reflects that they are not directly solving the query. Expert knowledge (solid lines in Figure 2) is pivotal for the success of GIVE_a and GIVE_a+c, as it contributes to knowledge extrapolated from expert connections among relevant entities.

For Essential References Not Discussed:

We will include discussion of CoN in our revised manuscript. GIVE and CoN differ fundamentally in several aspects: (1) Motivation: GIVE is a reasoning framework that formulates a faithful thinking process by populating the expert KG triplets towards the query, whereas CoN is a robust retrieval system excluding similar yet irrelevant documents. (2) Use of internal knowledge: GIVE employs LLM's internal knowledge for "veracity extrapolation," depicted by the transition from solid to dashed lines in Figure 2, whereas CoN uses it to create document summaries (notes) for accurate relevance analysis. (4) Use of external knowledge: All expert knowledge in GIVE is integral to its reasoning process, as shown by the solid lines in Figure 2. Meanwhile, CoN filters out irrelevant documents for question-answering. (5) GIVE is designed to reason $**beyond**$ the accessible KG, whereas CoN identifies and focuses on documents containing the essential context.

For Other Comments Or Suggestions:

We will include all connections in the Figure for our revised manuscript.

In-depth analysis of token consumption

We understand the reviewer's concern about the token efficiency and provide a comparison of token consumption between GIVE and ToG on 100 random questions from each dataset. For each question, we calculate the total number of input/output token in the whole problem-solving process. We use tiktoken with GPT3.5-turbo for this comparison. For GIVE we use n=1, and for ToG, we use D=5, the setting is the same as Table 8 in the Appendix.

The observation supports the conclusion of Table 8, where GIVE_n=1 effectively balances efficiency and accuracy in challenging scientific reasoning tasks. Specifically, in 5 out of 6 datasets, GIVE consumes around 10% more input tokens than ToG, generates just 80 and 20 more tokens for PubmedQA and BioASQ, respectively, while reaching a 3-fold and 5-fold increase in accuracy. The variance in token usage across datasets is due to the differing numbers of entity groups and numbers of candidate relations, leading to varying candidate connections, as depicted in Figure 6 of the appendix. GIVE also demonstrates strong generalizability in deployment on KGs with varying sizes and sparsities.

Discussion of KAG and HOLMES

Thank you for pointing out the related works; we will add a discussion of KAG and HOLMES in our revised manuscript. Although we acknowledge that KAG and HOLMES leverage KG to enhance LLM output, we respectfully contend that they are solving fundamentally different problems with GIVE.

KAG and HOLMES are advanced $**retrieval**$ systems that retrieve and integrate accurate expert knowledge in question-answering tasks. In contrast, GIVE, as a $**reasoning**$ framework, uses irrelevant information pieces to promote divergent thinking and extrapolates knowledge to generate answers. Thus, KAG and HOLMES concentrate on extracting quality KG from corpus or documents to aid precise information matching via a refined SemanticGraph or Graph Schema. They use KG for more accurate information matching. Here, the LLM iteratively synthesizes the responses as an answer generator. GIVE, however, guides LLMs in problem-solving like a human expert, foregoing preprocessing irrelevant knowledge by using the structured nature of KGs to seamlessly connect expert information with a query. KG data (entities and their connections) support the construction of a reasoning chain. LLM solves queries by verifying the inferred links suggested by KG, as illustrated in Figure 2. Although we were committed to including them as baselines, as suggested by the reviewer, HOLMES currently lacks an open-source implementation, and the public KAG code lacks the kag-model, rendering them unusable in our experimental settings with query and KG as inputs. We believe that these methods will perform similarly to GraphRAG, emphasizing information retrieval and injecting over reasoning.

Discussion of sensitivity to retrieved knowledge

We appreciate your insightful discussion. To clarify, the experiments in Figure 4 consistently show that GIVE effectively uses expert knowledge not directly related to the question to create an accurate reasoning process. This is evidenced by an improvement in performance from n = 0 to n = 1 and improved results with a higher ratio of expert knowledge. Furthermore, with the same expert knowledge ratio, BioASQ performs better as its questions are more easily connected to expert knowledge. The veracity extrapolation process links expert knowledge to the query by the extrapolated connections; a lower expert knowledge ratio means fewer such "bridges" for the model to leverage the unrelated KG information. We will incorporate this discussion into our revised manuscript.

Compared to RAG that can retrieve any text information, graph-based retrievers (also for GraphRAT, etc.) naturally require an external knowledge base graph that is of high quality and is typically not yet learned by the LLMs. This might not be a `weakness’ but the common limitation of graph-based retrievers.

We appreciate the recognition of our contributions and agree with the common limitation of graph-based retrievers as mentioned by the reviewer. GIVE plays a pivotal role in addressing the challenge where high-quality Knowledge Graphs (KG) are not readily available. Knowledge Graphs possess certain distinct advantages over textual corpora. The primary goal of GIVE is to bridge the gap between the limited parametric knowledge typically available and the accurate reasoning required for scientific queries by effectively populating limited expert information towards the query. Our choice of KGs as a data source is due to their structured relational knowledge, which naturally provides a reasoning pathway. This choice simplifies the task to effectively creating connections between the expert KG knowledge with the ultimate question-answering task. In such scenarios, the process of constructing a relevant entity set and retrieving the connections between different groups proves efficient due to the inherent structure of graph data. Recent studies [1] have focused on converting text into KGs, and we are optimistic that textual corpora and KGs will become mutually interchangeable with advancements in this field of research.

[1] KGGEN: EXTRACTING KNOWLEDGE GRAPHS FROM PLAIN TEXT WITH LANGUAGE MODELS.

GIVE requires the preprocessing on the knowledge graph, the extraction of query information, and retrieval, which take some time. Some of the processes are before inference (like preprocessing), while others (extraction and retrieval) are during inference.

We appreciate the constructive feedback from the reviewer. We agree with the reviewer's insight that the reasoning process used by GIVE demands additional inference time, particularly in the construction of entity groups and the extrapolation of veracity. To substantiate our claims, we present comprehensive experiments in Table 8 in the appendix. These experiments demonstrate that when the number of entities per group (n) is set to 1, GIVE surpasses other competing baselines in performance, while maintaining similar or even lesser execution times. Notably, this trend is more significant when applying larger and denser Knowledge Graphs (KGs). Efficiency of GIVE could be further improved by incorporating batch pruning for the veracity extrapolation process. Moreover, as elaborated in Section 5, future research could involve automating the "veracity extrapolation" procedure, for instance, by developing a corresponding process reward that can be incorporated into Reinforcement Learning (RL) post-training frameworks. We are confident that the contributions of GIVE offer substantial benefits for future endeavors in this area of research.

The method introduces some additional hyperparameters like m and n, increasing the difficulty of adjusting hyperparameters.

We appreciate your effort in highlighting this issue. It is important to elaborate further that GIVE's only hyperparameter is the number of entities per group, denoted as $n$. In contrast, $m$ represents the number of entity groups which is a characteristic inherent to the query, as explained in Sections 3.1 and 3.2. For each queried entity, an entity group is created using its analogous concepts in the Knowledge Graph (KG). The findings illustrated in Figure 4 and Table 8 substantiate that GIVE offers commendable performance even with $n = 1$. This facilitates GIVE in achieving an optimal balance between efficiency and accuracy without the necessity of hyperparameter tuning. Moreover, experiments detailed in Appendix C.1 reveal that the average value of $m$ is typically 3 or 4 across all datasets.

Since I did not find the code link, I just want to ask the authors whether they would provide the corresponding repository to the public?

Thank you for raising this issue. We hold open-source research in high regard and are committed to including a link to the repository containing all the relevant codes and datasets in the revised manuscript.

From Figure 4 we see that the expert knowledge set plays an important role for GIVE. However, from Table 8, GIVE achieves better performance than ToG and RAG when using n=1 and only the affirmative knowledge set, why can GIVE achieve good performance without using the expert knowledge set?

We are grateful for the insightful discussion brought up by the reviewer. It is essential to further elucidate that GIVE employs the expert knowledge extracted from the Knowledge Graph (KG) with the purpose of enabling faithful reasoning through knowledge extrapolation, rather than being directly utilized for answer generation. As demonstrated in the experiments reflected in Figure 4, the findings indicate that the efficacy of GIVE's "veracity extrapolation" is intricately linked to the ratio of expert knowledge. This ratio evaluates the degree to which the KG is pertinent to the questions, encompassing both the entities and their interrelations. The reason for this relationship is that expert knowledge constitutes the KG's connections among the pertinent entities, as detailed in Section 3.2 and depicted by the solid lines in Figure 2 of our initial manuscript. Although this expert knowledge might not directly facilitate the answering of questions, as it is not immediately connected to the queries, it serves as a valuable source of "inspiration" for the model to execute divergent reasoning. This is a primary rationale behind proposing a framework that integrates retrieval with reasoning when dealing with external knowledge that is sparse and noisy.

In Table 3, why does CoT harm the performance of GPT4, as CoT does not introduce new knowledge which causes hallucination, as clarified by the authors.

We appreciate your question. In Section 4.4, we critically evaluate various approaches by assessing their performance on scientific benchmarks that demand intensive reasoning. Large language models (LLMs) lack sufficient parametric knowledge to effectively address these complex queries. This is primarily because it is impractical to extensively train these models on highly specialized domains. Our observations, derived from comparing the outcomes of GPT-3.5-turbo and GPT-4 on datasets such as PubMedQA and ProcessBank, illustrate that simply scaling up pre-training does not necessarily enhance accuracy. In such instances, the Chain-of-Thought (CoT) approach seeks to promote a logical step-by-step reasoning process that relies exclusively on the model's internal knowledge, thus results in fluctuating performance levels due to its inherent randomness.

In Figure 3, there is a huge performance gap between GIVE_a and GIVE_a+c, GIVE_a+c+e. Can the authors clarify the cause of this performance gap

Thank you for raising this question. Figure 3 illustrates our experiments with TruthfulQA, an open-domain text generation task. This dataset is included to demonstrate that GIVE is effective across both domain-specific and open-domain tasks. GIVE_a achieves superior results primarily because inquiries are directed at information accuracy. Consider the example query for TruthfulQA "What is the original meaning of SOS?" In this scenario, incorporating an affirmative knowledge set naturally leads to a more coherent context, whereas integrating counter-factual information tends to result in hallucinations. Additionally, as we employed the 10% version of ConceptNet in this experiment, the retrieved expert knowledge is not directly associated with the query. Instead, it is utilized to motivate the model towards "veracity extrapolation," which significantly contributes to the excellent performance of GIVE_a. This is elaborated upon in Section 4.6 (lines 332-338), and the solid to dashed line process in Figure 2 of our original manuscript.