Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sources

Dear reviewer TzUb,

Thank you for your recognition of our extensive experiments and analysis. We would like to address your queries and concerns here.

This paper bears a high resemblance to VE with the main distinction being the inclusion of multiple knowledge sources, which might be considered a relatively ordinary contribution.

The key contributions of our work are the following:

1. Querying diverse heterogeneous knowledge sources with appropriate queries and aggregating answers from those in an effective manner posits unique scientific challenges and were not addressed in prior work. To the best of our knowledge, we are the first to enhance the factual correctness of LLMs with heterogeneous knowledge sources.
2. To the best of our knowledge, we are the first to progressively correct the rationales, ensuring that inaccuracies from preceding rationales do not propagate into the subsequent steps. Prior studies such as VE and ReAct leave errors from prior steps in the prompt, causing potential noise.
3. We propose AQG, which is versatile and can seamlessly transition between fine-tuned models and black-box LLMs, enabling scalability of knowledge sources.
4. Extensive experiments prove the effectiveness of CoK across ranges of domains, including factual, medical, physical, and biological.

I think this training process may make it challenging for AQG to generate SPARQL queries for complex and compositional questions. Because it cannot guarantee that the sub-questions derived from the CoT decomposition are all simple.

We used an AQG trained with data whose logical granularity is on par with the CoT rationales. For example, for WikiData, both training data and CoT rationales only contain one entity and one relation in each sentence. This enables the AQG to formulate more precise queries.

I'm wondering if the authors also incorporated these knowledge sources into the VE method. If not, I believe such a comparison would be unfair.

Yes. In our baseline results for VE, the retrieval system used is Google Search, which includes all the knowledge that CoK can access.

I'm curious about how often such a process needs to be introduced in practical experiments.

As mentioned in Appendix I, 86 out of 308 instances for HotpotQA 6-shot, and 127 out of 1000 instances for FEVER 3-shot pass through the knowledge adapting stage.

Which of the three extraction methods (natural language, SPARQL, and SQL) do the authors consider to be more reliable for LLMs?

Could you please clarify your question? If you are inquiring about which query format is most reliable for LLMs to produce, then the query format would depend on which knowledge source we are retrieving from. For example, for knowledge graphs such as wikidata, SPARQL is used. For natural sentence knowledge sources such as Medical Flashcard, natural language is used. Using different query languages that do not match the format of the knowledge source will likely result in suboptimal performances.

Dear reviewer SrPL,

Thank you for your recognition of our comprehensive experiments and contributions! We would like to address your queries and concerns here.

While the paper emphasizes the use of diverse knowledge sources, it might benefit from a more explicit discussion on the scalability and efficiency of the framework as the volume and variety of sources grow. What are the computational overheads introduced by the AQG and the dynamic knowledge adaptation process, especially when querying multiple heterogeneous sources?

As sources grow, CoK will remain computationally efficient due to the following reasons:

1. The knowledge domain selection step enhances the efficiency of knowledge retrieval by ensuring only the most pertinent knowledge sources are selected for retrieval.
2. As highlighted in Section 1, AQG is versatile and can either be a fine-tuned model or an off-the-shelf LLM. The fine-tuning process is a one-time effort. And using off-the-shelf LLMs only requires prompt engineering. This simplicity facilitates the scaling of both the volume and variety of knowledge sources.

The computational overheads of CoK are on-par with or less than existing retrieval augmentation methods: The inference of CoK is on par with Verify-and-Edit (VE) in terms of time and cost. Firstly, the inference time of CoK is similar to VE, with the only additional time required being for query generation. VE utilizes ChatGPT for this purpose. AQG can use either a fine-tuned LLaMA-2 or ChatGPT. The average inference time per instance for LLaMA-2 is 4 seconds, while for ChatGPT is 2 seconds. During inference, data retrieval from various knowledge sources is executed concurrently. As such, the maximum inference time overhead of CoK compared with VE per instance is only 2 seconds. Secondly, as shown in Table 12, the cost per instance of CoK is on par with VE as well. The extra costs are incurred by the dynamic knowledge editing stage, which is shown to boost performance in the main results.

CoK also costs much less than ReAct, incurring only around 40% of ReAct’s costs. A detailed cost analysis can be found in Appendix I and Table 12.

The framework's dependence on the AQG's capability to generate effective queries for all types of knowledge sources might be a potential bottleneck, especially for highly specialized domains.

As mentioned in the CoK framework (Section 3), AQG is a versatile component and can always be replaced by the SOTA model depending on the querying language. Furthermore, AQG can be either a fine-tuned model such as LLaMA-2 or a black-box LLM. For instance, as indicated in Table 11, when generating commonly-used querying languages like SQL, ChatGPT produces better results. However, for specialized domains that require specific querying languages, such as SPARQL, we find that a fine-tuned LLaMA-2 proves to be a more performant option.

Although the paper mentions improvement over the CoT baseline, deeper insights into the limitations and areas where CoK might not perform optimally would be beneficial. Except for the limitations of “Knowledge Sources” and “Knowledge Retrieval” discussed in Appendix G.

Thank you for the suggestion! One potential limitation is that CoK relies on the reasoning capability of LLMs. Thus, failure cases may stem from reasoning failures of LLMs. Please refer to the Appendix G in our revised submission for more information. We also include three examples of these failure cases in Appendix H.2.

How does the CoK framework handle situations where knowledge sources provide conflicting information? How are you going to solve it?

As highlighted in Appendix G, we only select highly authoritative knowledge sources which are unlikely to contain inaccurate or conflicting information. Additionally, we manually examined a random selection of 100 samples from HotpotQA and found no instances of conflicting information.

Have there been considerations or plans to extend the CoK framework to accommodate real-time or streaming knowledge sources?

CoK supports real-time and streaming knowledge sources. For knowledge sources such as Wikidata, Wikipedia, UpToDate, PhysicsClassroom, and CK-12, knowledge is retrieved from their corresponding online websites. These websites are consistently maintained and updated in real-time. Furthermore, CoK supports static data, such as local tables, Flashcard, ScienceQA Physics, and ScienceQA Biology. We believe that this further demonstrates the versatility of CoK.

Dear Authors and Chairs,

I have reviewed the rebuttal from authors, and really appreciate the authors' effort and further explanation on this work.

Their response have addressed my concerns, and I recognize the versatility of robustness and generalizability of CoK. The main reason that I choose to keep my score is that, the additional contribution of this work compared with Verify-and-Edit (VE) [1] is anyhow limited.

If this work can be further improved, for example, to propose a more general framework of refining reasoning process during training and after generation with adaptive knowledge retrieval, and provide more theoretical analysis, I think it will make this work much more solid and can better meet the ICLR acceptance criteria.

[1] Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework (ACL 2023)

Best Wishes,

Reviewer SrPL

How does the author ensure that LLM can give faithful answers (query) when performing knowledge adaptation?
When generating SQL (SPARQL) queries, how do the authors guarantee that the produced queries are valid?

We acknowledge that there is a possibility of AQG generating erroneous queries. However, we believe this is a common problem for all generative models. To mitigate this, we employ diverse techniques for query generation and knowledge retrieval:

1. For query generation, AQG utilizes either black-box LLM or fine-tuned Llama-2. When using black-box LLM such as ChatGPT, we employ few-shot prompting and incorporate all necessary information in the prompt. For example, when generating SQL queries, the table schema along with data snippets are included in the prompt. When utilizing fine-tuned Llama-2, we ensure the logical granularity of AQG training data is on par with the decomposed CoT rationales. For example for WikiData, both training data and CoT rationales only contain one entity and one relation in one sentence. This enables the AQG to formulate more precise queries.
2. Our knowledge retrieval relies on highly authoritative knowledge sources. Therefore, even if the generated queries contain inaccuracies, the retrieved knowledge is faithful. As outlined in Appendix A.1.3, during knowledge editing, we also include a demonstration in the training prompt where the original rationale is unchanged. Thus, if no usable knowledge is retrieved, the baseline scenario would be where the original CoT rationale remains unchanged.

As highlighted in Section 5.3, we provide both quantitative and human evaluations to demonstrate the effectiveness of our techniques. Results demonstrate that CoK enhances the factual accuracy of rationales, leading to more precise final answers.

I am interested in whether this method is only applicable to chain-style question? For more complex questions, where the corresponding SPARQL contains more than one inference path, can the proposed method handle it?

The chain-of-knowledge framework utilizes chain-of-thought decomposition, which is not only beneficial for multi-hop QA (i.e., chain-style questions), but also questions requiring parallel reasoning (i.e., questions containing more than one inference path). Experiments on FEVER dataset support its effectiveness as FEVER is not multi-hop (chain-style), but contains claims that require composition of independent evidence from multiple sentences [5].
Furthermore, we ensure that each rationale after CoT decomposition only contains a single entity and relation via prompt engineering as shown in Appendix A.1.1. As shown in Appendix D.1.2, the natural sentence input of the training data for AQG also contains a single entity and relation. This alignment in the complexity level of rationales and training data allows AQG to generate more accurate queries.

[5] Thorne, James, et al. "FEVER: a large-scale dataset for fact extraction and VERification." In Proceedings of NAACL, 2018.

Dear reviewer bxPp,

Thank you for your time and suggestions. We would like to address your queries and concerns here.

In my opinion, a key point in the effectiveness of the proposed method is how to construct accurate queries to retrieve grounding information, but the description in section 3 is still not clear enough.

Thank you for the suggestion! We have added more details about the adaptive query generator in Section 3. This includes task-specific details such as providing the table schema for SQL, and how the domain of the training data corresponds to the respective knowledge source.

How does the author determine the table or tables used when generating a SQL query for a rationale, and how can they ensure that the query generated by ChatGPT is valid and answerable if table schema information is not provided. Ditto for SPARQL.

Following the existing formulation for the FeTaQA task, each question is provided with a specific table and the table schema. To ensure a successful query generation, the AQG should be aware of the structure and schema of the knowledge source. Therefore we always feed in the table schema.

For SPARQL, no table schema is required. Instead, we use an AQG trained with data whose logical granularity is on par with the CoT rationales, i.e, both training data and CoT rationales only contain a single entity and relation in each sentence. This enables the AQG to formulate more precise queries.

Baseline performance on the hotpotQA dataset seems too poor. Existing work [1] has achieved an accuracy of 0.37 when reviewing ChatGPT.

Following hotpotQA [2] and other previous work [3,4], we adopt the exact match metric for evaluating hotpotQA. However in [1], the evaluation for hotpotQA is partial match, a less stringent standard that typically yields higher scores. Below, we include results using partial match for hotpotQA for your consideration. In this case, CoK still outperforms over baseline methods.

[1] Zheng, Shen, Jie Huang, and Kevin Chen-Chuan Chang. "Why does chatgpt fall short in providing truthful answers." ArXiv preprint, abs/2304.10513 (2023).
[2] Yang, Zhilin, et al. "HotpotQA: A dataset for diverse, explainable multi-hop question answering." In Proceedings of EMNLP, 2018.
[3] Zhao, Ruochen, et al. "Verify-and-edit: A knowledge-enhanced chain-of-thought framework." In Proceedings of ACL, 2023.
[4] Yao, Shunyu, et al. "React: Synergizing reasoning and acting in language models." In Proceedings of ICLR, 2023.

Thanks to the author for the reply. Some of my concerns are addressed. I expect that the authors will be able to explain the issues mentioned in more detail in the main body of the paper, including the validation of the query, and the choice of evaluation metric. I have updated the rating.

Dear reviewer 84zL,

Thank you for your recognition of our contributions! We would like to address your queries and concerns here.

do you use some library to convert the entities/relation in their specific identifier form (entity linking), how do you know which is the correct identifier, in specific cases where similar natural language entities have two identifiers.

We employ the commonly used GENRE model [1] for entity linking, which effectively addresses the cases of ambiguous entities by generating unique entity identifiers. We have also included the details in Appendix C.1 in our revised submission.

Maybe you should come up with an example where ChatGPT would hallucinate.

Thank you for your suggestion! In general, we observe that ChatGPT is more likely to hallucinate for domain-specific questions. Our chosen example highlights that even for questions needing common factual knowledge, ChatGPT can generate erroneous responses. We utilized gpt-3.5-turbo-0613 for our experiments, the latest version available at the time of experiments.

In the revised submission, we have updated the example where the model failed to provide a correct answer. The CoT result in the paper can be reproduced using gpt-3.5-turbo-0613 with a temperature of 0, following the below prompt. Furthermore, even the latest gpt-3.5-turbo model generates erroneous responses.

"""
Strictly follow the format of the below examples, provide two rationales before answering the question.
Q: This British racing driver came in third at the 2014 Bahrain GP2 Series round and was born in what year?
A: First, at the 2014 Bahrain GP2 Series round, DAMS driver Jolyon Palmer came in third. Second, Jolyon Palmer (born 20 January 1991) is a British racing driver. The answer is 1991.

Q: What band did Antony King work with that formed in 1985 in Manchester?
A: First, Antony King worked as house engineer for Simply Red. Second, Simply Red formed in 1985 in Manchester. The answer is Simply Red.

Q: How many inhabitants were in the city close to where Alberta Ferretti’s studios was located?
A: First, Alberta Ferretti’s studio is near Rimini. Second, Rimini is a city of 146,606 inhabitants. The answer is 146,606.

Q: What year was the Argentine actor who directed El Tio Disparate born?
A: First,
"""

It is possible that the knowledge injected through Wikidata doesn't answer the question asked, what would be done to answer the question in that case?

As illustrated in Appendix A.1.3, we include a demonstration in the training prompt where the original rationale is unchanged. Thus, if no usable knowledge is retrieved, the baseline scenario would be where the original CoT rationale remains unchanged (a fall-back strategy).

We acknowledge that retrieval of irrelevant information (i.e., doesn’t answer the question asked) is a possibility. To mitigate this, we employed two strategies in this paper throughout the framework, leading to SOTA results compared to existing methods, such as ReAct and VE, as shown in Table 2.

1. We used an AQG trained with data whose logical granularity (number of entities and relations involved) is on par with the CoT rationales. For example for WikiData, both training data and CoT rationales only contain one entity and one relation in one sentence. This enables the AQG to formulate more precise queries.
2. We incorporated multiple knowledge sources for each domain to get more validating evidence. The results in Table 4 confirm that using various sources enhances knowledge coverage.

Can you please explain how do you find specific relation for querying with an entity? There is a possibility of multiple entity attached with one relation. It is also possible that multiple relations connected with multiple entities, and it may be difficult to identify the relation depending on the question. What would be done in that case?

As illustrated in Appendix A.1.1, every rationale after the reasoning preparation stage should comprise a single entity and relation. Similarly, Appendix D.1.2 indicates that each training data sample for SPARQL contains only one entity and one relation. This consistency ensures that the training data aligns with the rationale. Consequently, AQG can effectively generate more precise and disambiguous queries.

[1] Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity retrieval. In Proceedings of ICLR, 2021.