Boosting the Potential of Large Language Models with an Intelligent Information Assistant

Dear Reviewer,

Thank you for your valuable feedback. We have responded to each of the weaknesses (W), questions (Q), and limitations (L) you raised. We hope the following responses clarify the contributions of our work and address your concerns.

---

R to W1:

While memory and knowledge management are common elements in agent design, our approach introduces significant innovations that set it apart:

1. AssistRAG is the first to integrate a complete agent solution within RAG scenarios, encompassing tool usage, action execution, memory building, and plan specification. This sets a foundational approach for future intelligent assistants for human.

2. By decoupling the RAG task into an Assistant LLM and a Main LLM, AssistRAG enhances the assistant's adaptability to RAG scenarios without compromising the main LLM's inherent capabilities through a two-phase training process.

3. AssistRAG adapts efficiently to new LLMs without the need for retraining from scratch, achieving significant improvements with preference alignment.

The following table highlights the distinctions between AssistRAG and other representative works:

---

R to W2:

We understand your concerns regarding the cost and adaptability of the proposed method. In fact, our Assistant-based RAG framework is specifically designed to improve adaptability to different foundation LLMs.

1. Decoupling the RAG task into an Assistant LLM and a Main LLM is intended to separate the training of the Assistant LLM from the Main LLM. The Assistant Learning phase remains unchanged regardless of the Main LLM. This allows a well-trained Assistant LLM to be adapted to various Main LLMs without the need for retraining from scratch.
2. The DPO training phase is designed to enhance compatibility between the Assistant and Main LLMs. Even without this phase, the Assistant LLM can still effectively support the Main LLM, as demonstrated in Table 2 (No DPO Training).

---

R to W3:

There may have been a misunderstanding regarding our presentation. We hope the following points clarify this:

1. The 58% refers to the proportion of errors caused by insufficient knowledge retrieval among all incorrect answers, not all questions.
2. We employ a highly advanced and widely used model LLM-Embedder for retrieval, which is proposed in 2023 and has been downloaded over 56k times on Huggingface.
3. Our focus is on complex multi-hop QA tasks, which require retrieving at least two correct documents to ensure knowledge completeness. This increases the difficulty for the retriever.

---

R to W4:

We would like to clarify the following points to address your concerns regarding the use of GPT-4:

1. Using GPT-4 to generate fine-tuned data has been widely adopted in various scenarios, such as WizardLM, WizardCoder, WizardMath.
2. Our proposed method does not rely exclusively on GPT-4. Recently released powerful open-source models, such as Llama-3.1 and DeepSeek-V2, can serve as alternatives.
3. We will open-source our training data and models. Community researchers will be able to adapt our method to their own LLMs without needing to use GPT-4 for generating training data.

---

R to W5:

We apologize for the unclear notation. We will revise the manuscript to provide clearer definitions and explanations for the notations.

---

R to W6:

We acknowledge the inconsistency you pointed out between Table 1 and section 4.3 (Inference Settings). To clarify, the base model used for training the Assistant LLM is ChatGLM3. In order to verify that the Assistant LLM can adapt to various main LLMs, we conducted inference using LLaMA2, ChatGLM, and ChatGPT, which also involved providing preference data. We will improve our wording in future versions to prevent any misunderstandings.

---

R to Q1:

We appreciate your request for additional details regarding the model versions used. The specific ChatGPT model employed in this paper is gpt-35-turbo-16k, provided by Azure, with a release date of 2023-05-15.

---

R to Q2:

There are several reasons for choosing 100-token wikipedia passages as the retrieval corpus:

1. Most baselines in its original papers used this setting, allowing for easier comparison and alignment with these baselines.
2. Wikipedia's official passage-level retrieval corpus segments articles into 100-token passages.
3. We do not input just one passage into the LLM but combine multiple retrieved passages. This length ensures that the combined input does not exceed the length limits of certain LLMs.

---

R to Q3:

This is indeed a valuable suggestion. We have expanded our study to include the ALCE-ELI5 dataset, which originates from the Reddit forum. This dataset is built upon the Sphere corpus, encompassing 899 million passages. The results are as follows:

---

R to L1:

We have conducted additional experiments comparing our proposed method with two Agent frameworks capable of performing RAG tasks: Toolformer and Reflexion. These experiments were conducted on the 2wiki dataset, and we will supplement results from other datasets in later version.

---

We hope these clarifications address your concerns and provide a better understanding of our work. If you have any further concerns, we would be delighted to continue the discussion with you.

Thank you very much for your valuable feedback and recognition of our paper. We hope that the following responses will address your concerns:

---

R to W1:

To further demonstrate the effectiveness of our model, we have included additional experiments with two agent solutions on the 2wiki dataset. The results are as follows:

These results show that AssistRAG outperforms both Toolformer and Reflexion, demonstrating its superior performance in RAG tasks.

---

R to W2:

Thank you for your insightful suggestion. In Appendix A.4, we have discussed some potential limitations and challenges in deploying AssistRAG in practical settings. We will provide a more detailed discussion in future versions to further enhance the paper's impact and applicability following your advice.

---

R to Q1:

Computational efficiency in real-time applications is indeed crucial. To address the concerns, we first analyze the composition of inference time when encountering a new question:

1. The time taken by the Assistant LLM to perform actions, including Question Decomposition and Knowledge Extraction.
2. The time taken by the Assistant LLM to call retrievers, including memory retrieval and knowledge retrieval.
3. The time taken by the Main LLM to generate the answer.

Among these, steps 1 and 3 are independent of dataset size. Dataset size only affects step 2. In step 2, to achieve low-latency retrieval, we have utilized the FAISS library to index the documents and employed an IVF index structure for acceleration. This allows for millisecond-level retrieval speed even with datasets in the millions of documents range. Compared to the seconds-level time taken by steps 1 and 3, the retrieval time in step 2 is negligible. Therefore, very large datasets (below the billion-level) would not significantly impact computational efficiency.

---

R to Q2:

Thank you for your insightful suggestion. We have conducted additional experiments to analyze the sensitivity of AssistRAG (ChatGLM 6B) to different hyperparameters, specifically the learning rate during the Assistant Learning phase and the number of documents retrieved (K) during the knowledge retrieval phase. The results are as follows:

We hope these additional analyses provide a clearer understanding of how AssistRAG performs under different hyperparameter settings and training configurations.

---

R to Q3:

This is indeed a valuable suggestion. Following your recommendations, we conducted a user study involving three participants. Each participant was asked to write 20 factual questions. Subsequently, answers were generated using ChatGPT in three configurations: CloseBook, Naive RAG, and AssistRAG. The participants then evaluated their satisfaction with the answers provided. The results are as follows:

These results demonstrate that users were more satisfied with the answers generated by AssistRAG compared to the other configurations. We believe this user study highlights the practical usability and effectiveness of AssistRAG in aiding human users.

---

R to L1:

We apologize for the confusion. Our intention in selecting models with varying parameter counts was to demonstrate that AssistRAG significantly outperforms CloseBook and Naive RAG inference methods, regardless of the model size. We appreciate your suggestion and agree that presenting the results by grouping models of similar sizes will make the comparisons clearer. We will revise Table 1 to reflect this improvement, ensuring that models with similar parameter counts are compared directly.

---

R to L2:

Thank you for your suggestion. In fact, we have conducted token usage analysis across all three datasets. Since these datasets share the same retrieval corpus, the lengths of the retrieved passages are similar, resulting in no significant differences in token usage across different datasets. Therefore, we chose to present the representative 2Wiki results. For consistency, we will include token usage results for all three datasets in future versions of the paper.

---

We greatly appreciate the time and effort you have invested in reviewing our manuscript. Your insightful comments have been invaluable in helping us improve the quality and clarity of our paper. Thank you once again for your constructive feedback.

Thank you for your recognition of our paper. We sincerely appreciate all the feedback from the reviewers and will make revisions to enhance the paper's impact and applicability based on your valuable suggestions.

Regarding the datasets, we have compiled the publication years of all commonly used QA datasets, as shown in the table below:

As our paper focuses on complex QA tasks, we selected Multihop QA datasets for our study. These include Bamboogle (2023), 2WikiMultiHopQA (2020), and HotpotQA (2018). Among these, Bamboogle is the most recent Multihop QA dataset available.

Thank you once again for your constructive feedback. We greatly appreciate the time and effort you have invested in reviewing our manuscript.