RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generation

Thanks for your comments.

1. Comparison with Knowledge Editing Methods (e.g., MQuAKE and PokeMQA)

To clarify, our work aims to enhance small models to explicitly refine queries for better answering both single-hop and multi-hop QA tasks, which do not overlap with the knowledge editing area mentioned by the reviewer.

We outline the differences between our work and knowledge editing as follows;

Knowledge editing addresses the issue that information stored in LLM becomes outdated quickly, and retraining from scratch is often not feasible. This has led to techniques for injecting new facts by updating model weights. Although knowledge editing and our work aim to address outdated parametric knowledge, our approach focuses on retrieval-augmented generation to improve factuality and reliability. Knowledge editing requires a set of new knowledge to be updated, focusing on effectively updating these new knowledge without disrupting other facts. In our work, no edited facts are provided, making the methods suggested by the reviewer inapplicable.

2. The Effect of Different Actions

As mentioned in response to reviewer PYYL, we carried out new experiments addressing the effect of different actions. For completeness, we include the results here.

Table1: The Ratio of Different Actions Under Different Sampling Strategies for 3 Tasks

Table2: F1 Score for Deliberately Using a Specific Action for All Instances of 3 Tasks

Main Findings:

• The proportion of different action selections is almost balanced, indicating minimal bias in the data as the reviewer was concerned (Table 1).
• Simply choosing the same action might not yield the best performance. Each action contributes uniquely, and combining different actions leads to better overall performance (Table 2).

We appreciate your constructive feedback and would like to address your concerns as follows:

1. Difference with Self-Ask

The reviewers mentioned that a primary weakness is the distinction between our approach and self-ask. Here, we justify the novelty of our work.

First, self-ask relies on large language models with strong instruction-following abilities. It barely works with smaller models that cannot refine their answers without further training. Our method meticulously curates a retrieval-augmented instruction tuning dataset, training a small model to possess the ability to refine queries.

Second, self-ask performs multi-step reasoning implicitly, restricting users' control over the types of follow-up queries. Conversely, our method uses special tokens to explicitly guide the model to generate various types of query refinements, providing greater flexibility.

Additionally, we provide new results for the methods for completeness as follows;

Highlight

Our newly trained model on LLaMA3 8B surpasses the self-ask method on both GPT-3.5 and GPT-4, suggesting the strong capability of our method and its robustness effect over stronger options. We expect this response to resolve the issue of more strong baselines and solidify its effectiveness.

2. Difference Between PPL and Beam Search

In our scenario, the sampling strategies we proposed resemble beam search. At each step, the different refinement action can be seen as a node to be explored. Among these nodes, there exists the most correct one (we use the upper bound to indicate the high potential of our system). The PPL or your mentioned universe SC can be seen as the heuristic to select the top-k node to expand. Under these circumstances, various heuristics can be proposed in our system. In this work, we preliminarily try out PPL and confidence, while other heuristics can be researched to further enhance the performance of our system.

Thanks for your insightful comments!

1. Claims of Novelty and Contributions

The main concern is how our method compares with Self-RAG and SAIL. We would like to clarify and highlight the novelty of our approach as follows:

Solely relying on RAG with its non-parametric parts shows some limitations, including but not limited to:

1. It requires strong instruction-following ability to reply considering the provided context, due to the different format from their original sft dataset.
2. It lacks interpretability when the retrieved documents contradict its parametric knowledge.

To address the above issues, our work focuses on effectively constructing retrieval-augmented instruction following datasets to train the model for internal learning and designing an effective mechanism for inference. This shares a common goal with previous work like self-RAG and SAIL. This methodology has also been explored by other works [1] after we submitted our research, highlighting the potential and value of this direction. While we draw inspiration from data construction processes such as self-RAG and SAIL, our key contribution lies in how we construct the instruction tuning dataset. This is where our paper differs from previous work and makes a significant contribution:

1. We explicitly make the model generate refined queries with more interpretability and controllability.
2. We do not reuse the original dataset output and claim that regenerating it is significant. To our best understanding, this is the first attempt to do so, and we view it as the novelty of this paper.

[1] Modarressi, A., et al. (2024). MemLLM: Finetuning LLMs to Use An Explicit Read-Write Memory. arXiv preprint arXiv:2404.11672.

2. Fairness When Compared to ChatGPT

Sorry for causing the misunderstanding. The method we compared (chain-of-thought and chain-of-note) uses retrieval settings where we utilize the generated thoughts and notes to retrieve relevant documents. Therefore, the comparison is FAIR, and we will make this clearer in our next version. Thank you for pointing this out.

Additionally, we have provided new results on our method trained on the new model LLaMA3 8B, which shows comparable results even to gpt4. This demonstrates the strong effectiveness and robustness of our method. Due to the limited characters allowed, please refer to the results shown in response to reviewer KkQG for more details.

Thanks for your insightful comments!

1. More Details on the Effect of Different Actions

Regarding the concern about the effect of different actions, we have complemented the new experiment as follows, and will provide case study to support the error analysis in our next version. Note that the newly provided results are based on our newly trained llama3 8b model.

• The ratio of different actions under different sampling strategies of 3 tasks.

• F1 score for deliberately using the specific action for all instances of 3 tasks.

The findings are as follows:

• Although the disambiguation process accounts for only 10% of our training data, the proportion of different action selections is almost balanced.

• Simply choosing the same action might not yield the best performance compared to the score with sampling strategies.

2. Inefficiency of the Disambiguation Process

The inefficiency arises from generating multiple queries and retrieving each piece of evidence, which increases inference latency. This is not a drawback of our method alone; multiple strong baselines, such as Self-Ask and ReAct, also generate follow-up questions multiple times to retrieve the necessary information. Regarding the numerous disambiguated queries generated, our trained model is designed to answer all of them, covering all generated queries.

3. Clarification of Iterative Query Refinement Annotation Process

We use the generated queries to retrieve documents from an external database, then concatenate them to regenerate Y_new, which happens only once. The intuition behind this process is that Y_new should be conditioned on all generated queries to cover all the queries. During inference, the model can decide to generate a refined query or choose to terminate and generate the final answer.