DMQR-RAG: Diverse Multi-Query Rewriting in Retrieval-Augmented Generation

Thank you for the feedback. Our response is detailed below.

Q1: The technical depth of the four rewriting strategies and the adaptive rewriting selection method is insufficient. Q2: The paper appears to contribute primarily a prompt without additional significant contributions

The primary focus of our work is on developing a robust ensemble strategy and verifying its feasibility. Therefore, experimental validation is our main approach. We conducted extensive experiments and also tested our strategy in industrial scenarios, all of which confirmed the effectiveness of our strategy.Please find our response in general comments to all reviewers and AC.

Q3: The adaptive rewriting strategy selection significantly increases the number of tokens and inference latency, which are not addressed in the paper.

Regarding cost analysis, the only distinction between our proposed method and the traditional RAG process lies in the query rewrite phase,as detailed in the fifth point of our summary in general comments to AC and all reviewers.

We sincerely appreciate your valuable suggestions for improving our method. Your concerns are addressed below.

Q1: Suggestions for improving statistical clarity in tables and figures

To address your feedback on our manuscript, while we appreciate your suggestion regarding bolding numbers and including additional elements in the figures, we believe that the current format sufficiently conveys the necessary information. To explain further:

1. Concerning bolding numbers that are statistically distinct from the second-best results in our tables: We have opted for a uniform formatting style across the document to ensure simplicity and ease of reading.
2. As for adding error bars and indicating p-values or confidence levels in Figure 4, specifically in Figure 4(c): We aimed to keep the figures as clear and straightforward as possible, directing the readers to the text for the detailed statistical context. However, adding p-values, confidence levels, or error bars indeed enhances the rigor of the experiments. We will consider incorporating these elements in subsequent experiments.

Q2: I would like to see a proper discussion that clearly describes the difference of the current paper's DMQR strategy, and other works such as RAG Fusion.

In fact, as detailed in the first and the second point of our summary in general comments to AC and all reviewers, our focus lies on introducing a novel ensemble strategy, rather than a new query rewrite method. Nonetheless, we have also introduced two new rewriting techniques: keyword rewriting and core content extraction.

Q3: I think limiting the setting to only bing + BGE is a potential bias-inducing practice. I would like to see authors compare their method with RAG Fusion while using different retrievers, as mentioned before.

Regarding your suggestion to compare different retrieval systems, we fully understand the importance of this for enhancing the reliability and generalizability of the research. However, due to the scope and focus of our current study, we have chosen to restrict our experiments to the Bing + BGE setup. This decision is primarily based on the following considerations:

1. Our primary interest lies in assessing whether different LLMs used as rewriters can improve the results of retrieval and generation. To better analyze the quality of the rewriting and to avoid introducing additional variables, it is necessary to keep the retrieval, reranking, and generation modules consistent. This approach helps to prevent the outcome from being skewed by performance variations in other modules.
2. The Bing + BGE combination, widely recognized as a robust configuration for retrieval and reranking, provides a solid baseline that facilitates direct comparisons with other research findings.

Despite this, we acknowledge that experiments conducted under different retrieval system configurations possess unique value. In our future work, we will explore how methods perform under various retrieval configurations.

We thank the reviewer for the encouraging feedback. Regarding the points you raised, we would like to clarify the following:

• Our contributions are not limited to the combination of different query rewrite strategies. Please find our response in general comments to all reviewers and AC, specifically the contribution part.
• Regarding cost analysis, the only difference between our proposed method and the traditional RAG process lies in the query rewrite phase, as detailed in the fifth point of our summary in general comments to AC and all reviewers.
• When analyzing baselines experimentally, we adhered to the original settings proposed by the developers of each method. If the proponents of a method employed a few-shot approach, we conducted our experiments using a similar few-shot setup. For those methods that do not use few-shot, such as rag-fusion, we have tried adding few-shot, and the results show it is on par with or even inferior to zero-shot.

---

Q1. 1: How have the proposed method solve the different problems mentioned in the paper? Does it reduce noise?

Thank you for pointing these out. Some experiments in the paper manage to illustrate certain facts.

In the ablation study section, we validated that each rewriting strategy plays a distinct and crucial role in complementing the weaknesses of other rewriting strategies. Additionally, in Section 3.2.6, we conducted a case study to demonstrate that our method effectively addresses the following issues:

1. Keyword Rewriting aligns with search engine preferences, thereby recalling more useful documents.
2. Core Content Extraction discards some detailed content to perform retrieval at a coarser granularity, which in turn recalls more relevant documents.
3. Pseudo-Answer Rewriting matches the pattern of answers, thereby hitting more relevant documents.

In the future, we will conduct more experiments to further refine this analysis.

Q1. 2: For what queries each type of rewrite is the most useful? Instead of asking an LLM to make the selection, would it be better to train a small selector for it?

Each query rewriting method serves a different purpose, as can be seen from the case studies. However, in some instances, it is not necessary to perform rewriting four times; three or even fewer rewrites can also retrieve relevant documents, as evidenced by the ablation studies. Consequently, we have designed a series of guidelines to guide the LLM in making selections, which shows in the third point of our summary in general comments to all reviewers and AC.

We could train a small selector to handle this task. However, considering that the focus of our current work is to validate the concept of an information-based ensemble approach, and not on training a new model, as detailed in the first point of our summary in the general comments to the AC and all reviewers, we did not pursue this in the present study. In future work, we plan to explore training a compact selector model.

Q2: Have you compared with more sophisticated fusion methods? There are a number of alternative approaches proposed in the IR literature for result fusion. They could be considered as alternatives.

We have reviewed fusion methods in the IR field, which primarily involve decomposing queries into sub-queries based on their intent, and then conducting searches for these sub-queries (Wang et al., 2024a; Besta et al., 2024) not to adopt this approach for the following reasons:

1. This approach addresses problems similar to those tackled by RQ-RAG in our baseline.
2. Our emphasis is on generic rewriting. In contexts with multiple intents, our method can be applied to each of the aforementioned sub-queries, allowing for multi-route rewriting and retrieval for each sub-query. The focus of these two approaches is different: one emphasizes the decomposition of intents, while the other emphasizes multi-route rewriting for each intent/query. Our experimental results on the HotpotQA dataset demonstrate that our method is still capable of effectively handling complex intent queries.

Q3: The proposed method is related to search result diversification, for which some studies proposed to diversify the search queries. Can the proposed method be related to these diversification methods

Our baseline can be viewed as a method to diversify search results. The connections between our proposed method and these approaches have been detailed in Appendix A.

[1] RichRAG: Crafting Rich Responses for Multi-faceted Queries in Retrieval-Augmented Generation

[2] Multi-Head RAG: Solving Multi-Aspect Problems with LLMs

We greatly appreciate your valuable feedback. Firstly, we would like to clarify the following point:

• Regarding the criteria behind the selection mechanism, we have outlined several guidelines provided to the LLM in the form of prompts. We also supplied a few-shot learning examples to enable the LLM to leverage its in-context learning capabilities for autonomous selection. This process is detailed in the third point of our summary in the general comments to the AC and all reviewers.

• Regarding why we did not opt for traditional modern query-rewrite strategies, we considered the fact that LLMs possess strong comprehension abilities, fully capable of handling such tasks and surpassing conventional methods. Meanwhile, we chose to maintain consistency with the settings of previous mainstream LLM-based query rewrite papers by selecting an LLM-based approach for comparison. Therefore, we did not consider traditional methods.

---

Q1: What are the implementation details of the methods presented in 2.1. Specifically, what novel additions were made to the baselines mentioned in the section explaining equality, expansions, and reduction?

Each of our methods is implemented through the prompt engineering. By carefully designing prompts, we accomplish various rewriting tasks. Specific details about the prompts can be found in Appendix C. We conducted experiments with different open-source and proprietary LLMs, demonstrating the versatility of our designed prompts.

Additionally, while our focus remains on how to effectively ensemble different rewriting methods, we have also introduced two new rewriting techniques along with a dynamic selection scheme. Please refer to our response in the general comments to all reviewers and AC, specifically the contribution part.

Q2: Did you consider any classical query rewriting methods ?

We have thoroughly reviewed recent LLM-based query rewriting methods and selected a few representative works from them. Specifically, we have categorized them into the two aspects, as detailed in Section 3 in our paper.

It is worth noting that recent works often focus on proposing a new framework. For a more effective comparison, we only considered those works purely dedicated to query rewriting. Additionally, our proposed method is "plug-and-play," allowing it to be integrated into any existing framework to further enhance performance.

Q3: Do your methods work for queries that are not multi hop or are not complex ?

In our experiments, we chose the AmbigQA dataset, a simple and straightforward QA dataset specifically designed to address the inherent ambiguity in Natural Questions. The FreshQA dataset also encompasses both simple and complex QA components. The exemplary performance of our method on these two datasets demonstrates that it can still outperform other rewriting methods in simple QA scenarios.