MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries

Thank you for your insightful comments. We believe there is some misunderstanding of our paper, and we hope to take this opportunity to clarify some important points.

1. Regarding your comments on “.. overlooks existing datasets such as HotpotQA and 2WikiMultiHop...” we indeed discuss HotpotQA and 2WikiMultiHop in Section 5 “Related Work”. These datasets primarily focus on assessing a model’s reasoning skills, and they do not emphasize the retrieval of evidence from a knowledge base, thereby not tailored for RAG context. Additionally, their primary data sources, Wikipedia, significantly overlap with the training data of most existing LLMs. If we use these sources for benchmarking RAG systems, there is a potential concern that LLM responses might rely on training knowledge rather than reasoning from the retrieved knowledge base.

2. Regarding your comments on “The evaluation methodology appears to inadequately address the core concept of multi-hop reasoning,” we acknowledge that there are more advanced methods to handle multi-hop queries, yet those methods are benchmarked on widely used public QA corpora. For example, [1] is evaluated on open-domain question-answering benchmarks, which is different from real-world RAG scenarios. Moreover, another outstanding concern is that those benchmark datasets are exposed to LLM pretraining. Multihop-RAG, instead, includes news articles published after September 26, 2023, beyond the knowledge cutoff of some widely-used LLMs, including ChatGPT and LLaMA, as of the time of writing. This selection also helps in teasing out the possibility of the underlying LLM having been exposed to these news articles, thereby simulating a common scenario of RAG where it is used to retrieve from non-public or proprietary datasets. Therefore, we believe Multihop-RAG can serve as a useful resource for benchmarking advanced RAG techniques.

3. Regarding your comments on "construction of the dataset raises several concerns," to clarify, we do not solely rely on LLMs for dataset construction. Human inspection and manual review by the authors are conducted to ensure data quality. Moreover, we have hired native English-speaking crowd workers to test the human capability of answering the questions. We will provide more detailed information in the revised manuscript.

[1] Yu, W. et.al. (2023). Chain-of-note: Enhancing robustness in retrieval-augmented language models.

Should you have further questions, we would be happy to answer them.

Thank you for your constructive comments! Below, we will respond to each of your comments in detail, and we hope can address your concerns satisfactorily.

Q1. Details of quality control

Your comment is well taken. Regarding the details of the review process, for each different question type, we randomly sampled 20 samples, which amounted to a total of 60 samples (excluding null queries). We then hired three native English-speaking crowd workers. Provide multiple full documents corresponding to each query and highlight the relevant evidence. We asked the annotators to answer the questions and rate their difficulty from 1 to 5. After keeping the accuracy of generated answers. We also summarized the difficult score distributions in the following table:

We can observe that the human assessment of question difficulty aligns with the results shown in Figure 3 of the paper. Temporal and comparison questions are more challenging compared to inference questions.

Q2. Lack of analysis of existing poor performance

We appreciate the insightful feedback. As you rightly point out, the QA performance highlighted in Table 6 is unsatisfactory and warrants further investigation. Our analysis points to two possible reasons:

1. Inadequate Retrieval: The retrieval component in RAG struggles to retrieve relevant evidence effectively based on multi-hop queries from the knowledge base, leading to lower overall response accuracy. This might indicate that existing embedding models fall short in retrieving the most relevant context, suggesting a need for improvements in embedding models.

2. Limited Reasoning: Even when provided with accurate ground-truth evidence, the response accuracy of LLMs, such as Mixtral-8x7B, remains unsatisfactory. This model particularly struggles with comparison and temporal queries (as shown in Figure 3), indicating that the comparison and temporal reasoning capabilities of LLMs are still weak, highlighting areas where further advancements are essential.

By showing the unsatisfactory performance of the Multi-hop RAG and making the dataset publicly available, we hope to spur further research and development in this important field.

Thank you for your constructive comment! Below, we respond to each of your comments in detail and hope to address your concerns satisfactorily.

Q1. The number of queries seems a bit small

Thank you for your observation. Indeed, expanding the dataset could enhance the robustness of our findings. Our current evaluation dataset comprises 2,556 queries, a size determined by two key factors: first, we excluded a significant number of low-quality queries to maintain data integrity; second, we think the current dataset size might be sufficient for preliminary evaluations of retrieval and QA capabilities. For comparison, the widely referenced SciFact dataset (Wadden, D. et al., 2020) contains 1,409 entries. However, we acknowledge the validity of your suggestion and will consider incorporating a larger and more diverse set of queries in future iterations of our work.

Q2. Further explanation for table 6

As you rightly point out, the QA performance highlighted in Table 6 is unsatisfactory and warrants further investigation. Our analysis points to two possible reasons:

1. Inadequate Retrieval: The retrieval component in RAG struggles to retrieve relevant evidence effectively based on multi-hop queries from the knowledge base, leading to lower overall response accuracy. This might indicate that existing embedding models fall short in retrieving the most relevant context, suggesting a need for improvements in embedding models.

2. Limited Reasoning: Even when provided with accurate ground-truth evidence, the response accuracy of LLMs, such as Mixtral-8x7B, remains unsatisfactory. This model particularly struggles with comparison and temporal queries (as shown in Figure 3), indicating that the comparison and temporal reasoning capabilities of LLMs are still weak, highlighting areas where further advancements are essential.

Per your suggestion, we will include these explanations in the main text. Thanks!

We appreciate your thorough review and remain available for further discussion if you have any additional questions or require more clarification.

Thank you for your constructive comments. In the following sections, we will provide detailed responses to each of your comments, aiming to address your concerns satisfactorily.

Q1. Lack of comparison to existing datasets

We completely agree with you that a comparative analysis demonstrating the unique challenges and benefits of MultiHop-RAG over previous datasets could strengthen the case for its necessity and utility. We have discussed related datasets in the introduction and related work sections of our paper. To clarify and emphasize our dataset's unique position, we list the limitations of related datasets as follows:

• Related Datasets:

  1. RAG Evaluation Datasets: RGB (Chen et al., 2023) ,RECALL (Liu et al., 2023)

     • These datasets focus on single-hop questions and primarily measure the accuracy of final answers, not retrieval robustness.

  2. Fact Extraction datasets: FEVER (Thorne et al., 2018), SciFact (Wadden et al., 2020), HoVer (Jiang et al., 2020), HoVer (Jiang et al., 2020)

     • These datasets neglecting broader QA performance and RAG framework robustness.

  3. Multi-document QA datasets: HotpotQA (Yang et al., 2018), MultiRC (Khashabi et al., 2018), 2WikiMultiHopQA (Ho et al., 2020).

     • Existing datasets predominantly utilize Wikipedia as their main data source, lack emphasis on the robustness of the RAG framework, and do not focus adequately on evidence retrieval capabilities.

• MultiHop-RAG (this work): This work uses multi-hop questions from multiple documents to test the robustness of the RAG framework, ideal for evaluating its performance in large-scale corpus retrieval and complex question-answering scenarios. Moreover, the dataset includes news articles published from September 26, 2023, to December 26, 2023, beyond the knowledge cutoff of some widely-used LLMs, including ChatGPT and LLaMA, as of the time of writing. This selection also helps in teasing out the possibility of the underlying LLM having been exposed to these news articles, thereby simulating a common scenario of RAG where it is used to retrieve from non-public or proprietary datasets.

Q2. English news articles as the sole source.

We acknowledge that relying exclusively on English news articles as a data source can introduce bias and potentially limit the scope of our queries. To address this limitation, we will discuss this in the limitations section of our paper and consider diversifying our sources in future research.