Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG

• Over-claiming of contributions.

Thank you for the comment. While we have already detailed the specific distinctions between our work and the five suggested papers, we want to emphasize the unique contributions of our analysis compared to the existing literature:

(1) Discovery of the inverted U-curve (Section 3.1): We are the first to systematically investigate the performance of various LLMs in RAG systems as the number of retrieved passages increases, leading to the discovery of the "inverted U-curve" phenomenon. While prior work like "lost-in-the-middle" [6] has explored the limitations of long-context LLMs, our research connects this to retrieval precision/recall in real-world RAG scenarios, providing a novel perspective on the interplay between retrieval and generation.

(2) Analysis of the inverted U-curve phenomenon (Section 3.2): We acknowledge the existing research on irrelevant retrieved information in RAG. However, we argue that simply focusing on noise is insufficient to explain the inverted U-curve. The initial performance increase with more retrieved passages indicates a complex interplay between relevant and irrelevant information. As the number of retrieved passages grows, the LLM must contend with both the benefits of increased relevant information and the detrimental effects of noise. This dynamic is not fully captured by existing work on noise in RAG, highlighting the distinct contribution of our research on long-context RAG. Moreover, we observe variations in the inverted U-curve across different retrievers, demonstrating that precision alone is an inadequate metric for quantifying the impact of irrelevant retrieved information.

(3) Deep understanding of the hard negatives (Section 3.3): Through synthetic experiments, we uncover a novel finding: the strength of the retriever directly correlates with the difficulty LLMs face in handling the corresponding hard negatives. This important insight, not previously identified in the literature, sheds new light on the complex relationship between retriever strength and LLM robustness in RAG systems.

• Limited technical contribution.

Thank you for your comment. We would like to highlight the key technical contributions of our proposed methods:

(1) Retrieval reordering: While this method is indeed simple and effective, its advantages extend beyond its ease of implementation. It offers high efficiency, significant performance gains, and seamless integration into existing RAG pipelines due to its plug-and-play nature. Importantly, the impact of retrieval order has not been extensively studied in the context of RAG. By introducing this method, we aim to draw attention to the importance of position engineering and encourage further research in this area.

(2) Tuning-based methods: We introduce two novel tuning-based methods to enhance the performance of long-context LLMs in RAG: an implicit tuning method and a method incorporating intermediate reasoning. These methods offer several advantages over existing approaches like RetRobust: Granularity of Focus: RetRobust primarily addresses scenarios where the entire retrieved context is irrelevant, focusing on deciding whether to utilize retrieved information at all. It predominantly considers retrieving a single passage and operates at the retrieval-level. In contrast, our work explores large-scale retrieval and delves into passage-level relevance, enabling finer-grained control over information utilization. Generalization: RetRobust primarily focuses on in-domain tuning, while our methods demonstrate strong out-of-domain generalization capabilities. This highlights the ability of our RAG-specific tuning to improve the inherent RAG capabilities of LLMs, leading to enhanced performance on unseen data. Intermediate Reasoning: Our approach explicitly trains LLMs to perform intermediate reasoning before generating the final answer. This differs significantly from RetRobust, which focuses on directly tuning the LLM to answer the question without an explicit reasoning step.

To further demonstrate the effectiveness of our methods, we conducted additional experiments comparing them with RetRobust [1]. The results show that our proposed techniques outperform RetRobust by a significant margin.

NQ

PopQA

(3) Comprehensive study of RAG-specific LLM tuning: LLM tuning for RAG purposes has a lot of design choices including LLM, context length limit, retriever, training/evaluation dataset etc.. We conduct a systematic study of different design choice components in Section 6 and Appendix M and N to provide design hints for researchers who are interested in implementing RAG-specific LLM tuning for their work or research.

[1] Making Retrieval-Augmented Language Models Robust to Irrelevant Context

[2] Chain-of-note: Enhancing robustness in retrieval-augmented language models

[3] Corrective retrieval augmented generation

[4] Evaluation of Retrieval-Augmented Generation: A Survey

[5] Benchmarking large language models in retrieval-augmented generation

[6] Lost in the Middle: How Language Models Use Long Contexts

[7] Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

[8] Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?

[9] Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context

[10] The Llama 3 Herd of Models

[11] Mistral NeMo

We sincerely appreciate your insightful feedback and have carefully addressed your comments below.

• Wrong research basis.

We appreciate you pointing out those relevant references. We've carefully reviewed your feedback and revised our manuscript to ensure a comprehensive literature review and accurately position our contributions. We have modified the motivation and illustration in Abstract/Section 1 and included a comparison with the suggested previous works in Section 2.

To be specific, (1) Lines 14-16 and 46-47: We've rephrased these lines to accurately reflect their basis in long-context LLM research [7][8], rather than implying broader RAG claims. (2) Section 1: We've clarified that prior work hasn't focused on evaluating long-context LLMs in real-world RAG scenarios, highlighting the novelty of our approach. (3) Contribution: We've sharpened the description of our analysis contribution in Section 1 to ensure accuracy.

Importantly, we would also like to argue that none of the mentioned works have studied long-context LLMs with RAG and confirmed that “under the newly emerged strong long-context LLM development [9][10][11], increasing the number of retrieved passages still cannot consistently improve the RAG performance due to the noise or incorrectness of the retrieved texts”. We note that the lines of research on long-context RAGs vs. noise-robust RAGs are relevant but not exactly aligned - the caveats and optimal bottlenecks might have notable differences. Increasing the number of retrieved passages not only increases the amount of noise in the context (lower precision and hard negatives) but also introduces more potentially relevant passages (higher recall as shown in Figure 2). Since more relevant passages could potentially benefit RAG performance, it is underexplored how the increased noise could still affect the RAG system. In addition, it is also not studied what is the correlation between the hard negatives and retriever capability (discussed in Section 3.3). We clarify the contribution of analysis in Section 1 to reflect this.

We provide a detailed comparisons with existing works mentioned by the reviewer to highlight the key distinctions:

Comparison with [1]: While [1] focuses on determining whether to use retrieved information at all when the entire retrieved context is deemed irrelevant, our work explores a different dimension: passage-level relevance within large-scale retrieval scenarios. Increasing the number of retrieved passages introduces a complex interplay of relevant and irrelevant passages, creating a unique challenge for LLMs in effectively utilizing this mixed information. Our research delves into this under-explored area, providing a comprehensive analysis and proposing three novel solutions to improve LLM performance in such scenarios.

Comparison with [2]: [2] focuses on improving the robustness of Retrieval-Augmented Language Models (RALMs) in scenarios with a small number of retrieved passages (around 5) and relies on synthetic experiments for evaluation. In contrast, our work conducts a systematic empirical study of long-context RAG in real-world settings, specifically addressing the challenges posed by a large number of retrieved passages. This distinction is crucial because processing a larger volume of retrieved information demands more advanced long-context capabilities from the LLM. While previous research has explored long-context LLMs in synthetic benchmarks, their performance in real-world, large-scale RAG scenarios remains under-explored. Our work fills this gap by providing a comprehensive analysis and offering three potential solutions to enhance long-context LLMs in such settings.

Comparison with [3]: [3] proposes a filtering-based approach to identify and remove irrelevant passages from the retrieved context. Our work, however, focuses on improving the long-context LLM generators themselves within large-scale retrieval RAG systems. These two approaches are complementary and can be combined to further enhance the robustness of the overall RAG system.

Comparison with [4][5]: [4] and [5] focus on evaluating RAG systems, with one providing a survey of evaluation techniques and the other introducing a new benchmark corpus. Our work differs by specifically investigating the behavior and improvement of long-context LLMs within RAG systems utilizing large-scale retrieved contexts.

In summary, our work presents a novel and systematic empirical study of long-context LLMs in large-scale RAG scenarios. We offer unique insights into the challenges posed by increasing retrieval scale and propose effective solutions to enhance LLM performance in these settings. This focus differentiates our contributions from existing research, which primarily concentrates on different aspects of RAG, such as handling small retrieved contexts, filtering irrelevant information, or evaluating overall system performance.

Thanks for your rebuttal, I have read your response thoroughly, but I do not think it addresses my concerns. Specifically:

1. In your response about wrong research basis, you argue that an important difference between your RAG for long-context and previous studies on the robustness of RAG to noise is that longer contexts not only introduce more noise, but may also introduce more positive passages. But this point is still essentially consistent with the previous study about the robustness of RAG to irrelevant documents. The complex relationship between relevant and irrelevant passages brought about by long context is essentially still a robustness issue of LLM in RAG. Imagine that an LLM model is faced with a retrieved list containing multiple passages, which contains both relevant and irrelevant passages. As long as the robustness of the LLM is high enough, it can ensure that the interference of irrelevant passages can be avoided. In this case, the increase in the length of the retrieved list will bring a stable performance improvement. In this case, the question you are studying is no longer valid. Therefore, the seemingly complex relationship you emphasize is essentially still the problem of LLM's robustness to irrelevant passages in RAG. You have essentially taken a problem that has already been well studied in previous work, wrapped it in a seemingly more complex guise to highlight the contribution of your work, which is not encouraged.

2. In your response about contribution, this still ties back to the point I emphasized earlier: this is fundamentally an issue of RAG's robustness to retrieved documents. The inverted U-shaped curve you highlighted is merely a repackaging of this issue. Specifically, in the curve you described, as the length of the retrieved list increases, RAG's performance first improves and then declines. This is essentially because the positive samples are ranked higher in the list, while the later portions predominantly consist of negative samples. Initially, as the retrieved list grows, more positive samples are included, leading to an improvement in RAG's performance. However, as the list continues to grow, more negative passages are introduced in the later ranks, which interfere with the LLM and cause a decline in performance. Isn't this fundamentally a matter of RAG's robustness?

3. Regarding your point on the "Deep understanding of the hard negatives," specifically that "the strength of the retriever directly correlates with the difficulty LLMs face in handling the corresponding hard negatives," do you really think this is a point worth emphasizing? Isn’t it rather self-evident? For a negative sample that can confuse a stronger retriever (and thus qualify as a harder negative for that retriever), it naturally suggests a higher likelihood of confusing the LLM as well, leading to reduced robustness of RAG against it. This seems like an obvious issue—why would you frame it as a "novel" contribution?

So I still believe that this paper does not meet the standards of ICLR, and I will not revise my review.

Before answering your follow-up questions, we would like first to clarify the motivation behind our work and show that the problem researched in this paper (long-context RAG problem) is valid.

With the rapid advancement of powerful long-context LLMs [1][2][3], there is an ongoing debate about whether these models can fully replace retrieval-augmented generation (RAG) systems that rely on a small number of retrieved passages in real-world applications [4][5]. A prevailing argument suggests that, given a sufficiently capable long-context LLM, the intermediate retrieval step could be eliminated by directly inputting the entire knowledge corpus into the model’s context [4][5]. However, the transition from (a) RAG with a small number of retrieved passages to (b) a system solely reliant on long-context LLMs introduces an important intermediate stage: (c) RAG with a large number of retrieved passages. This intermediate stage (c) can be seen as an extension of (a) and a subset of (b). If long-context LLMs (b) can genuinely replace RAG systems (a), then the performance of (c) should at least match or even surpass that of (a).

Motivated by this insight, our work investigates (c)—RAG with long-context LLMs leveraging a large number of retrieved passages—and systematically evaluates whether (b) can truly replace (a). Furthermore, we propose solutions to improve the robustness and performance of long-context LLMs for RAG applications. By addressing this intermediate stage, our work sheds light on a critical aspect of the transition between retrieval-augmented systems and fully context-based LLMs, which has not been adequately explored in existing literature.

Then, we would like to answer your following questions:

• You have essentially taken a problem that has already been well-studied in previous work. Fundamentally a matter of RAG robustness. (Q1, Q2)

We agree that, in a broader sense, our work is related to improving the robustness of LLM generation with noise input. However, we would like to point out the distinction of our work: (1) Discovery of U-curve in long-context RAG: even though the reviewer provides a valid explanation of the U-curve phenomenon, this phenomenon is not shown in existing works [6][7] with short-context RAG (< 5 retrieved passages). It can only be observed with long-context RAG. (2) Systematic analysis of how different retrievers and LLMs affect the noise robustness: We conduct a systematic study examining how RAG performance varies across different LLMs and retrievers. For example, before our work, it was not known that certain strong LLMs, such as Gemini, demonstrate remarkable robustness to hard negatives and show continuously improving performance as the number of retrieved passages grows. In contrast, other open-source LLMs, such as Gemma and Mistral, exhibit significant sensitivity to hard negatives, resulting in an inverse U-shaped performance curve. Our work provides empirical evidence to support these observations and establishes a foundation for further analysis while existing works [6][7] mainly focus on either Llama-2 7B or 13B. (3) Serious analyses on hard negatives: While previous studies have identified the existence of hard negatives (as we have cited), we go beyond this by showing that precision alone is insufficient to quantify their difficulty. We also investigate their correlation with retriever strength, providing deeper insights into the interplay between retrieval quality and LLM robustness, which is not shown in existing works. (4) Comprehensive RAG-specific finetuning study: We perform a comprehensive study of the various design components of RAG-specific LLM tuning (presented in Sections 5,6 and Appendices M and N), to provide design hints for researchers who are interested in implementing RAG-specific LLM tuning for their work or research. This is not studied in existing works.

• Your paper's actual technical contribution to the RAG community is still minimal.

We appreciate the reviewer’s feedback and would like to clarify the value of our work to (a) the RAG community, as well as the distinct technical contributions we make.

(1) Empirical study of the dynamics between retriever and LLM generator: In an RAG system, there is an inherent balance between the retriever and the LLM generator. A strong retriever can retrieve relevant information in a small top-k set, reducing the need for a highly robust LLM. Conversely, a weak retriever requires a larger top-k set to find useful passages, necessitating a stronger and more robust long-context LLM. While this relationship might seem intuitive, we are the first to empirically demonstrate these dynamics between different retrievers and LLM generators, as detailed in Section 3.1. This insight provides valuable guidance for optimizing the interaction between retrieval and generation components in RAG systems.

(2) Deep understanding of hard negatives. We acknowledge the reviewer’s broader observation that our work relates to improving the robustness of LLMs against noisy inputs. However, our contribution goes beyond prior research by offering a systematic analysis of hard negatives: We show that precision alone is insufficient to quantify the difficulty of hard negatives, emphasizing the need for a nuanced evaluation of retrieval quality. We analyze the correlation between retriever strength and the difficulty of hard negatives, providing deeper insights into the interplay between retrieval quality and LLM robustness. This level of analysis has not been addressed in existing works and offers actionable insights for designing more resilient RAG systems.

(3) Retrieval reordering: In scenarios where a strong retriever is unavailable (e.g., out-of-distribution domains for pretrained dense retrievers), increasing the number of retrieved passages is often necessary to achieve acceptable recall. In such cases, we introduce retrieval reordering (Section 4) as a novel solution. This method is simple, plug-and-play, and highly efficient; demonstrates significant performance improvements in challenging retrieval scenarios; and represents an effective strategy to bridge the gap between retrieval and generation components when dealing with suboptimal retrieval quality.

(4) Comprehensive RAG-specific finetuning study: Our work also conducts a comprehensive study of the various design components involved in RAG-specific LLM tuning, as presented in Sections 5, 6, and Appendices M and N. These include key factors such as data distribution, training context size, and evaluation settings, which provide insights and practical guidelines for researchers and practitioners aiming to implement RAG-specific LLM tuning in their systems. This study fills a gap in the current literature, providing valuable design hints and actionable recommendations for advancing RAG-specific LLM tuning.

• As other reviewers have pointed out, they also believe that your research is essentially about the robustness of RAG.

We appreciate the reviewer’s perspective and agree that, in a broader sense, our work contributes to improving the robustness of LLM generation against noisy inputs. However, as outlined in our previous rebuttal, our research distinguishes itself in several key ways, which have also been acknowledged by other reviewers: (1) Empirical study of the dynamics between retriever and LLM generator; (2) Deep understanding of hard negatives; (3) Systematic analysis of noise robustness across retrievers and LLMs; (4) Comprehensive RAG-specific finetuning study.

We would appreciate it if the reviewer could re-evaluate the contribution of the paper from both the long-context LLM perspective and the RAG perspective. We thank the reviewer again for your valuable feedback and welcome any further suggestions to strengthen our work.

[1] https://github.com/gkamradt/LLMTest_NeedleInAHaystack

[2] RULER: What's the Real Context Size of Your Long-Context Language Models?

We appreciate you taking the time to provide such thorough and insightful feedback! We address your comments below:

• The reason why Gemini-1.5-Pro does not exhibit the increase-then-decrease trend.

We hypothesize that the robustness of Gemini-1.5-Pro to hard negatives stems from its advanced training methodology and high-quality pretraining/post-training data. To further investigate the factors influencing LLM robustness, we designed experiments to analyze the impact of: (1) context window length, (2) model size, and (3) pretraining and post-training data.

Focusing on the first two factors, we conducted experiments on the NQ dataset using the e5 retriever and two groups of LLMs with varying model sizes and context window lengths: (1) 8K window LLMs: llama3-8B-Chat, gemma-2-9B-Chat and llama3-70B-Chat; (2) 128K window LLMs: llama3.1-8B-Chat, mistral-nemo-12B-Instruct and llama3.1-70B-Chat.

Our findings indicate no clear correlation between the context window size and robustness in increasing passage numbers. Contrary to expectations, the smaller context LLM (llama3-70B-chat with 8k) exhibited greater robustness than the larger context LLM (llama3.1-70B-chat with 128k). Furthermore, models with the same maximum context window length (e.g., llama3.1-8B-chat and Mistral-Nemo-12B-Instruct) demonstrated significantly different robustness levels.

These results suggest that factors beyond context window size, such as pretraining or post-training data, may play a crucial role in determining LLM robustness. We hypothesize that despite supporting larger context windows, LLMs might exhibit a bias towards shorter contexts due to the prevalence of shorter training samples. However, without access to the specific training data used for these LLMs, further investigation is needed to confirm this hypothesis. We acknowledge this limitation and recommend it as an area for future research.

• If LLMs larger than 30B are susceptible to hard negatives.

Thank you for raising this important question. To investigate whether LLMs larger than 30B parameters exhibit similar susceptibility to hard negatives, we conducted experiments with Llama-3.1-70B-Chat on the NQ dataset using both e5 and BM25 retrievers.

The results clearly demonstrate that even this larger LLM remains sensitive to hard negatives. The characteristic inverted U-shaped performance curve persists as the number of retrieved passages increases. This suggests that model size alone is not the determining factor in LLM robustness to hard negatives.

We suspect that the quality and nature of pretraining and post-training data play a more significant role. However, due to the lack of publicly available information regarding the training data used for these LLMs, we are unable to definitively confirm this hypothesis at present. We acknowledge this limitation and recommend further investigation into the role of training data as a promising avenue for future research.

Thank you for your response. Upon reevaluating the revised paper, I have identified some additional concerns. The paper primarily explores how, for many long-context LLMs, the quality of the generated output initially improves but then subsequently declines as the number of retrieved passages increases. However, it seems there might be more rational ways to address this:

1. The "hard negatives" phenomenon is caused by inaccuracies in the retriever. A simpler approach could be to enhance the quality of the Retriever. If the quality of the retrieval results is high enough, further denoising by the LLM may not be necessary, although the robustness of the LLM is still worth studying.
2. In terms of the number of retrieval passages, RAG does not need to retrieve such long passages in practical use.

Based on the two considerations mentioned above, I decide to lower my score from 8 to 6.

• The computational requirements of the intermediate reasoning approach with Gemini-1.5-Pro. How sensitive is the intermediate reasoning approach to the quality and style of reasoning labels? (W4, Q7)

You raise a valid concern about the cost of generating reasoning labels using a large language model like Gemini-1.5-pro, especially for large datasets. To address this, we conducted additional experiments where we fine-tuned the LLM using reasoning labels generated by a smaller and more cost-effective model, llama3-8B-chat. This allows us to assess the robustness of our proposed method to variations in the quality of reasoning labels.

TriviaQA

PopQA

The results demonstrate the resilience of our approach. While using labels from a powerful LLM like Gemini-1.5-pro yields the best performance, utilizing labels from a smaller LLM like llama3-8B-chat still delivers significant improvements. Specifically, it outperforms both RAG implicit tuning and the vanilla RAG baseline. This finding offers a valuable practical trade-off: users can opt for smaller, less expensive LLMs for generating reasoning labels depending on their budget constraints, while still achieving substantial performance gains with our proposed method.

• What are the key properties of "hard negatives" which make them particularly challenging? Is it semantic similarity, structural patterns, or something else? How does your analysis generalize to other types of retrievers not tested in the paper? (Q1)

To further illustrate the nature of hard negatives, we present detailed case studies in Appendix D. Our analysis reveals that many hard negatives are passages that exhibit some relevance to the question but ultimately do not contain the correct answer. These passages often mention entities present in the question, leading to a degree of semantic similarity, but they fail to provide the necessary information to answer it accurately.

Furthermore, to demonstrate the generalizability of the observed "inverted U-curve" phenomenon, we expanded our experiments on the NQ dataset to include additional retrievers beyond e5 and BM25, namely contriever and bge. Our findings consistently show the presence of the inverted U-curve across all these retrievers.

• The cost-benefit analysis compared with reducing retrieval size and basic reranking (Q4).

To provide a comprehensive assessment, we analyze the cost-benefit trade-offs between basic reranking, retrieval reordering (Ours), and RAG-specific LLM tuning (Ours).

Basic reranking. Cost: Reranking involves processing every query in each testing dataset with a computationally expensive cross-encoder model. This process can be time-consuming, especially for large datasets. In our experiments, reranking the top 200 passages on the TriviaQA dataset (over 11k samples) took over 1.5 hours (with bge-reranker-large [1]) using DataParallel and 8 H100s (while retrieval only (without reranking) with e5 and gpu-based approximated nearest neighbor search - faiss-gpu - only takes 10 mins). This cost scales linearly with dataset size, making it a significant bottleneck for large-scale applications. Benefit: Reranking can improve retrieval accuracy, leading to better overall RAG performance, as demonstrated in our results above.

Retrieval reordering (Ours). Cost: This method is designed to be plug-and-play and can be seamlessly integrated with any existing RAG pipeline (retriever, reranker, and generator) without incurring additional computational costs; Benefit: It consistently improves RAG performance, particularly when using a larger number of retrieved passages. Importantly, it is both retriever-agnostic and reranker-agnostic, demonstrating its versatility and broad applicability (as shown in Figure 4 and the table above).

RAG-specific LLM tuning (Ours). Cost: Training an LLM requires computational resources and time. In our experiments, achieving a well-trained checkpoint took approximately 2 hours using DeepSpeed zero3 and 8 H100s. Benefit: This method yields substantial and consistent improvements in the LLM's ability to handle RAG scenarios across different numbers of retrieved passages (as shown in the table above). Crucially, this training is a one-time cost; the resulting LLM can be deployed for any testing dataset without incurring further computational overhead during inference (retrieval with e5 and gpu-based approximated nearest neighbor search - faiss-gpu - only takes 10 mins on TriviaQA, while reranking with bge-reranker-large [1] takes more than 1.5 hours).

It's important to highlight that these three approaches are orthogonal and address different aspects of the RAG system: Basic reranking focuses on improving the quality of retrieved passages. Retrieval reordering optimizes the interaction between the retriever and the LLM generator. RAG-specific LLM tuning enhances the generator's capabilities for handling retrieved information. This inherent complementarity suggests that combining these techniques can lead to further performance gains in RAG systems.

• Lack of analysis of failure cases. When do these methods break down? What types of queries are challenging with hard negatives? (W3, Q5)

We provide detailed case studies in the appendices to further illustrate our findings: Appendix D: Examines retrieved hard negatives, Appendix I: Demonstrates the effectiveness of our proposed methods, Appendix O (revised manuscript): Presents new case studies focusing on the analysis of challenging queries.

Our analysis of these challenging queries reveals no easily discernible pattern that distinguishes those queries which are answerable with a small number of retrieved passages but fail with larger retrieval sets. We hypothesize that these failures stem from complex interactions between the query, the retriever, and the knowledge corpus. Specifically, if increasing the retrieval top-k primarily introduces noisy or irrelevant information, while the relevant information was already present in a smaller top-k set, the likelihood of the LLM failing to answer correctly increases significantly.

We appreciate your thorough review and insightful comments! To ensure clarity and conciseness, we have grouped similar points raised in the weakness and question sections and addressed them as follows.

• Why long-context LLMs behave differently with hard negatives (W1).

Thank you for your insightful question. We believe the observed differences in how long-context LLMs handle hard negatives could be attributed to three primary factors: (1) context window length, (2) model size, and (3) pretraining and post-training data.

To disentangle the influence of context window length and model size, we conducted experiments on the NQ dataset using the e5 retriever and two groups of LLMs with varying model sizes: (1) 8K window LLMs: llama3-8B-Chat, gemma-2-9B-Chat and llama3-70B-Chat; (2) 128K window LLMs: llama3.1-8B-Chat, mistral-nemo-12B-Instruct and llama3.1-70B-Chat.

Our findings indicate no clear correlation between the context window size and robustness in increasing passage numbers. Contrary to expectations, the smaller context LLM (llama3-70B-chat with 8k) exhibited greater robustness than the larger context LLM (llama3.1-70B-chat with 128k). Furthermore, models with the same maximum context window length (e.g., llama3.1-8B-chat and Mistral-Nemo-12B-Instruct) demonstrated significantly different robustness levels.

These results suggest that factors beyond context window size, such as pretraining or post-training data, may play a crucial role in determining LLM robustness. We hypothesize that despite supporting larger context windows, LLMs might exhibit a bias towards shorter contexts due to the prevalence of shorter training samples. However, without access to the specific training data used for these LLMs, further investigation is needed to confirm this hypothesis. We acknowledge this limitation and recommend it as an area for future research.

• Comparison with reranking approaches or hierarchical retrieval methods (W2, Q3).

Thank you for this valuable suggestion. Our focus in this paper is on analyzing and improving the performance of long-context LLMs as generators in RAG systems. While reranking and hierarchical retrieval are important techniques for enhancing the retriever component, they fall outside the primary scope of our work.

However, we recognize the value of exploring the interplay between our proposed methods and advanced retrieval techniques. To address your comment, we have included additional experiments on PopQA and TriviaQA incorporating a hierarchical retrieval component (referred to as reranking). Specifically, we first retrieve 200 passages using the e5 retriever and then apply reranking with bge-reranker-large [1]. Llama3-8B-chat serves as the generator in these experiments.

PopQA:

TriviaQA:

The results demonstrate two key findings: (1) Superiority of our proposed methods: Both RAG-specific implicit tuning and RAG tuning with intermediate reasoning significantly outperform the baseline RAG system with reranking. (2) Complementary benefits: Combining reordering with reranking (RAG w. reranking+reordering) further enhances performance, especially when retrieving a large number of passages, compared to reranking alone (RAG w. reranking). This analysis highlights the effectiveness of our proposed methods even in conjunction with advanced retrieval techniques like reranking.

• Comparison with RetRobust and RA-DIT (Q2).

Thank you for highlighting these relevant models. While both RetRobust and RA-DIT aim to improve LLM robustness in RAG contexts, they differ from our work in key aspects.

RetRobust focuses on mitigating the impact of irrelevant retrieved passages by employing strategies at the retrieval-level, primarily deciding whether to use retrieved information or not. In contrast, our approach delves deeper into passage-level relevance within long-context scenarios, aiming to equip the LLM with a finer-grained understanding of the retrieved information.

RA-DIT, on the other hand, explores dual training of the retriever and LLM but primarily focuses on scenarios with small retrieved contexts (less than 10 passages). Unfortunately, due to the lack of publicly available code and resources for RA-DIT, we were unable to include it in our comparison.

Therefore, we focused our comparative analysis on RetRobust, evaluating both its NLI strategy and LLM tuning strategy against our proposed methods on the NQ and PopQA datasets (retriever: e5, LLM: llama3-8B-chat).

NQ

PopQA

As the results demonstrate, our proposed RAG implicit tuning and RAG tuning with intermediate reasoning consistently and significantly outperform both RetRobust strategies. This highlights the effectiveness of our methods in enhancing LLM robustness to hard negatives, particularly in long-context RAG scenarios where the LLM needs to process and synthesize information from a larger number of retrieved passages.

• The choice of the reordering strategies (Q6).

You correctly identify the motivation behind our proposed retrieval reordering strategy: to counteract the "lost-in-the-middle" phenomenon inherent in LLMs. By strategically placing high-scoring (and thus, more likely relevant) retrieved passages at the beginning and end of the input sequence, we aim to capitalize on the LLMs' tendency to prioritize information in these positions.

While we acknowledge the potential of exploring alternative reordering patterns, our primary goal in this paper is to establish the significance of passage ordering in RAG systems and introduce the concept of "position engineering." Given the broader scope of our work, which encompasses a comprehensive analysis of long-context LLMs in RAG and the development of multiple solutions, an exhaustive investigation of various reordering strategies falls outside the current focus. We consider this a promising direction for future research and appreciate your insightful suggestion.

[1] Making Large Language Models A Better Foundation For Dense Retrieval. (Link: https://huggingface.co/BAAI/bge-reranker-large)

Thank you once again for your thoughtful feedback and active engagement during the discussion phase! We truly appreciate the time and effort you have dedicated to reviewing our work. As the discussion phase concludes in two days, we would like to highlight the updates and enhancements we have made to our submission based on your valuable feedback: (1) Deep Analysis of the Reverse U-Curve Phenomenon; (2) Comparisons with Reranking Methods; (3) Comprehensive Cost-Benefit Analysis; (4) Intermediate Reasoning Analysis; (5) Hard Negative Analysis; (6) Detailed Comparison with Existing Works; (7) Clarification of Reordering Strategies. We hope these updates address your questions and provide a clearer picture of the contributions and significance of our work. If there are any remaining concerns or additional points for discussion, we would be more than happy to engage further before the deadline. In light of these clarifications and improvements, we kindly request your consideration of a higher assessment for our submission. Your feedback has been instrumental in refining this work, and we are grateful for your thoughtful review.

We appreciate your insightful feedback and believe it has significantly strengthened our manuscript. We have carefully addressed each of your comments as detailed below:

• Difference between e5 curve and bm25 curve in Figure 1.

You correctly observe that the performance curves exhibit an inverted U shape with both strong (e5) and weak (BM25) retrievers. However, we emphasize that there are key distinctions between the two. As shown in Figure 1, the strong retriever (e5) generally achieves peak performance earlier (topk=10 for both Gemma-7B-chat and Gemma-2-9B-Chat) and then declines more sharply compared to the weak retriever (BM25, which peaks at topk=30).

This behavior, as explained in Sections 3.2 and 3.3, stems from the fact that irrelevant passages retrieved by the stronger e5 retriever have a more detrimental impact on performance. We have further clarified this observation in the "Insights" paragraph of Section 3.1 to highlight this important nuance.

• It is possible that hard negatives are ranked high.

You raise a valid point about the potential for certain hard negatives to be ranked highly by the retriever. While this can occur in individual cases, our analysis demonstrates that, on average, hard negatives are ranked lower than relevant passages.

To illustrate this, we conducted experiments on the NQ dataset with both e5 and BM25 retrievers, analyzing the top-k retrieved passages (k = 5/10/20/50/100). We calculated the average ranking of both relevant passages and hard negatives, first averaging at the query level and then across all queries.

e5

BM25

Our findings consistently show that the average ranking of relevant passages is higher than that of hard negatives for both retrievers and across all top-k values. This observation supports the rationale behind our retrieval reordering strategy: by prioritizing higher-ranked passages, we increase the likelihood that the LLM focuses on relevant information.

Furthermore, it's important to emphasize that the benefits of reordering extend beyond mitigating the impact of hard negatives. By placing higher-relevance passages in more prominent positions, we simultaneously reduce the influence of lower-relevance passages, leading to a more effective utilization of the retrieved information by the LLM.

• Knowledge base and the retriever for Sec. 5.

To enhance clarity and reproducibility, we have updated Section 5 to explicitly state that we use Wikipedia as the knowledge base and e5 as the retriever.

• Experiment on a corpus that is not in the pretraining data.

You raise a valid point regarding the potential overlap between LLM training data and Wikipedia. While the exact composition of these pretraining datasets remains undisclosed, it is important to note that even with potential overlap, LLMs often struggle to answer questions accurately without external knowledge (as demonstrated in Figure 5 when no passages are retrieved). Therefore, following established practices in RAG evaluation [1][2][3], we believe using Wikipedia as an external knowledge source remains a reasonable choice.

However, we acknowledge the value of exploring diverse knowledge sources to strengthen our findings. To address your suggestion, we have expanded our analysis to include RAG experiments on PubMedQA [4] and BioASQ [4], utilizing PubMed [5] as the retrieval corpus.

PubMedQA

BioASQ

These new experiments confirm the presence of the "inverted U-curve phenomenon" with both llama3.1-8B-chat and mistral-Nemo-Instruct as the generator across both datasets (with PubMed as the knowledge base and e5 as the retriever), further reinforcing our observations.

• Is there a correlation between context windows of LLMs and their robustness in increasing the number of passages? Would "infinite"-context window LLMs have the most robustness?

This is an insightful observation, and we appreciate you bringing it to our attention. To investigate the potential correlation between LLM context windows and their robustness to increasing numbers of retrieved passages, we conducted experiments on the NQ dataset using two groups of LLMs with varying context window sizes: (1) 8K window LLMs: llama3-8B-Chat, gemma-2-9B-Chat, and llama3-70B-Chat; (2) 128K window LLMs: llama3.1-8B-Chat, mistral-nemo-12B-Instruct and llama3.1-70B-Chat.

Our findings indicate no clear correlation between the context window size and robustness in increasing passage numbers. Contrary to expectations, the smaller context LLM (llama3-70B-chat with 8k) exhibited greater robustness than the larger context LLM (llama3.1-70B-chat with 128k). Furthermore, models with the same maximum context window length (e.g., llama3.1-8B-chat and Mistral-Nemo-12B-Instruct) demonstrated significantly different robustness levels.

These results suggest that factors beyond context window size, such as pretraining or post-training data, may play a crucial role in determining LLM robustness. We hypothesize that despite supporting larger context windows, LLMs might exhibit a bias towards shorter contexts due to the prevalence of shorter training samples. However, without access to the specific training data used for these LLMs, further investigation is needed to confirm this hypothesis. We acknowledge this limitation and recommend it as an area for future research.

• Fig. 7b indicates that there might not be much impact on the choice of the retriever.

We interpret these results from two key perspectives:

Retriever-Agnostic Improvement: Our proposed RAG-specific tuning consistently enhances LLM performance across all retrievers, demonstrating the generalizability of our approach. This improvement is independent of the specific retriever used, highlighting the retriever-agnostic nature of our fine-tuning method.

Superiority of Mixed Hard Negatives: LLMs trained with RAG-specific data incorporating hard negatives from multiple retrievers outperform those trained with data from a single retriever. This observation underscores the importance of exposing the LLM to diverse hard negatives during training. It suggests that "hard negatives" exhibit retriever-specific characteristics, and training with a mix of retrievers equips the LLM to handle a wider range of challenging examples. This finding also hints that independent improvements in retrievers might have limited impact, emphasizing the importance of a joint end-to-end design of retrievers and LLMs in RAG systems.

[1] RA-DIT: Retrieval-Augmented Dual Instruction Tuning. ICLR 2024.

[2] Making Retrieval-Augmented Language Models Robust to Irrelevant Context. ICLR 2024.

[3] RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs. NeurIPs 2024.

[4] Benchmarking Retrieval-Augmented Generation for Medicine. ACL 2024.

[5] PubMed: https://pubmed.ncbi.nlm.nih.gov/