LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs – No Silver Bullet for LC or RAG Routing

We appreciate the reviewer's constructive feedback, which helps enhance the clarity of our paper! Let us answer your question below.

More details about the QA generation process and answer to question “Could it be that the generation produces wrong answers, and the final LC or RAG answers are actually correct?”

Thanks for this advice. We conducted human inspection after the generation. After constructing prompts and example QAs, we sample 40 generated QAs for each context type and task, which are then manually verified for validity. We only stop modifying the prompts and in-context examples for a given context type and task when the accuracy reaches above 90%. Additionally, larger models compared to smaller ones, and stronger proprietary models compared to open-source models, consistently demonstrate higher accuracy, which further validates the overall correctness of the QAs. If the correctness of QAs were not guaranteed, we would likely observe random results. While this process cannot ensure that all QAs are 100% correct, it does guarantee a very high accuracy rate, making them effective for evaluation purposes. We will include these details in our revision.

The impact of retrieval strategies and answer to question “What role does retrieval play in comparing LC vs. RAG?”

We agree that comparing RAG and LC is influenced by many factors, and a complete advanced RAG system can be more complex and have more modules, including query rewrite, different retrieval strategies, reranking, summarization, etc. However, we would like to highlight that this does not affect the value of our work from two perspectives. First, [5] conduct a systematic analysis of RAG implementations, and we adopt their advice to use a hybrid search strategy combining vector search and BM25. We choose gte-1.5 [4], a very strong embedding model released in late 2024, for search and reranking, ensuring that our RAG implementation already employs a strong strategy. On the other hand, our benchmark itself is one of the core contributions, providing effective support for future systematic exploration of the impact of different RAG modules in long-context QA.

Below are the experimental results of replacing gte-1.5 with bge-m3 [6] and adding Recomp [7] as a summarization module (also suggested in [5]). As can be seen, further complicating the RAG process does not bring significant additional gains, but instead makes retrieval a computationally-intensive process.

Is it possible that GPT-4o creates more complex questions than GPT-4o can answer?

Yes, this holds true under our generation process, as we employ a short-context generation method for creating QAs. At Line 244, we mention that generating QA pairs for long texts is inherently a long-context generation problem. To improve generation quality, we divide long contexts into several short segments and generated QA pairs based on these individual segments. This means that GPT-4o only needs to process a short context when generating QAs, but when answering, it needs to find answers from the complete context, which is far more challenging than generating them.

What is the stance of prior literature on your results?

While [1] observes that longer contexts pose challenges for RAG, our work provides a more nuanced analysis, demonstrating that RAG performance is comparable to LC LLMs on tasks such as location and hallucination detection. [2] states that open-source models perform poorly on QA, so we discuss the results of open-source and proprietary models separately in our paper. [3] finds that more chunks can increase performance of RAG; we also conduct related experiments in Figure 2 to verify the impact of using more chunks. Detailed discussions of these connections to prior work will be added to the appendix in the final version.

References

[1] Leng, Quinn, et al. "Long context rag performance of large language models." arXiv 2024.

[2] Schimanski, Tobias, et al. Towards faithful and robust llm specialists for evidence-based question-answering. arXiv 2024.

[3] Li, Zhuowan, et al. Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach. EMNLP 2024.

[4] Zhang, Xin, et al. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. ACL 2024.

[5] Wang, Xiaohua, et al. Searching for best practices in retrieval-augmented generation. EMNLP 2024.

[6] Chen, Jianlv, et al. Bge m3-embedding: Multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation. arXiv 2024.

[7] Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval-augmented lms with compression and selective augmentation. ICLR 2024.

We appreciate the reviewer's constructive feedback, and we are very happy that the reviewer liked how our dataset is structured! Let us answer your question below.

RAG retrieval setup: the potential unfairness in the RAG setup, as it relies on a limited number of chunks without further experimentation

Yes, in the main results, each query retrieves 5 chunks. The specific method is to search for 5 chunks based on similarity using the GTE embedding model and search for 5 chunks using BM25, then take their intersection. If fewer than 5 chunks are found, the remaining chunks are equally selected from the two retrieval methods. We adopt this hybrid search strategy with reference to [1], which is a strong method.

In Figure 2 of the draft, we provide experimental results with different chunk numbers and sizes, comparing the impact of the amount of retrieved information on smaller models (Qwen-2.5-7B) and larger models (Qwen-2.5-72B). We gradually increase the number of chunks from 5 to 30. Since 30 retrieved chunks means the total input length has already reached 30 * 600 = 18,000, which is on the same order of magnitude as long-context, further increasing the number of chunks would cause RAG to lose its significant efficiency advantage. We set 5 as the default number of chunks and 600 as the chunk size primarily by referencing previous works [1, 2].

We further explore the impact of increasing the number of chunks to 100 on these two models with 128k context, and the results are provided below. We find that too many chunks can cause RAG's performance to decrease rather than increase.

The definition of "long context" needs further clarification, as the experiments are capped at 128k context. What happens beyond this limit, and does RAG remain relevant in such scenarios?

This is one of the key designs of our benchmark. Under extremely long context, RAG has an overly obvious advantage. An extreme example is knowledge base-based QA, where LC LLM cannot process the entire knowledge base and struggles to rely on external knowledge to answer correctly.

As emphasized in the paper, when collecting contexts, we choose texts that are as close as possible to the LLM's limit without exceeding it. This allows for a fair comparison of the actual capabilities between RAG and LC LLMs. If the context exceeds the LLM's input limit, we need to employ tricks like truncation, which could result in the answer to a question not being present in the LLM's input, thus failing to reflect the LLM's true capabilities. In Table 1, we conduct relevant experiments to verify the impact of such excessively long contexts. In lines 127-149, we specifically analyze why these overly long contexts cannot be appropriately used to compare RAG and LC LLMs in long-context QA scenario.

We will clarify this point more clearly in the final version.

The framing of the "Hallucination detection" task

Thanks for this suggestion. We will rename "hallucination detection" to the more appropriate term "hallucination occurrence" to express whether RAG and LC LLM produce hallucinations.

References

[1] Wang, Xiaohua, et al. Searching for best practices in retrieval-augmented generation. EMNLP 2024.

[2] Li, Zhuowan, et al. Retrieval augmented generation or long-context llms? a comprehensive study and hybrid approach. EMNLP 2024.

We would like to Thank you for your constructive review, as well as your positive feedback! Let us answer your question below.

Some implementation details are missing from the main paper---How many chunks were used in the setting of RAG experiments?

In Section 4, paragraph “Implementation of RAG (line 269)”, we provide the details that "Our evaluation employs a standardized configuration with a chunk size of 600 tokens, 5 chunks per document, and an overlap of 100 tokens between chunks." In the main results, we use a chunk size of 600 and 5 chunks. Additionally, we further explore the impact of more chunks in Figure 2.

Some parts (with 2024 knowledge) of the benchmark may not be less useful for future LLMs.

For future LLMs, all existing benchmarks may become outdated due to information leakage when they are used to train newer LLMs. However, on one hand, we have already used nearly the most recent corpus in our data selection, with almost all contexts being released in the second half of 2024, except for novels. On the other hand, in addition to providing datasets for evaluation, we also provide concrete pipeline for creating new data, which can be used to generate new datasets with more updated contexts.

Is hallucination detection a valid task? How do we know the performance change is not purely because of the change in context length?

Hallucination remains a significant challenge for LLMs. While RAG can potentially mitigate this issue, we provide a quantitative assessment of RAG's effectiveness in reducing hallucinations specifically in long-context scenarios. Our evaluation spans models of various sizes, including both proprietary and open-source LLMs, comparing RAG against standard long-context input. Our experimental design focuses on measuring models' ability to abstain from answering when presented with unanswerable questions—defining abstention as correct behavior and hallucination as incorrect.

The experimental results yield three key findings: (1) LC LLMs are substantially more susceptible to hallucinations compared to RAG; (2) Model strength does not correlate with reduced hallucination rates that stronger LLMs do not demonstrate fewer hallucinations; (3) Increasing context length corresponds to higher hallucination probability. Two of three findings are independent of the increase in context length, therefore we believe this is an effective task that provides strong support for the study of hallucinations in RAG and LC LLM.

In Line 367, are the results for 128k and 32k contexts directly comparable? Do they share the same input?

Yes, they are directly comparable. At Line 244, we mention that generating QA pairs for long texts is inherently a long-context generation problem. To improve generation quality, we divide long contexts into several shorter segments and generated QA pairs based on these individual segments. This means that for both 32K and 128K contexts, GPT-4o generated QA pairs using the same prompts and similarly sized segments. Therefore, the distribution of these QA pairs can be considered approximately equivalent across different context lengths.

Specifically, in lines 249-252 we wrote "we split the long context into multiple segments, each approximately 10k tokens in length, and input them individually into GPT-4o to generate QAs." In lines 267-273, we clarified that we have different segmentation strategies for different types of contexts. We will clarify this point more clearly in the final version.

Are there task-specific scores for the results in Figure 2? Do different tasks share the same trend?

We find that the trends across different tasks are generally similar, but there are some exceptions. Below we provide the results of Qwen-2.5-72B at 32k length. We find that for the location task, novels perform worse than other context types, possibly because novels contain more similar content, making it difficult to locate answers. For the reasoning task, papers performed best, which we speculate is because papers have stronger logical structure, lower information redundancy, and are more conducive to reasoning. We will add a systematic analysis of performance across different tasks on various context types in the appendix.

Thank you very much for your positive review! Please see our responses below.

Discuss with [1]

Thanks for pointing out this important related work. [1] mainly investigated the phenomenon that increasing the number of retrieved passages does not consistently improve the performance of LLMs. As the amount of retrieved information increases, performance first increases and then decreases. Some of our experimental results align with the conclusions in [1]. In Figure 2, we observe that as the number of retrieved chunks and chunk size increase, LLM performance first improves and then declines. Furthermore, weaker models compared to stronger ones, such as Qwen-7B versus Qwen-72B, begin to show performance degradation earlier, indicating that weaker models are more susceptible to the influence of large amounts of irrelevant noise in the retrieved information. We will add the discussion with [1] in the revision.

We further explored the impact of increasing the number of chunks to 100 on Qwen-2.5-7b-instruct and Qwen-2.5-72b-instruct with 128k context and find that too many chunks can cause RAG's performance to decrease.

I would recommend the authors add another “RAG” and “LC” column in Table 2 to make things clearer.

Thanks for this advice, and we will change it in our revision!

Typos

Thanks for pointing out these typos. We have fixed them in the revision and will keep polishing our paper.

References

[1] Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG. ICLR 2025.