Sufficient Context: A New Lens on Retrieval Augmented Generation Systems

Thank you for your time, feedback, and positive review! We respond to your comments and questions below. To answer your questions, we have included new discussions about our autorater and its broad applicability for both practical systems and research endeavors.

How is "sufficient context" defined for multi-hop reasoning? Clarify the specific criteria or thresholds the autorater uses, especially for complex multi-hop queries.

The criteria for multi-hop is the same as for single-hop. The context must have enough information to answer the question. In the multi-hop setting, the context needs to have answers to the intermediate questions when it is necessary. Since the main autorater we use is a prompted version of Gemini, we do not need to include precise rules to differentiate between single-hop and multi-hop questions.

How does the autorater handle ambiguous or conflicting contexts? What rules are applied when contexts contain contradictory or unclear information?

This is a very good question. The case of contradictory information requires handling on a case-by-case basis. For example, if the context contains multiple valid answers to the question, then this is the “ambiguous context” case, and the context is sufficient. If the context contains unclear information, where the question is not definitively answerable, then it is insufficient.

How does retrieval quality impact context sufficiency and model performance? Could improved retrieval reduce hallucination rates by providing more sufficient contexts?

Yes, absolutely. This is one of the main takeaways from our work, coupled with the issue of models hallucinating even with sufficient context. At a high level, the best solution to improve RAG performance is to develop a way to get a very high percentage of sufficient context. The models all have a higher % of correct responses with sufficient context. Nonetheless, another important finding is that models hallucinate 10–26% of the time even with sufficient context. Hence, even perfect retrieval will not lead to completely factual answers! This implies that there is more work to be done around pre-training or fine-tuning LLMs to improve the utilization of the retrieved contexts.

What are the computational demands of the selective generation framework? Is it feasible for real-time applications, particularly with large models?

Yes, it is indeed feasible to incorporate with very large models! The two components, model confidence (i.e., P(Correct)) and the sufficient context label are both quite efficient. To get P(Correct), we can use the same generation step as the answer generation, where we also get a confidence score along the way. To get a sufficient context label, we use the FLAMe 24B model, which is significantly more efficient and cheaper to run than the largest LLMs.

Is the sufficient context framework applicable beyond QA? Could this framework be useful for other retrieval-augmented tasks, like summarization?

Yes, it is definitely applicable beyond QA. We are working on applications now as a follow-up work. Summarization is a good candidate. Another option is search queries, which are information-seeking but may not be fully formulated questions. A final use case is as a re-ranking signal after retrieving a large number of supporting documents. Overall, we believe sufficient context and follow-up ideas will be instrumental in shaping the future of RAG research, for academic and enterprise applications!

Dear Reviewer BUto,

As the rebuttal deadline approaches, please kindly check the papers' discussion threads and respond to the authors' rebuttals. If you haven't had a chance to respond yet, I’d greatly appreciate your input soon. Your insights are invaluable to the authors and the review process.

Thank you for your effort and support!

Best regards,

Area chair

Thank you for your time and feedback! We respond to your comments and questions below. To answer your questions, we have (a) clarified our research questions, (b) addressed a few potential misunderstandings about our definition and autorater, and (c) included new experiments.

What is it really that this paper shows that wasn't already known? Perhaps a more explicit formulation of hypotheses/research questions that have not yet been answered could help here.

Our work starts with the overarching question:

1. Can we develop a principled and scalable way to analyze RAG responses in a way that separates retrieval issues from response generation issues?

Then we move on to

2. How can we use insights from (1) to improve generation in RAG systems?

To address (1), we first develop and validate a new autorater for classifying sufficient context. As the ground truth may not be available in production, we focus on a signal that only depends on the question and context. Our autorater enables a scalable way to investigate retrieval failures across multiple datasets and models (without relying on ground truth answers). Using our autorater, we find new trends: (a) SOTA LLMs hallucinate even with sufficient context, (b) LLMs hallucinate rather than abstain with insufficient context, (c) sufficient context auto-rater labels can improve selective generation. For (2), we use our analysis in Section 4 to inform the interventions and improvements we propose in Section 5. Overall, we believe our sufficient context idea and intervention methods will resonate with engineers who build enterprise RAG systems. The distinction between relevance and sufficient context is subtle but integral to building factual generative search.

To assess which contexts are sufficient, the paper relies on LLM-generated assessments. This is a key limitation, and the current analysis should be expanded with a human-annotated experiment . . . What can we really conclude without manual validation of the auto-rater predictions?

We include a human-annotated experiment to validate the autorater in Table 1, where the LLM-based autorater is 93% accurate on representative examples. Overall, your question highlights the broader challenge of relying on LLM-based labeling versus human labels to analyze LLM behavior. We agree that this approach merits caution – it is important to verify that our observations are not just artifacts of auto-rater labeling. In this work, we can make conclusions without manual validation because the insights in the paper are robust to a small % change in the sufficient context labels.

Based on your comment, we conducted new experiments about model accuracy on data with human-labeled sufficient context labels – please see the tables and discussion in our overall response to all reviewers.

The notion of "sufficient context" is central to this paper, but it is not actually defined in a very precise way. First, the definition (line 157) is rather "circular": to determine if a context is sufficient, we need to determine if it allows us to infer the answer, but that actually depends on the capabilities of the LLM.

We want to clarify this point, as we think this may reflect a misunderstanding. The definition of sufficient context we put forth is independent of the ability of the LLM that answers the question. We aim to rate questions in a way that applies to LLMs, humans, and any other systems that use retrieval to answer a question.

The definition is also confusing, as it talks about "plausible answers" rather than the actual answer. I think what is meant is that a context can be sufficient even if the answer it contains is incorrect.

Your understanding here is accurate – the context can be sufficient even if it contains an “incorrect” answer. This is a key requirement, because (i) we would like to be able to use signal from sufficient context at inference time, where we do not have ground truth answers (see Section 5.1) and (ii) we hope to elucidate findings that are robust to ground truth label noise.

Is the difference between context sufficiency and NLI based methods really that significant (for practical purposes)?

The core difference is that NLI-based methods need a ground truth answer. Hence, they do not apply to on-the-fly analysis for RAG systems. This is critical for using sufficient context at inference time, as in Section 5.1, or when the ground truth is not available, as in production systems.

The definition assumes that an answer can be inferred up to 4 inference hops. Why 4?

Thank you for raising this. While this may seem arbitrary, existing RAG benchmarks have questions with 2–4 inference hops by design (e.g., Musique, HotPotQA). Hence, we limit the scope of sufficient context to match these. We will clarify this in the updated version.

minor issues with the writing and with formulations

Thank you for your careful reading and for flagging these issues! We’ve corrected these.

What are the key insights that your analysis has revealed that weren't known from previous work?

One of the key insights is that SOTA LLMs tend to hallucinate rather than abstain when given insufficient context. This was not known before, given that prior work did not distinguish between sufficient context with distracting information and insufficient context. This insight also informed our approach for selective generation (Section 5.1) and fine-tuning (Section 5.2), where we cannot just use sufficiency as a signal by itself for the model to answer or abstain. Another insight, as we mentioned above, is that models hallucinate on a non-trivial fraction of examples with sufficient context. Since you mention prior work, we want to point out that previous papers focus more on robustness to distracting content rather than distinguishing model behavior based on context sufficiency. For example,

• Xie et al (ICLR 2024) show that “LLMs can be highly receptive to external evidence even when that conflicts with their parametric memory.” which is a different goal from our paper.
• Wu et al (COLM 2024) show that “current LLMs still face challenges in discriminating highly semantically related information…. current solutions for handling irrelevant information have limitations in improving the robustness of LLMs to such distractions” which, in our language, is a study of robustness to different types of insufficient context.
• Yoran et al (ICLR 2024) “propose a method for automatically generating data to fine-tune the language model to properly leverage retrieved passages, using a mix of relevant and irrelevant contexts at training time.” Their source of relevant vs. irrelevant contexts involves high-ranked vs. low-ranked search results, and hence, this does not accurately separate sufficient vs. insufficient context.

In general, compared to these works, we add new insights about RAG system analysis, including the breakdown into accurate, abstain, and hallucinate, as opposed to just correct vs. incorrect. It is possible we failed to find or cite highly related work in our paper – if you find this to be the case please let us know!

Thank you for your time and feedback! We respond to your comments and questions below. To answer your questions, we have (a) clarified how our work generalizes across IR subsystems, (b) addressed a few potential misunderstandings about our definition and autorater, and (c) included new experiments with human labels.

How sensitive is the analysis to the choice of REPLUG and/or E5 as retriever components? The paper glosses over the IR subsystem here in an extreme way.

Thank you for this question – you raise an important point about how our work relates to research on information retrieval systems. We would argue that our paper generalizes beyond the IR subsystem rather than glosses over it. Concretely, we believe splitting into sufficient vs. insufficient context is a principled way to ablate the retriever since all retriever and re-ranker methods will produce a mix of sufficient and insufficient context. This is why we separately analyze both context types and derive insights into how LLMs respond in each case.

We also want to clarify only HotPotQA uses the REPLUG retriever. For FreshQA, we use the first 6000 tokens of the oracle URLs. For Musique, we use the provided context in full (which has 20 paragraphs, some of which are relevant to the question).

How good is the autorater? … Given that the autorater is not perfect, and given the subtle nature of its job (as discussed in the Remarks under the definition of Sufficient Context), how correlated are autorater mistakes with determinations of Correct/Hallucination in Figure 3?

This is a good question. We include a human-annotated experiment to validate the autorater in Table 1, where the LLM-based autorater is 93% accurate on representative examples. We also provide new experiments about model accuracy on data with human-labeled sufficient context labels in the overall response to all reviewers (see the added tables for Correct/Abstain/Hallucination). We corroborate our findings with this added experiment.

Your question highlights the broader challenge of relying on LLM-based labeling versus human labels to analyze LLM behavior. We agree that this approach merits caution – it is important to verify that our observations are not just artifacts of the auto-rater labeling. In this work, we can make conclusions without manual validation because the insights in the paper are robust to a small % change in the sufficient context labels. Even if the LLM autorater misclassified some examples as sufficient vs. insufficient, the findings still hold.

Will the 115 data points be released?

We would like to release the data, assuming we can get approval to release the dataset or a similar version. The challenge is that the context contains snippets taken from a variety of sources, which makes licensing complicated. As a workaround, we are currently working on an experiment to human-label a large subset of questions from Musique to verify the performance of the autorater, but this experiment may not finish before the end of the discussion period.

There's some risk of circularity here. What makes large LLMs so prone to hallucination if one could trust that a simple prompt could reliably discover these cases?

Thank you for this question. We want to clarify that we do not believe there is circularity here, but rather we discover a new disparity between LLM capabilities as an autorater vs. as a generation model. Indeed, we believe that the sufficient context autorater hallucinates less than the question-answering LLMs. Given that they are different tasks, there is no immediate contradiction here. Moreover, this disparity of capabilities can hold true for humans as well. It is believable that a person could achieve high accuracy on the binary labeling task of sufficient vs. insufficient context. But the same person may have lower accuracy in a free-response test that requires filling in the answers to questions.

Overall, it is a wonderful, and deep, question to ask “What makes large LLMs so prone to hallucination?” With a grain of salt, our intuition is that the model doesn’t have a good internal mechanism to decide whether to abstain or to answer the given question. Hence, the model starts to generate a response based on the context, even when it is insufficient.

Section 5 in our paper discusses ways to improve RAG systems by capturing the impressive capability of LLMs to rate sufficient context. For example, we show in Section 5.1 that we can do better with selective generation by using confidence and sufficient context as signals to decide whether to abstain or answer. In Section 5.2, we use the sufficient context labels to modify the fine-tuning data mixtures.

Thank you for your review and positive characterization of our work! We respond to your comments and questions below. To answer your questions, we have (a) clarified details about our autorater and error analysis, (b) addressed your comments about dataset release, and (c) included new experiments.

Can the automatic judgments be trusted out-of-domain?

This is a great question! Out of the box, the prompt we use for sufficient seems to work well for open-book QA questions. We have manually verified dozens of examples from FreshQA, Musique, and HotPotQA and found the accuracy to be consistent with the results in Table 1. We are also working on an experiment to human-label a large subset of questions from Musique to verify the performance of the autorater, but this experiment may not finish before the end of the discussion period. We have also looked at many more out-of-domain questions, including search queries or enterprise RAG systems (e.g., internal documents), and we find that small variations in the prompt usually suffice to achieve good performance here as well.

Since you observe rater error in Table 2, what is the estimated prevalence of this category?

In this category we have grouped together two sources of errors: (1) For the sufficient context autorater, the error rate is about 7%, which seems consistent across both the human-labeled data and the other datasets. (2) Another error rate is when the LLM Eval labels an answer as Correct when instead it should be Abstain or Hallucinate. Our observation is that this error is rare, and other papers (e.g., Krishna et al) find it to be 0.5% based on human quality checking. We have also taken care that our main conclusions remain valid even accounting for the prevalence of these two types of errors.

One other potential mechanism of the correct answer given insufficient context is that there are only one/few entities of the expected type (given the question) in the context. Is this mechanism never observed?

This is a good point. When we have observed this, it falls into the category of “Multi-hop: fragment” where the model assumes that the entity of the expected type is also the answer to the question by inferring an extra reasoning step to connect the dots. We will look more closely at the data and add another row to the table if this mechanism is observed in more cases.

What is the context source for FreshQA? Are there no examples of insufficient sources for FreshQA questions in the automated evaluation dataset?

Yes, the context sources for FreshQA are “oracle” URLs that support the answer. However, in some cases these sources are not complete, leading to insufficient context. This happens because the sources change or because the first 6k tokens in the source do not suffice to answer the question. For the automated evaluation dataset, we also included some examples where we manually modified the context where we deleted or rephrased the context to make it sufficient or insufficient. This also made the dataset roughly balanced.

While the paper spends a lot of time analyzing the case of correct answer with insufficient context, the opposite case of incorrect answer with sufficient context is equally interesting and warrants exploration

We agree! We investigated cases where the autorater labels an instance as having sufficient context while the LLM evaluator marks the answer as incorrect. One source of these discrepancies occurs when the ground truth answer conflicts with the answer provided in the source. Another source of errors arises when the autorater correctly identifies that the necessary information is present, but the model fails to properly compose the information (e.g., in multihop questions or questions requiring arithmetic). In a substantial number of cases, however, determining the source of the error proves challenging. Thank you for expressing interest in this question – we’ve added commentary on this topic to the current version.

Do you plan to release the sufficient context dataset?

We would like to, assuming we can get approval. The challenge is that the context contains snippets taken from a variety of sources, which makes licensing complicated. As a workaround, we are currently working on an experiment to human-label a large subset of questions from Musique to verify the performance of the autorater, but this experiment may not finish before the end of the discussion period.

Lines 185-190: Remark about ambiguous queries: It is unclear to me why the given example illustrates how ambiguous queries are handled.

Thank you for pointing this out. The remark titles were swapped (e.g., the remark labeled ambiguous queries had the example for ambiguous context). We have updated this.

Dear Reviewer,

Thank you again for your positive review! We are pleased you found that "The paper reports findings of unexpected model behavior with and without sufficient context, which are valuable to the community."

We hope that we have addressed your thought provoking questions through our responses above. As the discussion period is coming to an end soon, please do not hesitate to share any remaining comments you may have.

Best, Authors

Thank you for clarifying my questions.

I have raised my concern about the new results under the common response. I believe that the discrepancy there is at odds with your comments about the reliability of the LLM QA evaluator. This discrepancy raises questions about the main results and the importance of the human analysis of "correct answers with insufficient context".

Re: Reliability of the LLM QA evaluator: How does the QA evaluator behave when comparing against the ground truth vs an NLI-based evaluator (as suggested by Reviewer pMDo)? The NLI evaluator could serve as a way to judge the correctness of the LM-generated answer against the document context. This is different from the LLM QA evaluator because it would not be based on the ground truth answer. Looking at both evaluators may help to address the discrepancy.

As such, I believe that the main message is very valuable: "judging sufficiency correctly does not translate to being able to abstain. But given the mismatch, I am decreasing my score to 6.

Am I correct in comparing Table 1 and 2 to Figure 3 top and bottom row? If so, then I feel that the new results raise more questions about the reliability of the auto-raters.

The (your) overall pipeline for categorizing questions depends on (1) sufficient context judgment and (2) LLM QA evaluator (which compares the model response against a ground truth). Errors in the two stages can cascade. Your new results mitigate (to some extent) the errors in stage 1 by using the human-rater sufficiency labels. However, they are drastically different from the results with the autorater.

E.g., the rate of correct answers with insufficient labels for Gemini 1.5 Pro is 40-50%, according to Figure 3, but ~10% according to Table 2. Similar discrepancies exist in Table 1.

In light of the new results, how can we trust the main results of the paper?

We also want to respond directly to your questions about LLM QA evaluation by comparing models using a common lexical matching metric. Specifically, the comparison metric uses a deterministic measure that labels LLM responses based on whether they contain the ground truth answer or not. We use LLMEval to refer to the LLM QA Evaluator, and we record the fraction of instances labeled as “Correct” by the LLM. We refer to the deterministic measure as “Contains Answer” or just “Contains” for short, and we measure the fraction of responses that contain one of the ground truth answers for the question or false otherwise.

We believe that the LLM QA Evaluator should be more aligned with human expectations for labeling correctness as this task is quite simple: the model only needs to compare the LLM response to a set of possible ground truth answers. To verify this, we also include several examples below where the two metrics disagree. These examples illustrate the inherent limitations of a lexical matching approach.

Here is a table of results comparing model performance on 3 datasets, split based on our sufficient context autorater. This directly matches the setup for the results in Figure 3, where the LLMEval numbers correspond to the correct percentage (green bars in Figure 3).

The ‘Contains Answer’ evaluation is generally stricter than the LLM Eval, as it misses correct answers with different formatting. For example:

Question: What date did the creator of Autumn Leaves die?
Ground Truth Answer: 13 August 1896
Model Response: August 13, 1896.
Contains Answer: False
LLM Eval: Correct

Checking ‘Contains Answer’ also fails on answers that don’t match exactly but are semantically correct. For example:

Question: What former Los Angeles Lakers majority owner is the father of Jeanie Marie Buss?
Ground Truth Answer: Gerald Hatten Buss
Model Response: Jerry Buss.
Contains Answer: False
LLM Eval: Correct

Using ‘Contains Answer’ also occasionally marks answers as “Correct” due to the nature of the response. This happens more frequently with smaller models (e.g., Gemma), which are more likely to output poorly formed answers. This example is from Gemma 27B (insufficient context)

Question: What is Amazon Prime Video's most watched premiere ever?
Ground Truth: The Rings of Power
Model Response: The series explores the forging of the Rings of Power, the rise of Sauron, and the last alliance of Elves and Men.
Contains Answer: True
LLM Eval: Hallucinate

Two of our main insights are that (i) LLMs hallucinate quite a bit even with sufficient context, and (ii) LLMs struggle to abstain with insufficient context, which leads to a much higher hallucination rate with insufficient context. Importantly, our conclusions about both (i) and (ii) remain true even if we account for rater errors. Moreover, if we use `Contains Answer’ as the response evaluator, then we still see (i) and (ii). Together, these build trust in our paper’s conclusions.