MSRS: Evaluating Multi-Source Retrieval-Augmented Generation

Thank you for reviewing our work! We would like to address your concerns and questions as follows:

For OdSum-Story, the original long documents have been segmented into shorter documents using natural HTML delimiters such as hr: have these been used consistently by the authors? What is the variance over the length?

We segmented the original story texts from SQuALITY into the chapter documents of ODSum-Story using HTML delimeters. Specifically, we applied the following procedure uniformly across the dataset: split stories by <hr class="chap"/>, strip leading white-space characters and HTML tags, and collect the non-empty segments as chapters. To verify the consistency and quality of our segmentation process, we report the following statistics for ODSum-Story.

• Number of stories: 127
• Number of documents (i.e. chapters): 1,138
• Average number of chapters per story: 8.96
• Average document length: 556.93 words / 845.44 tokens
• Global standard deviation of document lengths: 382.07 words / 577.29 tokens
• Average per-story standard deviation of document lengths: 307.28 words / 465.31 tokens

As shown in the provided histogram (anonymous link: https://postimg.cc/WF9vRhMQ), document lengths are tightly clustered around the mean, indicating that most segments are of comparable length. These statistics highlight the natural distribution of chapter lengths in SQUaLITY, which our segmentation process preserves in ODSum-Story.

And, more critically, who has decided what are the ground-truth documents for a specific query? If I understand correctly, this is the key annotation for the performance evaluation of the retrievers.

We fully agree that the ground-truth annotation is central to the evaluation of retrieval performance, and we discuss this point further in a later response. Regarding ODSum-Story, each query is derived from SQuALITY, where queries are answerable using chapters within a single story. Accordingly, we define the ground-truth as the full set of chapters from the sole relevant story.

The description of the construction of OdSum-Meet addresses query and summaries, but not the segmentation into documents and the annotation of the ground-truth set of documents.

For ODSum-Meet, the segmentation and ground-truth annotation were derived from the query clustering process described in Appendix B. Each QMSum query is associated with an annotated set of relevant meeting transcripts, which we use without modification as the documents in ODSum-Meet. We cluster similar QMSum queries based on embedding similarity to form a broader, merged query. Our human verification in Table 3 confirms the high quality of the queries and summaries formed through this process. The ground-truth set for each merged query is defined as the union of the ground-truth documents for the constituent queries.

Also the human evaluation of these datasets does not seem to have assessed the correctness of the ground-truth annotation (yes for precision, not for recall). Overall, it looks as if the hardest part of the retrieval annotation has been skipped.

For ODSum-Story, each query corresponds to a single story, and we include all chapters of that story in the ground-truth set, ensuring that the ground-truth annotations are accurate in both precision and recall. For ODSum-Meet, we acknowledge that additional relevant documents may exist beyond the ground-truth set, perhaps due to overlaps in meeting content that may not have been captured by the original QMSum annotations or our query clustering process.

To account for this, we report G-Eval scores, which assess retrieval quality through downstream summarization performance, alongside standard retrieval metrics in Tables 4 and 5. Notably, G-Eval reflects the contribution of all retrieved documents, not just those in the annotated ground-truth set. As a result, even if a relevant retrieved document is not in the ground-truth, it can still boost the G-Eval score by providing helpful context for a better summary.

On a separate note, I believe the title is misleading: the paper does not cover “generation” at large, but only query-based summarization, and should be retitled as such.

Thank you for this suggestion. While our task is centered around query-based summarization, we use the term "retrieval-augmented generation" in the title to reflect the importance of both retrieval and generation in the benchmarks we propose. The central challenge in our benchmarks lies in generating natural language outputs that involve not only summarization based on a user query, but also filtering and integrating relevant content from multiple “retrieved” documents.

That said, we recognize the reviewer’s concern that the current title may suggest a broader coverage of generation tasks beyond query-based summarization. We will revise the paper to clarify the generative nature of the task, and will revisit the framing of the title and abstract to ensure that the scope is accurately conveyed.

Thank you for reviewing our work! We agree that query decontextualization is a critical step in the benchmark creation process that deserves further analysis, so we conducted an additional experiment to directly evaluate its impact on both retrieval and downstream summarization performance.

Do the models perform better when these decontextualized queries are provided?

Yes, our experiment demonstrates that decontextualized queries lead to consistently better performance across both traditional retrieval metrics (Precision, Recall, NDCG, MAP) and downstream generation quality (G-Eval). This aligns with our motivation for decontextualization: while many original SQuALITY queries rely heavily upon a provided story document for meaning, our benchmark requires queries to stand alone in order to simulate realistic retrieval conditions.

The decontextualization process does not remove any details or modify the intent of the queries, but rather introduces minimal, generic hints that help make the queries retrievable. We include the following example on line 164:

For instance, a generic query from SQuALITY like "Describe the setting of the story." was transformed into a more context-specific query such as "In a story where two men navigate a tense and dangerous situation involving a power source that could revive a struggling planet, describe the setting of the story."

We first analyzed the original SQuALITY queries from which the 260-query test set for ODSum-Story was derived and found that many lacked sufficient standalone information. For example, 52 queries were slight variations of "What is the plot of the story?"—asking about the plot without providing any specific context. An additional 30 queries asked about the setting in similarly unspecified ways, such as "What is the setting of the story?" or "Describe the setting of the story." In total, nearly one-third of these original SQuALITY queries lacked enough context to support meaningful retrieval without access to the full story.

We evaluated the impact of query decontextualization using both a sparse retriever (BM25) and a strong dense retriever (gemini-embedding), comparing performance between the decontextualized queries in the ODSum-Story test set and their original counterparts from SQuALITY. Both sets of queries map to the same set of gold-standard summaries, isolating the effect of decontextualization on retrieval and downstream summarization performance.

In the tables below, the "Decontextualized Queries" results are from Table 4 in the paper, and the "Original SQuALITY Queries" results were obtained during this experiment. As mentioned in Table 4, G-Eval #1 and G-Eval #2 refer to the G-Eval scores for the summaries produced by Qwen2.5-7B and Gemini 2.0 Flash, respectively, for each retrieval setting.

The results show a clear and consistent benefit from query decontextualization across all retrieval settings. With BM25, decontextualized queries outperform the original SQuALITY queries on all metrics, including a 5+ point gain in G-Eval scores, demonstrating that even sparse retrievers benefit from more self-contained queries.

The effect is even more pronounced with gemini-embedding, where retrieval metrics improve dramatically (e.g., NDCG increases from 43.86 to 72.33), and G-Eval scores improve by 11+ points. These results confirm that decontextualization enables more effective retrieval by making previously underspecified queries solvable and more suited to the retrieval needs of real-world RAG settings.

Thank you for reviewing our work! We would like to address your concerns and questions as follows:

Q1: The large headroom in the improvement of response quality with Oracle documents is concerning

Thanks for your suggestions! We conducted a small-scale error analysis study with the top-performing models in Tables 6 and 7 for the oracle setting. These models were DeepSeek-V3 for ODSum-Story (G-Eval score of 69.85) and GPT-4o for ODSum-Meet (G-Eval score of 53.67).

For this study, forty summaries—twenty per subset—were randomly sampled from the pools of summaries generated by DeepSeek-V3 given the ODSum-Story oracle documents and GPT-4o given the ODSum-Meet oracle documents. The evaluations were divided up equally among two annotators, who selected all the error type labels that applied to each generated summary given the query, the gold-standard summary, and the oracle documents as reference. The table below depicts each error type label along with the prevalence of that error for the two subsets.

The source of the remaining errors is concerning because it could point to (1) a low upper-bound score even with better retrievers

Our error analysis study indicates that the generated summaries in the subsets often lacked many minor details (40% for ODSum-Story and 60% for ODSum-Meet), with ODSum-Meet also lacking many major details (35% of the time) and additional error types appearing less frequently. These initial findings are consistent with works [1] [2] [3] that demonstrate that even frontier LLMs may still find multi-document tasks difficult due to limitations in multi-document abilities. Furthermore, our human evaluation found ODSum-Meet to be more difficult than ODSum-Story, which aligns with the delta in G-Eval scores across the subsets (53.67 vs. 69.85). This observation is also supported by the original QMSum benchmark paper and recent works [4] [5] performing evaluations upon QMSum that attest to this benchmark’s difficulty.

(2) it could point to an issue with the G-Eval metric and its use in this benchmark,

In our study, we observed trends where the examples for which the annotators did not observe any errors had higher G-Eval scores. Since our error analysis study here is limited by its small scale, we will scale up the study for the revised manuscript, with the addition of annotators evaluating the reliability and consistency of the G-Eval metric and computing its correlation with human evaluations.

(3) errors in the summary aggregation process for ODSum-Meet

The summary aggregation process during the ODSum-Meet benchmark construction described in Appendix B involved the usage of an LLM as solely a tool to combine the multiple smaller gold-standard summaries for the queries being merged together. We do not anticipate errors from this process given the restricted scope of the LLM’s task, and further confirm the validity of the aggregated summaries in our dataset quality human evaluation study in Table 3.

Some analysis of the remaining error will be useful to rule out potential issues with the dataset creation and metrics

We refer to the existing results in our paper to help analyze the remaining error beyond the oracle setting. As shown in Tables 6 and 7, the summarization performance of models, even for advanced generators, is heavily dependent on the quality of the retrieved documents. Additionally, Table 8 in the appendix illustrates that even having access to the most relevant oracle document underperforms having access to all oracle documents by a noticeable margin. This means that the inability of retrievers to fetch more than one relevant document—especially in ODSum-Meet, which had ~30% recall during retrieval (Table 4) out of an average of 3.04 oracle documents per query (Appendix E.1), resulting in ~1 retrieved document on average—would also negatively impact downstream generation performance.

[1] Hosseini et al. Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly. COLING 2025

[2] Kurisinkel and Chen. LLM Based Multi-Document Summarization Exploiting Main-Event Biased Monotone Submodular Content Extraction. arXiv 2023

[3] Tang et al. L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding? arXiv 2024

[4] Mullick et al. Long Dialog Summarization: An Analysis. arXiv 2024

[5] Chen et al. LongLeader: A Comprehensive Leaderboard for Large Language Models in Long-context Scenarios. NAACL 2025

Thank you for reviewing our work! We would like to address your concerns and questions as follows:

While there is a fair attempt made to provide some data on the data quality through data annotation, 40 random examples may not be representative of the quality of the entirety of the dataset which is order of magnitudes larger. This also makes the validation of the quality of future datasets generated using this framework not as scalable as they are always limited by human annotation

The quality of the multi-document RAG benchmarks created using our methods depends heavily on the quality of the underlying datasets—SQuALITY and QMSum, in our case. The summaries in SQuALITY were written by a team of expert writers, with each query paired with four human-written reference summaries validated for correctness by four reviewers. For each query in QMSum, annotators rigorously marked all relevant text spans, which were then reviewed for accuracy and used as the basis for query-focused summaries generated via LLMs.

Therefore, we expect the results in Table 3 from our dataset quality human evaluation study, which sampled roughly 6% of queries in ODSum-Story and 9% in ODSum-Meet, to scale to larger sample sizes due to the high quality of the underlying datasets. Furthermore, we will verify this claim by reviewing additional samples and updating Table 3 with the results from a larger-scale evaluation in the revised manuscript.

It may have also been insightful to see an error analysis of what the models are most challenged in attempting successfully as this may shed light on what the model's current capabilities are able to achieve compared to what is something newer models need to work towards

We conducted a small-scale error analysis study with the top-performing models in Tables 6 and 7 for the oracle setting. These models were DeepSeek-V3 for ODSum-Story (G-Eval score of 69.85) and GPT-4o for ODSum-Meet (G-Eval score of 53.67). We plan to conduct a larger-scale error analysis in the revised manuscript.

For this study, forty summaries—twenty per subset—were randomly sampled from the pools of summaries generated by DeepSeek-V3 given the ODSum-Story oracle documents and GPT-4o given the ODSum-Meet oracle documents. The evaluations were divided up equally among two annotators, who selected all the error type labels that applied to each generated summary given the query, the gold-standard summary, and the oracle documents as reference. The table below depicts each error type label along with the prevalence of that error for the two subsets.

Our error analysis study indicates that the generated summaries in the subsets often lacked many minor details (40% for ODSum-Story and 60% for ODSum-Meet), with ODSum-Meet also lacking many major details (35% of the time), and additional error types appearing less frequently. These initial findings are consistent with works [1] [2] [3] that demonstrate that even frontier LLMs may still find multi-document tasks difficult due to limitations in multi-document abilities. Furthermore, our human evaluation found ODSum-Meet to be more difficult than ODSum-Story, which aligns with the delta in G-Eval scores across the subsets (53.67 vs. 69.85). This observation is also supported by the original QMSum benchmark paper and recent works [4] [5] performing evaluations upon QMSum that attest to this benchmark’s difficulty.

We refer to the existing results in our paper to help analyze the remaining error beyond the oracle setting. As shown in Tables 6 and 7, the summarization performance of models, even for advanced generators, is heavily dependent on the quality of the retrieved documents. Additionally, Table 8 in the appendix illustrates that even having access to the most relevant oracle document underperforms having access to all oracle documents by a noticeable margin, suggesting the necessity of multi-document retrieval to be another potential challenge for models.

[1] Hosseini et al. Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly. COLING 2025

[2] Kurisinkel and Chen. LLM Based Multi-Document Summarization Exploiting Main-Event Biased Monotone Submodular Content Extraction. arXiv 2023

[3] Tang et al. L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding? arXiv 2024

[4] Mullick et al. Long Dialog Summarization: An Analysis. arXiv 2024

[5] Chen et al. LongLeader: A Comprehensive Leaderboard for Large Language Models in Long-context Scenarios. NAACL 2025