Retrieval-Augmented Generation with Conflicting Evidence

Thank you for your review and we are glad that you found our dataset RAMDocs to be “a challenging dataset right on the topic” and “close to the real data distribution”. Please refer to our response to your comments below:

Scalability with large numbers of retrieved documents

In the general response, to provide a concrete comparison of the computational cost of our method, we calculated the average number of input and output tokens for three representative methods, prompt-based, single agent with self-reflection, and MADAM-RAG. As shown in the table, MADAM-RAG requires 3,186 input tokens and 1,547 output tokens on average—higher than prompt-based RAG (822/126) but comparable to the Single Agent with Self-reflection baseline, which consumes over 5,000 input and 800 output tokens per instance. This demonstrates that while MADAM-RAG introduces some overhead due to its multi-agent setup, it remains computationally tractable and roughly as efficient as strong single-agent iterative alternatives, while outperforming both zero-shot and single-agent approaches.

To improve the scalability of the method w.r.t. the number of retrieved documents, future work could additionally include clustering, document pre-filtering, and shared-context compression to reduce the number of agent instances. However, given the focus of our work on developing the RAMDocs benchmark as well as the MADAM-RAG approach, we feel that such practical/engineering contributions are out of the scope of our paper.

Performance improvements on RAMDocs

We would like to clarify that the absolute gains on RAMDocs are on a very challenging testbed combining ambiguity, misinformation, and retrieval noise. As shown in Table 1 and Figure 3 from the original paper, all methods struggle in this setting, and our method consistently yields the highest performance – we argue that this indicates the difficulty of our proposed RAMDocs setting more than anything. Notably, MADAM-RAG outperforms Speculative RAG by 3.8% on RAMDocs and by larger margins on other datasets (e.g., +13.9% on AmbigDocs), as detailed in the general response. Given the scale and difficulty of these datasets, even modest absolute improvements reflect meaningful gains in robustness and reliability.

Question 1: Single LLM with iterative self-reflection

Thanks for your suggestion, which we address in the general response by adding self-reflection as a baseline. As shown in the table, MADAM-RAG outperforms the single agent with self-reflection baseline by 17.40% (absolute) on FaithEval, 3.70% on AmbigDocs, and 4.0% on RAMDocs.
Iterative single-agent methods operate over concatenated or chunked documents and rely on a single LLM to internally manage conflicts. This approach is prone to long-context issues, frequency bias, and missing less frequent but valid answers, especially under ambiguity or evidence imbalance.

Question 2: Robustness trend with increasing misinformation

As shown in Fig. 3 (b) in the original paper, while the performance of all systems declines as misinformation increases, MADAM-RAG yields the highest performance. This degradation is expected since a greater amount of conflicting evidence increases uncertainty around the query, making it more difficult for the model to identify reliable information. These observations are consistent with prior findings $1$ , which show that even strong LLMs can be misled by persuasive misinformation.

$1$ Rongwu Xu, et al. “The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversation.” ACL 2024

Thank you for your review and we are glad that you appreciate that MADAM-RAG is a “novel framework to handle conflicting evidence scenario in RAG” and it has “superior performance”. Please find the detailed response to your individual comments below:

Methodological details of agentic system

As mentioned in Section 4, each agent is assigned a unique document and generates an intermediate response independently in the first round. In subsequent rounds, each agent receives the aggregator’s summary of all responses from the prior round and may revise its answer accordingly. This allows agents to incorporate cross-document signals while maintaining their independence.

The aggregator takes the full set of agent responses and generates a unified summary answer and an explanatory rationale. Its prompt explicitly encourages attention to consistency, evidence traceability, and disambiguation. In ambiguous settings, it may preserve multiple valid answers, while in misinformation settings, it aims to discard unsupported claims. We will incorporate the details in the final paper.

Computational efficiency

In the general response, we provide a concrete comparison of computational efficiency. Here, we calculated the average number of input and output tokens for three representative methods, prompt-based, single agent with self-reflection, and MADAM-RAG. As shown in the table, MADAM-RAG requires 3,186 input tokens and 1,547 output tokens on average—higher than prompt-based RAG (822/126) but comparable to the Single Agent with Self-reflection baseline, which consumes over 5,000 input and 800 output tokens per instance. This demonstrates that while MADAM-RAG introduces some overhead due to its multi-agent setup, it remains computationally tractable and roughly as efficient as strong single-agent iterative alternatives, while outperforming both zero-shot and single-agent approaches.

Novelty of the proposed benchmark

While RAMDocs builds on AmbigDocs, it introduces key extensions to simulate real-world retrieval more faithfully. Specifically, it combines multiple sources of conflict—ambiguity, misinformation, and noise—in a single query instance. It also varies supporting document distributions, creating realistic imbalances that challenge majority-biased systems. Our results (Section 5.2, Figure 3) show that RAMDocs is substantially more difficult than existing benchmarks. We believe this multifactor composition and the diagnostic nature of the dataset represent a meaningful step forward, as highlighted by Reviewer YVUj (“this paper proposes a novel dataset” which addresses “a blocker for advancing the accuracy of RAG”) and by Reviewer Zrjy, who mentions that our paper “presents a realistic benchmark that effectively captures key challenges faced by practical RAG systems”.

Additional baselines

Thank you for this suggestion, which we have addressed in the general response, We compare MADAM-RAG against three more baselines: Single Agent with Self-reflection, Self-RAG, and Speculative RAG (details in the general response). As shown in the table, MADAM-RAG consistently outperforms three additional baselines across all three datasets, which demonstrates the effectiveness of our method.

Long context LLMs

We would like to clarify that our results already include long-context LLMs. Specifically, we have included results with GPT-4o-mini, a long-context-capable model $1$ , as shown in Table 1. The results show that MADAM-RAG still achieves the best performance across all three datasets.

$1$ https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence

Thanks for your review and for highlighting the fact that our paper “presents a realistic benchmark that effectively captures key challenges faced by practical RAG systems”. We have conducted additional experiments to include three more baselines and show that MADAM-RAG outperforms them, as detailed in the general response. We have sought to address your remaining comments below:

Use of the term “multi-agent”

We agree that the term “multi-agent” has been used in varying ways across different papers. In our context, each instance of the LLM, instantiated with isolated input and persistent dialogue memory, is referred to as an agent; this follows the discussion presented in $1$ . Each agent in our method is independently instantiated with a single document and engages in multiple rounds of discussion, updating responses based on prior global summaries. While some work has used the term “multi-agent” to refer to modular pipelines of agents performing different tasks $2$ , our use of multi-agent is consistent with a large body of work on multi-agent debate/discussion, where agents perform the same task $3,4,5$ . We will clarify this point in the final version.

$1$ https://gist.github.com/yoavg/9142e5d974ab916462e8ec080407365b

$2$ https://arxiv.org/pdf/2308.08155

$3$ https://arxiv.org/abs/2309.13007

$4$ https://arxiv.org/abs/2305.14325

$5$ https://arxiv.org/abs/2308.07201

Comparison with Self-RAG.

Based on this suggestion, in the general response, we compare MADAM-RAG with Self-RAG on Llama3.3-70B-Inst as a representative model. As shown in the table, MADAM-RAG consistently outperforms Self-RAG across all three datasets. The gains are particularly notable on FaithEval and RAMDocs, where misinformation needs to be identified and removed; for instance, MADAM-RAG outperforms Self-RAG by 30.70%(absolute) on FaithEval and 7.60% on RAMDocs. We thank you for your suggestion and will include this result using an extra page in the final version.

Related Work Coverage and comparison with Speculative RAG.

We have cited some of these RAG works, such as Self-RAG (Line 122), Astute RAG (Line 123), and Main-RAG (Line 126). While existing RAG systems (e.g., Speculative RAG) also have parallel document-level processing followed by aggregation, they fundamentally differ from MADAM-RAG in that they do not support multi-turn debate among agents. Our framework enables each document-conditioned agent to revise its answer iteratively based on the summarized responses of others, a mechanism that promotes conflict resolution, minority answer preservation, and misinformation suppression in a more dynamic and interactive way. In the general response, we show empirically that MADAM-RAG outperforms Speculative RAG by 1.3% (absolute) on FaithEval, 13.9% on AmbigDocs, and 3.8% on RAMDocs, which demonstrates the effectiveness and robustness of our multi-agent debate method. We appreciate your suggestion and will add these additional results in the final version.

Question 1: agent processing order

Thank you for pointing this out. We do not process aggregated results in a specific sequence. To mitigate the position bias, we shuffle the responses from agents before passing them to the aggregator. We will clarify this point in the final version.

Question 2: token-level computational efficiency

In the general response, we provide a concrete comparison of computational efficiency. Here, we calculated the average number of input and output tokens for three representative methods, prompt-based, single agent with self-reflection, and MADAM-RAG (see the second point in the general table). As shown in the table, MADAM-RAG requires 3,186 input tokens and 1,547 output tokens on average—higher than prompt-based RAG (822/126) but comparable to the Single Agent with Self-reflection baseline, which consumes over 5,000 input and 800 output tokens per instance. This demonstrates that while MADAM-RAG introduces some overhead due to its multi-agent setup, it remains computationally tractable and roughly as efficient as strong single-agent iterative alternatives, while outperforming both zero-shot and single-agent approaches.

Thank you for your review and for appreciating our “reasonable and relatively solid” approach and the utility of our proposed dataset. We have conducted additional experiments to include three more baselines and show that MADAM-RAG outperforms them, as detailed in the general response. Please find responses to the remaining comments/questions below.

Multi-agent vs. single agent with self-reflection

Based on this suggestion, we have additionally implemented a single agent with self-reflection and evaluated it on all three datasets, with the full results detailed in the general response. MADAM-RAG outperforms the single agent with self-reflection baseline by 17.40% (absolute) on FaithEval, 3.70% on AmbigDocs, and 4.0% on RAMDocs. We will incorporate this baseline into the final version.

These results show some of the shortcomings of single-agent systems: while single-agent approaches like self-reflection $1,2$ can identify internal inconsistencies, they still process concatenated input and are therefore vulnerable to context length limits, frequency bias, and order effects $3,4$ . In our setting, as shown in Fig. 3, processing all documents jointly may obscure low-frequency but valid evidence, especially when misinformation or noise dominates the retrieved set. Our multi-agent setup encourages each document to be independently summarized and defended, allowing underrepresented but correct views to be preserved and strengthened through debate.

$1$ Aman Madaan, et al. "Self-refine: Iterative refinement with self-feedback." NeurIPS 2023

$2$ Jie Huang, et al. "Large Language Models Cannot Self-Correct Reasoning Yet." ICLR 2024

$3$ Nelson F. Liu, et al. “Lost in the Middle: How Language Models Use Long Contexts.” TACL 2023

$4$ Tony Z. Zhao, et al. “Calibrate Before Use: Improving Few-Shot Performance of Language Models.” ICML 2021

Relationship between "conflict" and "ambiguity, misinformation, and noise"

We frame the conflict in our paper (Line 47-52) as inter-context conflict which is the disagreement among retrieved contexts (documents). These conflicts arise from different causes:

• Following $5$ , we treat ambiguity as cases where multiple retrieved documents offer different but valid answers due to underspecified queries.
• Following $6$ misinformation is defined as documents containing factually incorrect information.
• We define noise as documents are irrelevant or only weakly related to the query.

We will clarify these distinctions further in the final version.

$5$ Yoonsang Lee, et al. "AmbigDocs: Reasoning across Documents on Different Entities under the Same Name." COLM 2024

$6$ Yifei Ming, et al. "FaithEval: Can Your Language Model Stay Faithful to Context, Even If 'The Moon is Made of Marshmallows'." ICLR 2025

Thank the authors for the response. How is your single-agent-based method implemented? Note that a single agent doesn't mean that it must process all documents jointly, right? It can be a single agent that handles each task separately, just like a person can handle different tasks independently.