Understanding Bias Reinforcement in LLM Agents Debate

Thank you for your valuable and insightful feedback. Your comments have greatly helped us enhance and develop our research. We have thoughtfully considered all of your concerns and will now address each of your comments individually.

1. Additional Evaluation on Open-domain NLP Tasks

We recognize that our current benchmark is centered on structured games. To assess broader applicability, we evaluated DReaMAD on AIME 2024 [1], AMC 2023 [2] problems involving algebra and number theory and CommonSense QA [3]. These problems require symbolic reasoning and multi-step inference. We also conducted additional experiments on reasoning models, such as o3‑mini, to investigate whether their performance improves in the same way as that of general language models. Our results show consistent improvement over baselines:

gpt-o3-mini:

gpt-4o:

gpt-4o:

gpt-4o-mini:

2. Further Comparison with Additional Baselines

To strengthen comparisons, we expanded our baseline set from ReAct [8], Self-Refinement [4], MAD [6] to include recent prompting-based methods such as Self-Consistency [5], MAD2 [7]. The tables below show that DReaMAD consistently outperforms all baselines, including these recent methods:

gemini-1.5-flash:

gpt-4o-mini:

3. Code and Dataset Availability

We appreciate the request for open-sourcing. Since the code factorization isn’t complete yet, we’re releasing the NIM-Normal game first for all baselines link. We confirm that upon acceptance, we will release with refactorized version:

• MetaNIM Arena (dataset + simulator)
• Full implementation of DReaMAD
• Example notebooks with benchmark setups This will ensure full reproducibility and encourage further research.

[1] MAA., American invitational mathematics examination - aime. In American Invitational Mathematics Examination - AIME 2024, February 2024.
[2] MAA., American Mathematics Competitions - AMC., https://huggingface.co/datasets/AI-MO/aimo-validation-amc
[3] Talmor et al., CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. NAACL. 2019
[4] Maddan, A, et al. Self-refine: Iterative refinement with self-feedback. NeurIPS 2023.
[5] Wang, X., et al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
[6] Du, Y., et al. Improving Factuality and Reasoning in Language Models through Multiagent Debate. ICML 2024.
[7] Liang, T., et al. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. EMNLP 2024.
[8] Yao, S., et al. React: Synergizing reasoning and acting in language models. ICLR 2023.

We thank you for your constructive and insightful comments, which have significantly improved our research. We address your concerns below.

1. Applicability of Diverse Amplication on Self-Reflection and Self-Consistency

We understand this concern as asking whether other self-correction methods—such as Self-Refinement [1] and Self-Consistency [2]—can benefit from structured guidance that enhances reasoning diversity. In DReaMAD, this is operationalized through the Strategic Prior Knowledge Elicitation (SPKE) module, which prompts the model to reinterpret the problem and formulate general strategies before engaging in debate.

To isolate SPKE’s impact, we evaluate DReaMAD$^{(-)}$, which includes SPKE but excludes debate (see Table 3 in main manuscript). We further apply SPKE to Self-Refinement and Self-Consistency and compare them to their vanilla versions. The results show that SPKE alone consistently improves performance across settings.

gpt-4o-mini:

gemini-1.5-flash:

2. Further comparison on additional prompting methods

We expanded our ablation to include ReAct [5], Self-Refinement [1], MAD [3], and two recent prompting strategies:

• Self-Consistency [2]: generates multiple reasoning paths and majority votes for the answer.
• MAD2 [4]: uses a MAD framework with two agents arguing adversarially.

The tables below show that DReaMAD consistently outperforms all baselines, including these recent methods.

gpt-4o-mini:

gemini-1.5-flash:

3. Evaluation on Additional Benchmark

We address this in two ways. First, we expand MetaNIM Arena from 4 to 5 environments by adding Corner Queen [6], which introduces a non-trivial structure of winning positions. Second, we evaluate on the AIME 2024, AMC 2023 and CommonSense QA benchmarks to test applicability beyond games.

We include Corner Queen benchmark in the rightmost column of the above tables, where DReaMAD consistently outperforms prior prompting methods.

On AIME, DReaMAD improves accuracy from 76.7% (MAD) to 90.0%. On CommonSenseQA, it improves from 83.6% (Self-Consistency) to 84.4%. Detailed results for the reasoning benchmark can be found in the link (also available in the response under Reviewer LsBK).

[1] Maddan, A, et al. Self-refine: Iterative refinement with self-feedback. NeurIPS 2023.
[2] Wang, X., et al. Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023.
[3] Du, Y., et al. Improving Factuality and Reasoning in Language Models through Multiagent Debate. ICML 2024.
[4] Liang, T., et al. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. EMNLP 2024.
[5] Yao, S., et al. React: Synergizing reasoning and acting in language models. ICLR 2023.
[6] Corner queen game is mathematically equivalent to Wythoff's game.

We thank for your constructive and insightful feedback. Your comments have significantly helped us enhance and advance our research. We have carefully considered each concern you raised. Below we address the main concerns regarding clarity and efficiency.

1. Figure and Table Clarity

We agree that visual clarity is important. We will update figures and tables with increased font sizes and clarity. These enhancements will be reflected in the upcoming version.

2. Cost-effectiveness and Scalability Analysis

While DReaMAD requires a single model, its inference involves additional prompt steps (e.g., prior knowledge elicitation) and a debate process. However, we believe this test-time scaling method is much more efficient than train-time scaling. As we utilized language models through an API, it was challenging to perform a precise quantitative comparison (e.g., GPU usage time) between our test-time scaling approach and traditional model training. Therefore, we compared the costs using dollar amounts, specifically contrasting the API cost per single game using our method versus the costs incurred when OpenAI fine-tuning models on constructed datasets for NIM-N games.

The NIM-N dataset is constructed by mixing data from three different variants of the NIM game. In all variants, the game starts with 31 stones remaining; however, the rules differ in terms of the maximum number of stones that can be removed per turn—3, 4, or 5, respectively. For each variant, eight distinct game states were sampled and the corresponding optimal action was used as the label, yielding a total of 24 training examples.

Additionally, testing was conducted on the aforementioned three scenarios (each 50 games) by using the gpt-4o model as the opponent. The win rate was measured for each scenario, and the average win rate across these variants was reported.

gpt-4o-mini:

The results of this comparison are illustrated in the table above. When fine-tuning a model using API-based fine-tuning (gpt-4o-mini), the performance gradually improved with additional training epochs, achieving win-rates of 0.253, 0.300, 0.420, and 0.460 at 1, 2, 3, and 4 epochs respectively, with corresponding API costs of $0.013*, *$0.023, $0.033*, and *$0.043 (Here, the cost of constructing dataset is not included).

In contrast, our proposed method, DreaMAD, achieved significantly higher performance (0.966) with substantially lower API costs ($0.0098). These results strongly suggest that our approach not only outperforms traditional fine-tuning methods but also is far more cost-efficient.

All the price is calculated by the pricing policy: https://openai.com/api/pricing/