Multi-Agent Debate for LLM Judges with Adaptive Stability Detection

We are pleased that you found the paper well-written and that the early stopping framework was highlighted as a significant contribution. We understand that there are concerns regarding the comparison with majority voting and the clarity of some of the theoretical aspects. We aim to clarify these points below.

---

Weakness 1: No baseline comparisons with other multi-agent debate frameworks from the past.

(1) Positioning of our baselines. We treat SoM (simple majority voting) [3] as the baseline for multi-LLM aggregation. Many “debate” style systems either (a) modify majority voting (e.g., RECONCILE’s confidence-weighted voting) [4] or (b) adopt a different interaction protocol (e.g., MAD: two opposing sides + a judge) [2]. These variants lie outside the scope of our framework, which specifically models an iterative, belief-refinement process. Thus, we did not initially position them as direct comparators.

(2) We agree more empirical baselines are needed. We fully agree with the reviewer’s suggestion that more empirical baselines are needed. Due to time and computation constraint, we conduct experiment using Gemini-2.0-Flash on MAD framework[2], a representative sequential querying approach that structures debates adversarially with multiple debaters presenting arguments for and against a position, moderated by a judge to reach a final decision. The results are somewhat puzzling, as MAD does not even exceed the single-model baseline in accuracy. Our suspicion is that MAD's balanced exposure to both sides gives the incorrect side an equal chance to persuade the judge, skewing outcomes in judgment tasks where nuanced refinement is key.

---

Weakness 2: Distinction between theory and practical results.

Our paper integrates a theoretical framework (Sec. 4) with empirical validation (Sec. 6) to demonstrate the efficacy of the multi-agent debate framework with adaptive stability detection. The theoretical contributions, grounded in latent concept theory and Bayesian inference (Sec. 3, Theorems 4.1-4.2), establish that iterative debate amplifies correct responses (Theorem 4.1) and outperforms majority voting (Theorem 4.2) under Assumptions 4.1-4.5. These guarantees predict that collaborative reasoning corrects biases and improves accuracy, assuming a true concept $\theta^*$ drives responses.

Practically, these predictions are validated across six benchmarks (Table 1), showing consistent accuracy gains over majority voting (e.g., +4.08% on LLMBar for Gemini-2.0-Flash, +2.29% on TruthfulQA). The adaptive stopping mechanism (Sec. 5), based on a Beta-Binomial mixture model, ensures computational efficiency by halting debates in 2-7 rounds with minimal accuracy loss (<0.7%, Table 2). For instance, Theorem 4.1’s prediction of response amplification is reflected in the convergence of judgment distributions to bimodal patterns (Fig. 2), where agents align on the correct answer or fail collectively. The KS statistic’s rapid decline (Fig. 3, e.g., <0.05 by Round 3 on BIG-Bench) confirms the theoretical expectation of stable consensus (Sec. 5.3).

---

Question 1: Conditional independence (L120)

We thank Reviewer 9DD2 for their insightful critique of Assumption 3.1, which posits that an agent’s response $z_i^{(t+1)}$ is conditionally independent of the task $x$ and prior responses $Z^{(t)}$, given the latent concept $\theta$ and model parameters $\phi_i$, i.e., $\mathbb{P}(z_i^{(t+1)} \mid \theta, x, Z^{(t)}, \phi_i) = \mathbb{P}(z_i^{(t+1)} \mid \theta, \phi_i)$. This assumption, inspired by Estornell and Liu (2024) [1], asserts that $\theta$, representing the semantic interpretation of the task (e.g., “Max Verstappen won the 2021 F1 Championship”), and $\phi_i$, the model’s parameters, jointly determine the response. In encoder-decoder architectures, $\theta$ and $\phi_i$ correspond to the encoder’s embedding, which captures the task’s core meaning, rendering $x$ and $Z^{(t)}$ redundant for generation.

This assumption is justified because $\theta$ acts as a sufficient statistic, encapsulating the relevant information from $x$ and $Z^{(t)}$. Marginalizing over $x$ is valid when $\theta$ fully represents the task’s semantic intent, as validated empirically by diverse initial judgments converging to bimodal distributions (Fig. 2, Appendix B.3). The use of diverse models (Gemini, Llama, Qwen, Gemma) with distinct architectures further mitigates potential correlations, supporting the assumption’s practical applicability.

However, the assumption may not hold if $x$ contains nuanced contextual details not captured by $\theta$, such as ambiguous phrasing. We acknowledge that our original explanation was insufficient and will revise Sec. 3.3 in the camera-ready version, ensuring clarity on when the assumption may weaken.

---

Question 1: L140

1. "True Concept": The term $θ∗$ refers to the concept that maximizes the probability of the correct answer. While calling it "true" may seem to imply perfection, it simply means that this concept is ideal for generating the correct answer in theory. It’s not about "luck," but about defining the optimal concept for the task.
2. "Simplifying Misinterpretations": This refers to reducing the complexity of handling multiple potential misinterpretations by assuming a single, ideal latent concept. It helps focus the reasoning process on refining beliefs toward the correct answer, rather than dealing with many possible incorrect interpretations.

We will revise the paper to clarify these points further. Let us know if this clears up the confusion!

---

Question 3: Agent Independence (L152)

We would like to clarify that similarity in outputs does not imply probabilistic dependency.

• Stochasticity: LLMs generate outputs stochastically, meaning even models trained on similar data can produce diverse responses, ensuring independence in predictions.
• Ensemble Methods: In ensemble learning, models trained on similar data can still operate independently, as their predictions are generated separately without influencing each other.

---

Question 4: EM algorithm leading to local optimum

While it’s true that the E-M algorithm can converge to a local optimum, we want to emphasize that, even in such cases, the solution closely tracks the real distribution of judgments. In practical applications, local optima in this context often still lead to highly accurate results.

Our multi-agent debate framework is designed to refine beliefs iteratively, and the agents engage in collaborative reasoning to progressively converge to a solution that approximates the correct distribution of responses. The adaptive stability detection mechanism ensures that the process stops when the distribution stabilizes, preventing unnecessary iterations and ensuring computational efficiency.

Additionally, the empirical results clearly show that the framework outperforms simpler aggregation methods like majority voting, indicating that even if the algorithm converges to a local optimum, it provides substantial improvements over static methods. This suggests that the practical benefits of the framework are not undermined by the theoretical risk of local maxima.

We hope this explanation clarifies how the approach still yields valuable practical outcomes despite the theoretical considerations. Please let us know if you need further details or clarification.

---

Reference

[1] Andrew Estornell and Yang Liu. Multi-LLM Debate: Framework, Principles, and Interventions. In A. Globerson et al., editors, Advances in Neural Information Processing Systems, volume 37, pages 28938–28964. Curran Associates, Inc., 2024.

[2]Encouraging divergent thinking in large language models through multi-agent debate, EMNLP 2024

[3] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.

[4] Justin Chen, Swarnadeep Saha, and Mohit Bansal. 2024. ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7066–7085, Bangkok, Thailand. Association for Computational Linguistics.

We thank Reviewer 4Cos for their constructive review and positive feedback on the paper's clarity, EM algorithm novelty, and comprehensive experiments. We address the weaknesses and questions below, incorporating clarifications to resolve concerns. These will improve the paper in the camera-ready version, and we hope this elevates our score.

---

Weakness 1 - L122, Debate History (Conditional Independence Assumption)

The conditional independence assumption (Assumption 3.1) is a reasonable simplification grounded in latent concept theory, enabling tractable Bayesian modeling while aligning with practical LLM behavior.

In our framework, agents receive only the previous round's debate history $Z^{(t-1)}$, not the entire history, to inform their current response $z_i^{(t)}$, as outlined in Algorithm 1. This limited access ensures that, given the latent concept $θ$ and parameters $φ_i$, responses remain independent of distant history, focusing on immediate refinements. The assumption holds because $θ$ encapsulates semantic interpretations, rendering prior inputs redundant once inferred (Lemma 3.1). Empirically, this is validated by convergence to bimodal distributions (Fig. 2), where agents refine beliefs without full history dependence.

---

Weakness 2: L179, Initial Response

If no response is generated via the correct concept initially, it becomes highly unlikely to converge to the correct answer from a probabilistic perspective. This is because, without a correct starting point, the iterative refinement process is unlikely to guide the agents toward the correct concept, especially if the agents are not initially aligned with the true concept.

---

Weakness 3: L159，Consistency

In our framework, the term "consistency" refers to how well a response aligns with the correct concept, as defined by the task. Specifically, the likelihood is the probability that a given response reflects the true concept $\theta^*$ given the task $x$ and the agent's belief. In simpler terms, we are measuring how well a response matches the correct answer based on the agent's internal reasoning. We will revise the explanation in our final version.

---

Question 1: Difference between our Framework and Related Works

Compared to MADISSE [1], the debate structure is similar, but our main contributions are the theoretical analysis, Beta-Binomial mixture model fitting, and adaptive stopping mechanism. We do not claim novelty in the core debate framework, which has been explored at least since Improving Factuality and Reasoning in Language Models through Multiagent Debate[2].

Related works can be broadly classified by querying type: sequential (iterative, dependent interactions where agents refine responses based on prior outputs) or parallel (simultaneous, independent generations followed by aggregation). The MAD[6] framework represents sequential querying, using a debate setup similar to real-life debates with multiple debaters and a judge, similar to Debate Helps Supervise Unreliable Experts[7]. Other related works are extensions or variants of the parallel debate framework formulated in Improving Factuality and Reasoning in Language Models through Multiagent Debate[2], including CIPHER[3] (novel communication protocol instead of using human words), CAMEL[4] (similarly LLM-Harmony[5], using tuned prompted to serve as a role-playing framework), etc.

---

Question 2: Have the LLM achieve the maximum token limitation in the experiments?

No, in our experiments, we carefully managed the input sizes and configured the model’s maximum token limit to 16,000 tokens. Neither the queries nor the responses exceeded this limit. We are confident of this because if the token limit is exceeded, the vllm framework used would throw an error.

---

Question 3: Could we understand the latent concepts as the knowledge base?

We apologize if we have misunderstood the term "knowledge base" to which you are referring. To address your question regarding whether latent concepts in our framework can be understood as a knowledge base, we provide the following clarification.

If you are referring to a knowledge base in the context of knowledge graphs or structured databases (e.g., a repository of explicit facts, entities, and relations), then no, latent concepts are not equivalent. In our framework (Sec. 3.2), latent concepts $\theta \in \Theta$ represent underlying abstract interpretations or mental models that agents infer to understand and reason about a task. These are dynamic, task-specific abstractions derived from the input $x$ and debate history $Z^t$, evolving through Bayesian updates (Lemma 3.1). They serve as internalized frameworks for response generation, rather than static collections of facts.

However, if you mean a knowledge base in the sense of prior beliefs or distributions in Bayesian theorem (e.g., the prior $\mathbb{P}(\theta \mid \phi_i)$in Assumption 4.3), then yes, there is a conceptual alignment. Latent concepts incorporate agents' prior knowledge and are updated probabilistically based on observed evidence, facilitating belief refinement during debate.

We will revise Sec. 3.2 in the camera-ready version to more explicitly distinguish these interpretations for clarity.

---

Question 4: How can we know a true concept?

In the context of our framework, the true concept refers to the underlying, correct interpretation or solution to a given task. However, it's important to note that we cannot directly observe the true concept during the debate process; instead, we approximate it through probabilistic reasoning.

---

Question 5: How to understand responsibilities in L213? Are they the confidence score?

To clarify:

Responsibilities: In the Expectation-Maximization (EM) algorithm used in our framework, responsibilities are probabilistic weights that determine how much each agent's response contributes to the final estimate of the latent concept. These responsibilities are computed during the E-step of the EM process, where the algorithm calculates the likelihood of each agent’s response under different possible concepts (i.e., how likely a response is to belong to the true concept or another concept).

Confidence Scores: While responsibilities can be thought of as a type of "weight," they are not directly the confidence score of an agent's response. Confidence scores typically refer to how sure an agent is about a particular response or prediction. In contrast, responsibilities reflect the probability that a given response comes from the correct latent concept, which is a part of the belief update process in our model.

---

Question 6: The performance does not significantly improve in Table 1. How could we know whether the proposed model performs better than SOM?

We recognise that SoM does sometimes score higher accuracy than the proposed model. And we argue that the framework excels on complex, nuanced tasks with high initial judgment variance, where iterative refinement corrects biases via Bayesian updates (Theorem 4.1), yielding meaningful improvements. On simpler tasks with high consensus, SoM suffices as refinement adds limited value. We do not claim universal superiority but targeted applicability: debate is preferable when accuracy justifies costs in high-stakes scenarios. We will revise Sec. 6 to include this in the camera-ready version.

---

Suggestion

Thank you for the suggestion. We will revise the figures accordingly in the final version.

---

Reference

[1]Faithful, Unfaithful or Ambiguous? Multi-Agent Debate with Initial Stance for Summary Evaluation (Koupaee et al., NAACL 2025)

[2]Improving Factuality and Reasoning in Language Models through Multiagent Debate, Du et al., ICML'24

[3]Chau Pham, Boyi Liu, Yingxiang Yang, Zhengyu Chen, Tianyi Liu, Jianbo Yuan, Bryan A. Plummer, Zhaoran Wang, and Hongxia Yang. Let models speak ciphers: Multiagent debate through embeddings. In The Twelfth International Conference on Learning Representations, 2024.

[4]CAMEL: Communicative Agents for “Mind” Exploration of Large Language Model Society, NeurIPS 2023

[5]Sumedh Rasal. Llm harmony: Multi-agent communication for problem solving, arxiv 2024.

[6]Encouraging divergent thinking in large language models through multi-agent debate, EMNLP 2024

[7]Debate Helps Supervise Unreliable Experts, arxiv 2023

We sincerely thank Reviewer izXV for their thorough and constructive feedback. We greatly appreciate the recognition of our paper’s strengths, including the innovative debate framework, formal mathematical model, and broad applicability across tasks and models.

---

Weakness 1: Sensitivity to Stability Threshold?

Using data from JudgeBench with Gemini-2.0-Flash (n=7 agents, max 10 rounds), we analyzed thresholds from 0.01 to 0.20. The table below summarizes key metrics:

The mechanism shows low sensitivity around the default 0.05: thresholds of 0.02–0.08 halt at 6–8 rounds with similar KS statistics (mean ~0.096–0.130), maintaining accuracy within 0.5% of a full 10-round debate. Stricter thresholds (0.01) require all 10 rounds, while lenient ones (≥0.10) halt earlier (4–5 rounds), increasing convergence rate but risking premature termination. Accuracy remains stable across thresholds, with no errors reported.

For new models or tasks, start with 0.05 (robust across benchmarks, Fig. 3). Pilot a small subset to monitor KS convergence: use 0.03 for high-variance tasks or 0.07 for simpler ones. We will include this analysis in our final version.

---

Weakness 2: Dependence on Assumptions

We thank Reviewer 1eiT for requesting further clarification on the practical validity of Assumptions 4.1–4.5 (Sec. 4.1), which underpin our theoretical guarantees (Theorems 4.1–4.2). Specifically, Assumption 4.1 (True Concept Predictiveness) posits a true latent concept $\theta^*$ that maximizes the likelihood of correct responses, while Assumptions 4.2–4.5 enable agents to update beliefs based on observed responses via Bayesian inference (Sec. 3.3). Below, we discuss why these assumptions hold in practice, supported by empirical evidence.

The existence of a true latent concept $\theta^*$ is grounded in latent concept theory (Xie et al., 2022; Jiang et al., 2023), where LLMs encode semantic interpretations of tasks as abstractions. For example, in a fact-checking task like Who won the 2021 Formula 1 Championship? (Sec. 3.2), $\theta^*$ represents the correct interpretation (Max Verstappen won), which aligns with the ground truth. In practice, this holds because LLMs, trained on diverse datasets, can infer such concepts from task inputs, as evidenced by varied initial responses converging to bimodal distributions (Fig. 2, Appendix B.3), reflecting alignment with $\theta^*$ or collective failure. Our use of diverse models (Gemini, Llama, Qwen, Gemma) ensures at least one agent approximates $\theta^*$, satisfying Assumption 4.5 (Initial Seed), as seen in high initial accuracy for some agents (e.g., 81.74% for Gemini on BIG-Bench, Table 1).

Agents’ ability to update beliefs based on observed responses (Assumptions 4.2–4.4) is practical because LLMs can refine outputs when exposed to peer responses, a capability leveraged in multi-agent systems (Du et al., 2024). Our framework models this as Bayesian updates (Lemma 3.1), where agents adjust their posterior over $\theta$ using $x$ and $Z^t$. Empirically, this is supported by judgment convergence within 2–7 rounds (Fig. 3), with accuracy gains over majority voting (e.g., +4.08% on LLMBar, Table 1). The conditional independence assumption (4.4) holds in our parallel response generation (Algorithm 1), and diverse architectures reduce correlated errors (Appendix Table A.3, +1–2% via pruning).

In practice, these assumptions may weaken if tasks lack a clear $\theta^*$ (e.g., subjective JudgeBench tasks) or if agents share strong biases, violating independence. Our diversity pruning and model variation mitigate this, though highly ambiguous tasks may require further handling. We will revise Sec. 4.1 in the camera-ready version to elaborate on practical validity.

---

Weakness 3: Stop mechanism is not really effective?

To address the reviewer’s suggestion for comparison with a fixed-stop approach (e.g., 3 rounds), we evaluated Gemini-2.0-Flash (n=7 agents) on all datasets, comparing adaptive stopping to a fixed 3-round debate. Results, shown below, demonstrate that adaptive stopping achieves better accuracy. the improvement might not be significant, but we can also tune the parameters of the adaptive stopping mechanism to make it stop earlier than the current setting.

---

Weakness 4: Limited baselines

(1) Positioning of our baselines. We treat SoM (simple majority voting) [1] as the baseline for multi-LLM aggregation. Many “debate” style systems either (a) modify majority voting (e.g., RECONCILE’s confidence-weighted voting) [2] or (b) adopt a different interaction protocol (e.g., MAD: two opposing sides + a judge) [3]. These variants lie outside the scope of our framework, which specifically models an iterative, belief-refinement process. Thus, we did not initially position them as direct comparators.

(2) We agree more empirical baselines are needed. We fully agree with the reviewer’s suggestion that more empirical baselines are needed. Due to time and computation constraint, we conduct experiment using Gemini-2.0-Flash on MAD framework[3], a representative sequential querying approach that structures debates adversarially with multiple debaters presenting arguments for and against a position, moderated by a judge to reach a final decision. The results are somewhat puzzling, as MAD does not even exceed the single-model baseline in accuracy. Our suspicion is that MAD's balanced exposure to both sides gives the incorrect side an equal chance to persuade the judge, skewing outcomes in judgment tasks where nuanced refinement is key.

---

Question 1: Stop mechanism

See Weekness 3.

---

Question 2: Majority vote implementation and performance on simpler tasks.

In our framework, majority voting (SoM) aggregates responses from $n=7$ agents, each generating one response per question at Round 0 (Sec. 6.1, Table 1). This yields 7 generations, matching the debate’s initial round token budget, ensuring a fair comparison (Appendix Table A.4). Unlike debate’s iterative refinement (up to 11 rounds, ~77 generations), SoM uses a single round, taking the mode of the 7 responses. On BIG-Bench and JudgeAnything, SoM occasionally outperforms debate. These tasks are simpler, with high initial consensus, reducing the need for iterative refinement. Debate excels in complex tasks like LLMBar (+4.08%) where initial variance, aligning with Theorem 4.2’s claim of superiority over static ensembles in such scenarios. This does not contradict our main claim, as our framework targets tasks requiring robust judgment under uncertainty (Sec. 1). We will clarify task-specific applicability in Sec. 6 revisions.

---

Typo

Thank you for pointing that out! We will fix that in the revised version.

---

Reference [1] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.

[2] Justin Chen, Swarnadeep Saha, and Mohit Bansal. 2024. ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7066–7085, Bangkok, Thailand. Association for Computational Linguistics.

[3] Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17889–17904, Miami, Florida, USA. Association for Computational Linguistics.

We sincerely thank Reviewer 1eiT for the thoughtful evaluation and constructive feedback. We greatly appreciate the recognition of our paper's strengths, particularly the solid mathematical framework, the cross-domain experiment validation, and the relevance of addressing the unreliability of a single judge. Below, we address the key concerns and suggestions:

---

Weakness 1 - Missing baselines against prior multi-agent debate frameworks

(1) Positioning of our baselines. We treat SoM (simple majority voting) [1] as the baseline for multi-LLM aggregation. Many “debate” style systems either (a) modify majority voting (e.g., RECONCILE’s confidence-weighted voting) [2] or (b) adopt a different interaction protocol (e.g., MAD: two opposing sides + a judge) [3]. These variants lie outside the scope of our framework, which specifically models an iterative, belief-refinement process. Thus, we did not initially position them as direct comparators.

(2) We agree more empirical baselines are needed. We fully agree with the reviewer’s suggestion that more empirical baselines are needed. Due to time and computation constraint, we conduct experiment using Gemini-2.0-Flash on MAD framework[2], a representative sequential querying approach that structures debates adversarially with multiple debaters presenting arguments for and against a position, moderated by a judge to reach a final decision. The results are somewhat puzzling, as MAD does not even exceed the single-model baseline in accuracy. Our suspicion is that MAD's balanced exposure to both sides gives the incorrect side an equal chance to persuade the judge, skewing outcomes in judgment tasks where nuanced refinement is key.

---

Weakness 2 - Modest Accuracy Gains and Majority Voting Performance

We appreciate your observation regarding the modest accuracy gains in Table 1 and cases where majority voting (SoM) matches or slightly exceeds debate performance on BIG-Bench and JudgeAnything. These gains are modest because our framework's iterative refinement adds most value in complex tasks with high initial variance, where collaborative belief updates correct biases (Theorem 4.1), yielding significant improvements. On simpler tasks with high initial consensus, SoM performs comparably or better as refinement introduces minimal benefit, aligning with diminishing returns in low-variance scenarios.

We view this as reasonable and expected, supporting targeted applicability rather than universality: debate excels where accuracy justifies costs, while SoM suffices for straightforward tasks. We will revise Sec. 6 to include statistical significance and task-dependent guidance in the camera-ready version.

---

Weakness 3 - Computational Expense and Scalability

We acknowledge the concern regarding the computational cost of the debate framework. Despite the adaptive stopping mechanism, the iterative nature of the debate process requires multiple model calls, making it more expensive than single-judge approaches. For an ensemble size of 7 (our recommended setting, Appendix B.2), the initial round incurs 7 calls (one per agent), and adaptive stopping typically adds 1–6 more rounds (average ~5, Table 2), resulting in approximately 35 calls total—roughly 35 times the cost of a single-judge query in terms of generations.

Larger ensembles (>11 agents) exacerbate costs due to context dilution and diminishing returns (-0.7% accuracy average, Appendix Table A.1). For real-time applications, the framework suits offline batch processing in high-stakes scenarios (e.g., fact-checking), where accuracy gains (+4.08% on LLMBar) justify overhead. We recommend size 5–7 and will incorporate this clarification, including relative cost multiples, in Sec. 6 of the camera-ready version.

---

Weakness 4: Analytical Limitation.

We recognize the limited in-depth analysis beyond accuracy numbers and figures. Key insights include: bimodal convergence reflecting unanimous alignment or failure (Fig. 2, supporting Theorem 4.1); KS dynamics showing rapid stabilization (Fig. 3); size 7 optimizing accuracy-cost balance (Table B.1); temperature 1.0 maximizing performance (Table B.2); pruning aiding diversity without consistent gains (Table B.3). We will reorganize Appendix B.3 analyses into the main body in the camera-ready version, adding qualitative case studies.

---

Question 1: Conditions for Preferring the Debate Framework Over Simpler Approaches

The debate framework should be preferred over simpler methods like majority voting under the following conditions:

• Complexity of the Task: For tasks requiring high levels of reasoning, such as fact-checking or nuanced decision-making (e.g., LLMBar), the debate framework significantly outperforms simpler methods. It allows for the iterative refinement of outputs, leading to higher accuracy.
• Task Sensitivity: In tasks where minor errors have significant consequences (e.g., content moderation or judgment evaluation), the collaborative reasoning process of debate offers better outcomes, justifying the computational overhead. For example, Gemini-2.0-Flash showed substantial improvements in accuracy in high-stakes tasks.

We believe the computational cost is justified when high accuracy is essential, particularly for complex or high-stakes tasks.

---

Question 2: The Effect of Ensemble Size

Our analysis (Appendix B.2, L814) shows that an ensemble size of 7 provides the best balance between accuracy and computational cost across most tasks. Larger ensembles (Size-9 or greater) show diminishing returns in accuracy, while significantly increasing computational costs, smaller ensembles (Size-5) are sufficient to maintain accuracy with minimal cost. We recommend Size-7 as the optimal choice for most use cases.

We will explicitly include this guidance in the main body of the paper.

---

Question 3: Failure Cases

1. Small Models: When using smaller models, the limited diversity in reasoning can reduce the effectiveness of iterative refinement in the debate process, making the framework less beneficial.

2. Simple Tasks: For tasks that are straightforward or well-defined, where the correct answer is obvious, the additional computational cost of the debate framework does not lead to significant improvements. In these cases, majority voting can be more efficient.

In such scenarios, the added complexity of the debate framework does not justify the computational overhead. We will update the paper to emphasize these cases and provide clearer guidance on when the debate framework may not offer advantages.

---

Reference [1] Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org, 2024.

[2] Justin Chen, Swarnadeep Saha, and Mohit Bansal. 2024. ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7066–7085, Bangkok, Thailand. Association for Computational Linguistics.

[3] Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. 2024. Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17889–17904, Miami, Florida, USA. Association for Computational Linguistics.