AI Debate Aids Assessment of Controversial Claims

We thank reviewer W11V for their careful review and appreciation of our clear goals, contributions to prior Debate protocol research, helpful related work section, and efforts to address manipulation problems in judges. We address all concerns about foundational work citations, generalization claims, dataset choice, and societal impacts below.

---

Q: Limited generalization beyond COVID-19

To address reviewer concerns about generalization beyond COVID-19, we conducted an extensive additional study with human judges on climate change claims.

New Climate Change Study

To demonstrate generalization, we recruited 146 participants (73 per belief group) from an initial pool of 1,000 based on their climate change beliefs (human-caused vs. natural cycles), yielding 421 debate sessions and 424 consultancy sessions after quality filtering. Following the reviewer's suggestion to use other fact-checking datasets, we selected claims from the Climate-Fever dataset [1,2], which contains settled claims backed by scientific evidence, using GPT-4o screening followed by human curation to ensure they satisfy our scalable oversight criteria (details in Section 2.1.1 and Appendix B.1).

In this study, we use an even stronger consultancy baseline by allowing consultants to choose which position to argue rather than random 50/50 assignments. Now consultants argue for the correct answer ~92.54% of the time (giving consultants significant advantage), creating a much more challenging comparison that better reflects real-world scenarios where AI assistants like ChatGPT naturally provide their best assessment rather than randomly arguing for correct or incorrect positions.

Results

Table: Human judge accuracy before and after debate versus consultancy interventions for climate claims

Belief Groups:

• Belief Group 1: Climate change is primarily caused by natural cycles and variations
• Belief Group 2: Climate change is primarily caused by human activities

Debate achieves 72.1% accuracy compared to stronger consultancy's 68.3% accuracy—a 3.8% improvement. This shows debate's advantages generalize beyond COVID-19 to other contentious scientific domains even with a substantially stronger consultancy baseline.

Additionally, we conducted LLM judge studies on climate claims (detailed in Appendix D.2) which show similar patterns—debate provides better performance than consultancy across both COVID-19 and climate domains, with debate particularly benefiting weaker judges like Qwen-2.5-8B.

We will update Figure 1 to include these climate results and revise Section 2.1.2 to clearly describe our stronger consultancy protocol, directly addressing the reviewer's concerns about generalization beyond COVID-19.

---

Q: Reasoning for considering judge backgrounds

Thank you for highlighting this. We should indeed be more explicit about our rationale for considering judge backgrounds and beliefs.

We recruited judges with different prior beliefs for three key reasons:

First, the cognitive science literature documents how confirmation bias and motivated reasoning affect judgment [8,9]. In factual disputes about COVID-19, someone who believes vaccines are safe will evaluate evidence differently than a skeptic—not just in conclusions but in which sources they trust and how they weigh conflicting evidence.

Second, real-world AI oversight involves diverse judges with existing worldviews, not neutral evaluators. Understanding how debate performs across this diversity is crucial for practical deployment.

Third, this directly addresses the scalable oversight challenge. As noted in our introduction, humans exhibit cognitive biases that lead them to accept arguments aligning with their mental models. We empirically test whether debate helps people assess controversial belief-sensitive claims despite these biases.

We will add a clear statement in Section 2.1.3: "We recruited judges with different prior beliefs because real-world deployment must account for human belief diversity [10]. Testing debate's effectiveness across belief systems—rather than assuming idealized rational judges—is essential for building oversight mechanisms that work with actual humans."

---

Q: Tempering claims about generalization to high-stakes domains

Thank you for this important critique. To address this concern, we have conducted additional experiments on climate change claims, expanding our evidence beyond COVID-19. While our results now demonstrate debate's effectiveness across two controversial scientific domains (COVID-19 and climate change), we will acknowledge in the paper that each high-stakes domain has unique characteristics that require careful empirical validation.

Following the reviewer's suggestion, we will revise our conclusion (lines 375-376) from "a practical and scalable mechanism for oversight in high-stakes, belief-sensitive domains" to "a promising approach for oversight in belief-sensitive factual assessment tasks." We will also add explicit caveats about the limitations of generalizing from public health and climate claims to other high-stakes decisions, leaving detailed exploration of additional domains as future work.

---

Q: Foundational work and peer review concerns

We understand the reviewer's concern about the peer review status of foundational work in scalable oversight. While it's true that some early influential papers like Irving et al. [3], Bowman et al. [4], and Michael et al. [5] were initially published as preprints, we respectfully note that subsequent work building on these foundations has been rigorously peer-reviewed at top-tier venues. For example:

• Khan et al. [6] won the Best Paper Award at ICML 2024
• Kenton et al. [7] was published at NeurIPS 2024

The field is growing with an increasing number of peer-reviewed papers validating and extending these early ideas. Our work contributes to this growing body of peer-reviewed research by being the first to examine scalable oversight in belief-sensitive domains where judges hold strong prior biases—a critical gap that previous work has not addressed.

---

Q: Citation issue with withdrawn paper [50]

The reviewer is correct that reference [50] has an updated version. The KDD '25 accepted version became available on June 18, 2025, which was after the NeurIPS submission deadline (May 15, 2025). We will update this citation in our camera-ready version.

---

Q: Societal impacts concern

We appreciate the reviewer's important concern about the societal impacts of our work.

From the outset, we deliberately limited our experiments to scientific claims where there is clear, verifiable ground truth, rather than political or values-laden topics. Likewise, our debate protocol was designed to surface evidence and counter-evidence in service of truthfulness, not to maximize persuasion or engagement. Nonetheless, we fully agree that any system capable of influencing beliefs must be examined through an ethical lens.

To make this scrutiny explicit, we will add a dedicated "Broader Impacts and Ethical Considerations" section discussing:

• Potential misuse if debate systems are deployed with misaligned incentives
• Risks that our findings about persona-based persuasion could enable targeted manipulation

Finally, we will outline concrete safeguards to ensure debate remains an instrument for accurate judgment rather than exploitation. These include:

• Auditing and transparency requirements for source materials
• Routine human-in-the-loop checks
• Clear guidelines on when and where debate-based oversight is appropriate

By articulating these principles and boundaries, we aim to preserve human agency in high-stakes decisions and help guide the responsible deployment of AI debate systems.

---

We hope our comprehensive responses—the new climate study with strengthened baseline, clarification about foundational work and updated citation, explicit rationale for judge backgrounds, tempered claims about high-stakes implications, and expanded discussion of societal impacts with safeguards—fully address the reviewer's concerns. If the reviewer finds these improvements satisfactory, we would be grateful for their consideration in revisiting the score.

References

1. Diggelmann et al., "Climate-fever: A dataset for verification of real-world climate claims", arXiv 2020.
2. Gabriel et al., "Misinfo reaction frames: Reasoning about readers' reactions to news headlines", ACL 2022.
3. Irving et al., "AI safety via debate", arXiv 2018.
4. Bowman et al., "Measuring Progress on Scalable Oversight for Large Language Models", arXiv 2022.
5. Michael et al., "Debate Helps Supervise Unreliable Experts", arXiv 2023.
6. Khan et al., "Debating with More Persuasive LLMs Leads to More Truthful Answers", ICML 2024.
7. Kenton et al., "On scalable oversight with weak LLMs judging strong LLMs", NeurIPS 2024.
8. Kahneman, "Thinking, fast and slow", 2011.
9. Kunda, "The case for motivated reasoning", Psychological Bulletin 1990.
10. Irving and Askell, "AI safety needs social scientists", Distill 2019.

We thank reviewer bvi9 for their thorough review. We appreciate that the reviewer recognizes the novelty of applying Debate to real-world information evaluation, our careful consideration of judge biases, and our innovative use of persona-based LLMs. We address all the reviewer's concerns below.

---

Q: Explanation for COVID-19 focus

We selected COVID-19 claims because they uniquely satisfy our requirements: (1) people hold strong, divergent prior beliefs (mainstream vs. skeptical) that significantly affect judgment, and (2) the selected claims meet established scalable oversight criteria as detailed in Section 2.1.1 and Appendix B. These claims also have direct societal impact - as noted in lines 2-4, misinformation on public health topics can cause real harm, making it critical to study how AI debate can help people evaluate facts despite their biases. We will clarify this in Section 2.1.1.

---

Q: Sample selection, mainstream vs. false confound, and dataset bias

Thank you for this thoughtful critique.

We carefully selected claims through multiple rounds of GPT-4o screening and human curation to ensure claims have ground truth backed by scientific evidence. In our study, "mainstream" views align with scientific consensus (i.e., factually correct positions), while "skeptical" views diverge from it. Our goal is to examine whether debate helps people reach accurate judgments on controversial claims despite their prior beliefs. The claims satisfy scalable oversight criteria—controversial yet with verifiable ground truth from scientific literature—allowing us to evaluate protocol effectiveness.

We appreciate the reviewer's suggestion about using prediction market claims where ground truth emerges later. While such longitudinal studies would be valuable (requiring long waiting periods), we conducted an immediate additional study on climate change claims to demonstrate debate's effectiveness across different domains. We will include prediction markets in our future work section, acknowledging that such studies with initially unresolved claims could provide additional insights into separating mainstream bias from truth-seeking mechanisms.

---

Q: Limited generalization beyond COVID-19 and weak consultancy baseline

To address reviewer concerns about generalization beyond COVID-19 and baseline strength, we conducted an extensive additional study with human judges on climate change claims.

New Climate Change Study with Stronger Baseline

To demonstrate generalization, we recruited 146 participants (73 per belief group) from an initial pool of 1,000 based on their climate change beliefs (human-caused vs. natural cycles), yielding 421 debate sessions and 424 consultancy sessions after quality filtering. We selected claims from the Climate-Fever dataset [1] using GPT-4o screening followed by human curation to ensure they satisfy our scalable oversight criteria (details in Section 2.1.1 and Appendix B.1).

Critically, we strengthened the consultancy baseline by allowing consultants to choose which position to argue rather than random 50/50 assignments. This addresses the reviewer's concern that our original baseline was artificially weak. Now consultants argue for the correct answer ~92.54% of the time (giving consultants significant advantage), creating a much more challenging comparison that better reflects real-world scenarios where AI assistants like ChatGPT naturally provide their best assessment rather than randomly arguing for correct or incorrect positions.

Results

Table: Human judge accuracy before and after debate versus consultancy interventions for climate claims

Belief Groups:

• Belief Group 1: Climate change is primarily caused by natural cycles and variations
• Belief Group 2: Climate change is primarily caused by human activities

Debate achieves 72.1% accuracy compared to stronger consultancy's 68.3% accuracy—a 3.8% improvement. This shows debate's advantages generalize beyond COVID-19 to other contentious scientific domains even with a substantially stronger consultancy baseline.

Additionally, we conducted LLM judge studies on climate claims (detailed in Appendix D.2) which show similar patterns—debate provides better performance than consultancy across both COVID-19 and climate domains, with debate particularly benefiting weaker judges like Qwen-2.5-8B.

We will update Figure 1 to include these climate results and revise Section 2.1.2 to describe our stronger consultancy protocol, addressing both concerns.

---

Q: Missing literature

Thank you for highlighting Costello et al. [2] - we cite it in line 233 (reference 14) and use their persuasion taxonomy. We agree we should better position our work at the intersection of scalable oversight and misinformation. While previous scalable oversight studies focus on reading comprehension tasks [3, 4], and misinformation work primarily examines single-advisor consultancy setups [2, 5], our debate approach represents a novel approach. As per your suggestion, we will expand our discussion in Related Work (Section 4, "Scalable oversight" paragraph) and Introduction (Section 1) to highlight how debate's adversarial structure might address some limitations of consultancy-based approaches for controversial claims.

---

Q: Educational background and susceptibility to persuasion

Thank you for this insightful suggestion. In our main paper (lines 207-228), our mixed-effects models already control for LLM experience, which correlates with educational background. However, to directly address reviewers' comments about education, we conducted additional analysis examining how educational background affects susceptibility to persuasion (i.e., likelihood of changing one's initial belief) across our 896 sessions with 68 participants on COVID-19 claims.

Our findings:

Education significantly moderates debate's effectiveness: Higher education enhances debate's protective effect against belief manipulation. PhD holders showed the strongest resistance—only 15.9% changed their beliefs in debate versus 36.2% in consultancy (a 20.3 percentage point protective effect).

Debate reduces education-based disparities: In consultancy, belief change rates vary widely across education levels (from 31.4% for undergraduates to 58.3% for secondary education). In debate, rates are more consistent across all education levels (15.9% to 29.7%), suggesting debate helps equalize outcomes regardless of educational background.

This analysis reveals that while participants with advanced degrees benefit most from debate, the protocol improves judgment across all education levels. We will include this educational breakdown in our revised manuscript.

---

Q: Influence of prior conviction strength between belief groups

We analyzed initial confidence levels and found both groups hold similarly strong convictions: mainstream believers averaged 70.1% confidence vs skeptical believers at 68.1%—essentially very small difference. This contradicts the hypothesis that non-mainstream believers are less sure of their beliefs. Figure 3 details how initial confidence levels (Low/Moderate/Strong) impact protocol effectiveness, showing debate benefits apply regardless of belief type.

---

Q: Debate's differential impact and implications for polarization

The reviewer is correct that debate strengthens mainstream believers' accuracy more than it converts skeptics. However, it's crucial to note that both groups move toward truth—mainstream believers improve from 64.2% to 71.1% accuracy, while skeptics improve from 62.0% to 69.0%. The "gap" reflects differential movement toward accuracy, not increased polarization where groups move apart.

Our climate study shows similar patterns: both belief groups improve accuracy with debate, though mainstream believers benefit more. This suggests the finding generalizes beyond COVID-19.

Regarding real-world political polarization, the key distinction is incentive structure. Our debate protocol explicitly optimizes for truth-seeking with verifiable ground truth, which is why we chose scientific over political claims—the latter often lack verifiable ground truth and raise ethical concerns about belief manipulation. We'll add a "Broader Impacts" section discussing how debate systems need truth-seeking incentives to avoid polarization.

---

Q: IRB and ethical considerations

Our IRB-approved study collects only broad demographic categories (age ranges, location type, education level, income brackets, political/religious affiliations) - No personally identifiable information is collected. We ensure complete anonymity through aggregate profiles and obtain informed consent. We will add these safeguards to Section 2.1.3 and are happy to share full IRB documentation.

---

We hope these comprehensive responses fully address the reviewer's concerns and would be grateful for their consideration in revisiting the score.

References

1. Diggelmann et al., "Climate-fever: A dataset for verification of real-world climate claims", arXiv 2020.
2. Costello et al., "Durably reducing conspiracy beliefs through dialogues with ai", Science 2024.
3. Michael et al., "Debate Helps Supervise Unreliable Experts", arXiv 2023.
4. Khan et al., "Debating with More Persuasive LLMs Leads to More Truthful Answers", ICML 2024.
5. Costello et al., "Just the facts: How dialogues with ai reduce conspiracy beliefs", 2025.

Dear Reviewer bvi9,

Thank you for your response. We believe there may be a misunderstanding about the Khan and Bowman papers.

However, Khan et al. explicitly state: "We give the debaters full access to the underlying text while judges have no access to the text." The debaters use both the provided text AND their training knowledge.

Similarly, Bowman et al. state: "We expect the model to be able to use significant domain knowledge from pretraining that our human participants aren't familiar with."

Both papers test information asymmetry (judges lack access to source material) while AI systems actively use their background knowledge—exactly as in our study. Neither paper removes AI's training knowledge.

Regarding your concern that "results can be explained by the true claim simply having more backing in the literature," we reiterate that we provide equal numbers of reference sources for both correct and incorrect positions. The sources supporting incorrect claims are carefully curated from credible-appearing outlets to ensure they seem as convincing as mainstream sources. Moreover, Sharma et al. (2024) show LLMs can be sycophantic and persuasive even when optimized for truthfulness, Wen et al. (2024) and Hubinger et al. (2024) demonstrate that LLMs' persuasive capabilities persist through safety training, and Salvi et al. (2024) find LLMs can construct compelling persuasive arguments that outperform humans. These findings suggest that LLMs can construct strong persuasive arguments supporting incorrect positions.

We respectfully reiterate that testing on topics where AI lacks knowledge would eliminate the very capability asymmetry that scalable oversight must address.

We hope this clarifies our experimental controls and alignment with foundational scalable oversight research.

Best regards,

Authors of Submission 20482

References:

• Sharma et al., "Towards Understanding Sycophancy in Language Models", ICLR 2024.
• Wen et al., "Language Models Learn to Mislead Humans via RLHF", arXiv 2024.
• Hubinger et al., "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training", arXiv 2024.
• Salvi et al., "On the Conversational Persuasiveness of Large Language Models", arXiv 2024.

Dear Reviewer bvi9,

We are glad that our response helped address your other concerns. Regarding your main concern that "correct view having more arguments backing it in the literature," we want to clarify that this is not the case in our experimental design.

As stated in lines 101-102 of our paper: "Both protocols provide AI systems with relevant reference sources for evidence-based arguments that use proper citations." Importantly, we provide equal numbers of reference sources for both correct and incorrect positions to ensure fairness.

As detailed in Appendix B.3, we carefully curated source pools for both sides:

• For correct claims: We provided factual sources
• For incorrect claims: We provided sources that appear reliable but support incorrect positions, collected through targeted searches across various domains

The key point is that misinformation sources were specifically selected to be plausible and convincing, not merely obvious misinformation. We ensured both sides had substantial, citable material of comparable quality and quantity. This design choice eliminates the advantage that correct positions might have from simply having "more arguments in the literature."

To make this clearer, we will expand Section 2.1.2 to explicitly detail:

1. Equal number of reference sources provided to both positions
2. The careful curation process ensuring misinformation sources appear credible
3. How this design controls for any potential literature advantage

This experimental control ensures that debate's success stems from its adversarial mechanism for truth-seeking, not from an imbalance in available evidence.

Thank you for highlighting this important clarification need. We hope this clarification addresses your main concern about the potential confound, we would be grateful if you would consider revising your score, which would mean a great deal to us.

Best regards,

Authors of Submission 20482

We thank reviewer feUp for their constructive feedback. We appreciate that the reviewer finds our work addresses an interesting and valid issue in AI alignment and safety, recognizes our clear and well-presented motivation, values our intuitive use of adversarial AI debate to correct cognitive bias, and acknowledges our comprehensive two-study design covering both human-in-the-loop and AI-only supervision with diverse belief groups.

We address the reviewer's concerns about baseline strength, generalization, appendix reliance, and visualization needs below.

---

Q: Abstract clarity, visualization needs, and reducing appendix reliance

We appreciate the reviewer's thorough feedback about improving clarity and accessibility. We understand that the paper's structure and key details need to be more transparent in the main text.

In the final version of the paper, we will revise the manuscript to address all the specific concerns:

Abstract restructuring

Following the reviewer's suggestion, we will restructure the abstract to clearly map out:

1. Study 1 examines how human judges with mainstream vs. skeptical COVID-19/climate change beliefs evaluate claims through debate (two AI agents argue opposing sides) vs. consultancy (single AI advisor)
2. Study 2 tests whether AI judges with/without human personas can effectively supervise these same protocols

This will make the relationships between studies, judge types, and protocols immediately clear as requested.

Introduction and visualization

Addressing the specific questions about study setup and claim examples, we have prepared a comprehensive overview figure showing: example claims (e.g., "Homemade cloth masks are less effective than surgical ones"), how evidence is presented through web sources with URL citations, complete participant flow from screening to final judgment, and side-by-side protocol comparisons. Due to NeurIPS rebuttal guidelines, we cannot include it here, but this figure will appear prominently on page 2 of the camera-ready version.

Moving core content to main paper

Following the recommendation to reduce appendix reliance, we will relocate essential methodological details including:

1. Table of representative claims showing the controversial yet verifiable nature of our task
2. How LLM debaters/consultants access and cite legitimate web sources during arguments
3. Participant recruitment process and demographics
4. Detailed protocol specifications from the appendix that are crucial for understanding our work

The camera-ready version will ensure the main paper stands alone with all information needed to understand and evaluate our methods and findings. Thank you for this constructive feedback that will improve our manuscript's clarity and accessibility.

---

Q: Limited generalization beyond COVID-19 and weak consultancy baseline

Thank you for raising these important concerns about generalization beyond COVID-19 and the strength of our consultancy baseline. Following the reviewer's specific suggestion to test climate change claims, we conducted an extensive additional study with human judges on climate change claims to directly address both issues.

New Climate Change Study with Stronger Baseline

To demonstrate generalization, we recruited 146 participants (73 per belief group) from an initial pool of 1,000 based on their climate change beliefs (human-caused vs. natural cycles), yielding 421 debate sessions and 424 consultancy sessions after quality filtering. We selected claims from the Climate-Fever dataset [1,2] using GPT-4o screening followed by human curation to ensure they satisfy our scalable oversight criteria (details in Section 2.1.1 and Appendix B.1).

Critically, we strengthened the consultancy baseline by allowing consultants to choose which position to argue rather than random 50/50 assignments. This addresses the reviewer's concern that our original baseline was artificially weak. Now consultants argue for the correct answer ~92.54% of the time (giving consultants significant advantage), creating a much more challenging comparison that better reflects real-world scenarios where AI assistants like ChatGPT naturally provide their best assessment rather than randomly arguing for correct or incorrect positions.

Results

Table: Human judge accuracy before and after debate versus consultancy interventions for climate claims

Belief Groups:

• Belief Group 1: Climate change is primarily caused by natural cycles and variations
• Belief Group 2: Climate change is primarily caused by human activities

Debate achieves 72.1% accuracy compared to stronger consultancy's 68.3% accuracy—a 3.8% improvement. This shows debate's advantages generalize beyond COVID-19 to other contentious scientific domains even with a substantially stronger consultancy baseline.

Additionally, we conducted LLM judge studies on climate claims (detailed in Appendix D.2) which show similar patterns—debate provides better performance than consultancy across both COVID-19 and climate domains, with debate particularly benefiting weaker judges like Qwen-2.5-8B.

We will update Figure 1 to include these climate results and revise Section 2.1.2 to clearly describe our stronger consultancy protocol, directly addressing the reviewer's concerns about both generalization and baseline strength.

---

Q: Limitations regarding deception and potential misuse

We agree this is an important consideration. We will add a dedicated "Broader Impacts and Ethical Considerations" section addressing the dual-use nature of our findings. While our work demonstrates how debate guides people toward truth despite biases, we acknowledge that understanding persuasion mechanisms could potentially be misused.

We will clearly discuss:

1. How adversarial debate systems could be exploited if incentives are misaligned
2. Risks that persona-based persuasion insights could enable targeted manipulation (see our response to reviewer bSCe for discussion of personalization)
3. Safeguards for responsible deployment—ensuring debate systems optimize for truth-seeking rather than mere persuasiveness

Our goal is developing bias-resilient oversight that promotes accurate judgment, not enhancing deceptive capabilities. Thank you for highlighting this critical ethical dimension.

---

Q: Formatting issue at lines 277-294

Thank you for noting the formatting issue. We will ensure the final version is properly formatted according to the guidelines.

---

We hope our responses—the extensive new climate study with strengthened baseline, our commitment to restructure the paper with visualization and reduced appendix reliance, and expanded discussion of broader impacts and ethical considerations—fully address the reviewer's concerns. If the reviewer finds these improvements satisfactory, we would be grateful for their consideration in revisiting the score.

References

1. Diggelmann et al., "Climate-fever: A dataset for verification of real-world climate claims", arXiv 2020.
2. Gabriel et al., "Misinfo reaction frames: Reasoning about readers' reactions to news headlines", ACL 2022.

We thank the reviewer for their thorough and constructive feedback. We appreciate that the reviewer recognizes our work's practical significance in addressing misinformation and LLM safety, acknowledges our careful experimental design with real-world COVID-19 claims and diverse belief groups, and values our nuanced findings showing specific scenarios where debate outperforms consultancy.

We address the reviewer's thoughtful concerns about baseline comparisons, generalization, and ethical implications of personalized AI judges below.

---

Q: Limited generalization beyond COVID-19 and weak consultancy baseline

Thank you for raising these important concerns about generalization beyond COVID-19 and the strength of our consultancy baseline. We conducted an extensive additional study with human judges on climate change claims to directly address both issues.

New Climate Change Study with Stronger Baseline

To demonstrate generalization, we recruited 146 participants (73 per belief group) from an initial pool of 1,000 based on their climate change beliefs (human-caused vs. natural cycles), yielding 421 debate sessions and 424 consultancy sessions after quality filtering. We selected claims from the Climate-Fever dataset [1,2] using GPT-4o screening followed by human curation to ensure they satisfy our scalable oversight criteria (details in Section 2.1.1 and Appendix B.1).

Critically, we strengthened the consultancy baseline by allowing consultants to choose which position to argue rather than random 50/50 assignments. This addresses the reviewer's concern that our original baseline was artificially weak. Now consultants argue for the correct answer ~92.54% of the time (giving consultants significant advantage), creating a much more challenging comparison that better reflects real-world scenarios where AI assistants like ChatGPT naturally provide their best assessment rather than randomly arguing for correct or incorrect positions.

Results

Table: Human judge accuracy before and after debate versus consultancy interventions for climate claims

Belief Groups:

• Belief Group 1: Climate change is primarily caused by natural cycles and variations
• Belief Group 2: Climate change is primarily caused by human activities

Debate achieves 72.1% accuracy compared to stronger consultancy's 68.3% accuracy—a 3.8% improvement. This shows debate's advantages generalize beyond COVID-19 to other contentious scientific domains even with a substantially stronger consultancy baseline.

Additionally, we conducted LLM judge studies on climate claims (detailed in Appendix D.2) which show similar patterns—debate provides better performance than consultancy across both COVID-19 and climate domains, with debate particularly benefiting weaker judges like Qwen-2.5-8B.

We will update Figure 1 to include these climate results and revise Section 2.1.2 to clearly describe our stronger consultancy protocol, directly addressing the reviewer's concerns about both generalization and baseline strength.

---

Q: Why scientific claims rather than political/ethical topics

We appreciate the reviewer's suggestion about political/ethical topics. We selected scientific domains (COVID-19 and climate change) because they provide controversial claims with verifiable ground truth from scientific evidence—essential for measuring whether protocols guide judges toward factual accuracy. Political topics often lack objective ground truth and raise ethical concerns about changing people's beliefs. Our approach ensures we can rigorously evaluate protocol effectiveness while maintaining ethical standards.

---

Q: Ethical implications of personalized AI judges

We thank the reviewer for this insightful question. We agree that personalized LLM judges pose ethical risks and warrant careful consideration during real-world deployment [5,6]. While we do not endorse any collection of private user information without consent, it is well-known that social media platforms already track user behavior for personalization of feeds [7] and ad-targeting [8]. We envision that users would be able to opt-in to use of personalized debaters or judges that rely on similar tracking or user-volunteered demographic information. While LLM judges can be susceptible to manipulation risks and bias amplification, we find that the adversarial debate ensures that judges are actively exposed to arguments and counterevidence from both sides thus mitigating confirmation bias and echo chamber effects [3,4].

We analyzed the LLM judge's transcripts with LLM-as-judge (Gemini-2.5-flash) to identify biased arguments (evidence of confirmation bias or echo chamber effect) and validated findings through manual checking of randomly selected transcripts, finding no significant differences to the non-personalised LLM judge. We find that the personalized LLM judges consider both the debaters' arguments when reasoning for the final verdict with no bias towards the debater that aligns with their belief personas. We will add this analysis in the final manuscript for further clarity.

We will discuss the ethical considerations of personalized oversight in Section 3.2 and Section A. Specifically, we report how the personas are collected, the prompts used for the LLM judges and release the aggregate profiles for accountability and transparency. Furthermore, we leave it as our future work to develop pluralistic and multi-persona aggregation techniques on our methodology to make personalized oversight more robust.

---

Q: Practical implementation for real-world workflows

Thank you for raising this important concern about practical deployment. Based on our implementation, we can outline concrete integration pathways:

Integration examples

AI debate could replace single-consultant summaries in existing interfaces, such as fact-check panels on news sites or research assistants in enterprise search. Users would see a compact debate widget with 2-3 rounds of opposing arguments drawing from the same trusted sources, providing evidence and counter-evidence at a glance.

Infrastructure and training requirements

Minimal overhead beyond current model-serving stacks—just two parallel LLM calls per round, lightweight orchestration, and a simple UI to present arguments side-by-side. Implementation requires only API access (for closed models) or open-source model deployment with efficient frameworks like vLLM/SGLang (7B models need ~21GB memory, or 4GB with 4-bit quantization). Human evaluators need minimal training—our non-expert judges achieved 70.1% accuracy (vs 60% baseline) with text based bullet-point instructions. The debate format naturally exposes weak arguments since contradictions become obvious when AI agents argue opposing sides.

We will open-source our complete debate/consultancy transcripts with both human and LLM judges to enable training specialized truth-seeking and fact-checking models. Our Discussion section will be expanded to detail these implementation pathways.

---

We hope our responses—the extensive new human judge study on climate claims with stronger baseline, ethical analysis of personalized judges, and practical implementation pathways—fully address the reviewer's concerns. We would be grateful for their consideration in revisiting the score.

References

1. Diggelmann et al., "Climate-fever: A dataset for verification of real-world climate claims", arXiv 2020.
2. Gabriel et al., "Misinfo reaction frames: Reasoning about readers' reactions to news headlines", ACL 2022.
3. Shi et al., "Argumentative Experience: Reducing Confirmation Bias on Controversial Issues through LLM-Generated Multi-Persona Debates", arXiv 2025.
4. Orbach et al., "Out of the Echo Chamber: Detecting Countering Debate Speeches", ACL 2020.
5. Dong et al. "Can LLM be a Personalized Judge?", Findings of EMNLP 2024.
6. Liu et al. "Evaluating Large Language Model Biases in Persona-Steered Generation", Findings of ACL 2024.
7. Michiels et al. "What Are Filter Bubbles Really? A Review of the Conceptual and Empirical Work," UMAP Adjunct 2022.
8. Zeng et al, "What factors affect targeting and bids in online advertising? a field measurement study," IMC 2022.