Improving Factuality and Reasoning in Language Models through Multiagent Debate

Thank you for your detailed comments. Please see our clarifications below.

Novelty. The primary novelty of the paper is proposing the methodology of multi-agent debate as an inference-time approach to improving the performance of LLMs across both factuality and reasoning. While prior works have explored how both using ensembles of models improves performance as well using self-reflection of a single model, our paper illustrates how both ideas may be jointly combined to generate more accurate responses.

This combination goes beyond simply having an ensemble of self-reflection agents, as the debate procedure lets each agent reflect on both the generations of the original language model and that of other models. We illustrate the power of the debate procedure in new Figure XXXI (screenshot here) and Figure XXXII (screenshot here) in the updated paper, where we show how debate can be used to self-correct two initially incorrect answers.

We have further provided an additional experiment using debate on the open-source chat-Llama 7B model. Below, we find that similarly, debate improves final performance.

We have added this to Appendix A.1 of the paper.

Computational Expense. While our approach is computationally expensive compared to a single forward pass to a language model, it can be used as a method to take a computationally cheap model (i.e. GPT-3.5) and have it perform similarly to a significantly more computationally expensive model (i.e GPT-4 which is over 100 times more expensive), providing computational savings. In addition, the multiagent debate procedure can be distilled down into another model both for faster inference as well as a method for self-improvement in a language model.

Thank you for your valuable feedback on the manuscript. We have responded to each concern you’ve mentioned below.

Short vs Long. In Figure 3, short refers to a prompt to a large language model that induces debates to be shorter in length, while long refers to a prompt to a large language model that induces debates to be longer in length. We have updated the paper to make this clearer.

Understanding Gains in Debate. The Multiagent Debate procedure in the paper tightly couples two sources of performance gains in LLMs, (1) wisdom of the crowds, where multiple instances of LLMs can jointly leverage to improve performance (i.e. from majority voting) (2) self-reflection, the ability for LLMs to use previous generations to improve their performance. However, the procedure goes beyond directly combining these two aspects because individual LLMs can reflect not only based on their thought process but also that of other agents (which is referred to as debate in the paper). The core reason multiagent debate improves This allows LLMs to recover and generate correct answers in a subsequent round of reasoning even if all answers in the current round are incorrect. This is illustrated in Figure 11 (screenshot here) of the original paper and in Figure A.XXXI (screenshot here) and A.XXXII (screenshot here) in the updated paper.

To more clearly understand the performance of debate, we also consider a new baseline of running ensembles over self-reflection, where we instantiate multiple instances of a self-reflection agent and then take a final ensemble vote over generated self-reflection results. We find that debate outperforms this baseline, indicating that discourse between agents is helpful overall beyond ensembling answers from a self-reasoning language model.

To further understand the performance of debate with more agents, we also consider using a total of 10 agents for debate across 2 rounds of debate in the table below compared to majority vote over 50 agents. We find that having more agents in debate continues to improve the performance.

Gains through Summarization. We found that when context lengths became long, LLMs would struggle to be able to process the entire length of the message (also discussed in [1]). Summarization serves as a way to remove extraneous repeated information generated by each agent to condense relevant information to solve the task. In addition to results on the arithmetic task, we ran additional experiments and found that summarization on GSM leads to a performance of 87.0 compared to the original performance of 85.0 with 3 agents and 2 rounds of debate. We found that summarization on MMLU leads to a performance boost of 73.0 compared to the original performance of 71.1. We have added these experiments to Appendix A.1

Analysis of Agreeableness. In our experiments, we found that agents were agreeable regardless of whether they knew the answer was correct so long as there were similar answers from other agents. To verify the relation between “agreeable” and “a result of instruction tuning or RLHF”, we consider the LLama-2 7B and the RLHF chat-LLama-2 7B models. On the GSM8K task, we find that LLama-2 7B model achieves a consensus of 51.3% after debate compared to consensus of 62.7% with aligned chat-LLama 2 model. On the MMLU task, we find that LLama-2 7B model achieves a consensus of 74.3% after debate compared to consensus of 100.0% with aligned chat-LLama 2 model. This supports the conclusion that RLHF models tend to be more agreeable than the previous unaligned checkpoints. We have added this experiment to Appendix A.1.

[1] Liu et al. Lost in the Middle: How Language Models Use Long Contexts

Thank you for your response.

The reasoning procedure that occurs during multiagent debate is the use of different lines of reasoning from separate agents to stitch together an overall response. When generating initial answers, different agents provide different approaches to solve a problem, where each approach has errors in different places in its reasoning. Self-refinement through multiagent debate allows each agent to see both their original solution and the solution of other agents and allows an agent to stitch together the correct parts of reasoning in its original solution with the correct parts of reasoning in other agents' solutions. This allows an agent to turn two initially incorrect answers to a fully correct one.

We illustrate one example of this process in Figure 11 (screenshot here) of the paper. In this example, in its initial response, the chatGPT agent has a flaw in its reasoning where it does not realize that Carla needs to fully redownload a file. At the same time, in its initial response, the Bard agent realizes that Carla needs to fully redownload a file, but incorrectly computes the time it takes to download a file. Through the multiagent debate procedure, the chatGPT agent can take the correct path of reasoning from Bard in combination with its own reasoning to synthesize a correct solution to the full problem.

Another example of this can be found in the newly added Figure XXXI (screenshot here) and Figure XXXII (screenshot here). In Figure XXXI (screenshot here), in the first round, both agents make different mistakes in evaluating the arithmetic expression, where the first agent incorrectly adds each component together at the end and the second agent carries over the wrong answer from a previous computation. Both agents stitch the reasoning of the other agent with their original reasoning in the previous round to get the final answer correctly.

Dear Reviewer CMZe,

Thank you for your time reviewing the paper. We have run additional evaluations to address the concerns you raised earlier in your response. As the discussion period draws to a close tomorrow, please let us know if there is a need for additional clarification.

Thanks,

Paper Authors

Thank you for your valuable feedback on the manuscript. We respond to each concern you have listed above below and have also updated the paper.

Analysis of Performance Gains From Debate. The Multiagent Debate procedure in the paper tightly couples two sources of performance gains in LLMs, (1) wisdom of the crowds, where multiple instances of LLMs can jointly leverage to improve performance (i.e. from majority voting) (2) self-reflection, the ability for LLMs to use previous generations to improve their performance. However, the procedure goes beyond directly combining these two aspects because individual LLMs can reflect not only based on their own thought process but also that of other agents (which is referred to as debate in the paper). The core reason why this helps performance is that now LLMs can stitch together correct paths of reasoning in one response with that of other responses. This allows the debate procedure to take a round of responses where all answers are incorrect, and recover answers in the subsequent round that are fully correct, by fixing misunderstandings/errors in reasoning using responses from other agents. This is illustrated in Figure 11 (screenshot here) of the original paper and in Figure A.XXXI (screenshot here) and A.XXXII (screenshot here) in the updated paper.

To more clearly differentiate the performance of debate, and simply running ensembles over self-reflection, we consider a new baseline, where we instantiate multiple instances of a self-reflection agent and then take a final ensemble vote over generated self-reflection results. We find that debate outperforms this baseline, indicating that discourse between agents is helpful overall simply just ensembling answers from a self-reasoning language model.

Scaling Up Agents. We compare running multiagent debate over 10 agents over 2 rounds with majority vote across 50 agent responses. We find that even with a very large number of agents, running multiagent debate still outperforms using a majority vote.

Other Model Bases. We have added additional results using the chat-Llama 2 7B model. Below we report the performance of multiagent debate and each baseline. We find that multi-agent debate is also helpful on open-source language models.

We have also added this table in Appendix A.1

Thank you for your valuable feedback on the manuscript. We have addressed each issue you have listed below and have also updated the paper.

Majority Voting in Minerva. We would like to clarify that the multiagent majority baseline we include in Table 1 and 2 of the paper also corresponds to the multiagent majority (auto-debate) baseline in Minerva. We generate multiple answers using a single instance of a language model and use the majority vote to determine the final answer. We have updated the description of the multiagent majority baseline to clarify this in Section 3.1 and have also added an additional reference to the Minerva paper there and in the related work.

Multiagent Majority in Related Work. We have added Table 3 in the related work comparing the difference between the various multiagent majority voting baselines and debate. The primary difference is that we use multiple rounds of communication between agents to achieve the final answer while multiagent majority simply takes the majority vote at the end of the first round (no discourse between models). Both approaches have a flexible number of agents $N$, while the rounds of debate is 1 in multiagent majority and $T$ in multiagent debate.