DebateGPT: Fine-tuning Large Language Models with Multi-agent Debate Supervision

Thank you for your thoughtful review! We respond to the points below:

Question:If I'm understanding correctly, the Multi-Agent debate method used to generate data seems to following the approach in [1]. However, it's important to note that when comparing it to the baseline models, we're only looking at GPT-3.5 and GPT-4, and there's no direct mention of [1].

Answer: We thank the reviewer for the valuable comment. We have added the results of fine-tuning GPT3.5 using the data generated by the method proposed by [1] on three evaluation benchmarks. Our method consistently outperforms [1].

Question:In the main contributions of the paper, it claims to "offers a more economical alternative to leveraging models like GPT-4". However, when we look at Figure 6, we see that both options are similarly priced. Furthermore, taking into account their divergent performance makes it difficult for me to fully support this claim.

Answer: Figure 6 shows that on 500 examples, generating responses to 500 examples with multi-agent debate costs 30 dollars less than GPT-4. As the number of examples increases, this cost will add up considerably. In our case with generating a dataset of 5K examples, we saved approximately 300 dollars using our approach with multi-agent debate.

Furthermore, our performance is comparable with GPT-4. Table 1 shows that our method has a win rate accuracy of 88.6 and GPT-4 is 89.0. Table 6 in Appendix F (pasted below) shows that our method is consistently better than other methods at generating high quality data.

Question: Which LLMs are included in the Multi-Agent used for data generation?

Answer: As stated in Section 3.2, we use GPT-3.5 in the multi-agent debate for data generation.

Thank you for your thoughtful review! We respond to the points below:

Question: it's worth noting that the utilization of confidence scores and filtering techniques is a standard practice in the field of dataset augmentation, lacking any groundbreaking innovation apart from the incorporation of multi-agent debate.

Answer: Based on our knowledge, confidence scores and filtering techniques are the first time used for data generation under the multi-agent debate framework. We haven’t seen any work use confidence scores or filtering specifically in designing text datasets. We would appreciate it if the reviewer could point to papers that use confidence scores and filtering techniques in the field of dataset augmentation.

Question: To make a meaningful assessment of its effectiveness, more comprehensive experiments should be conducted to compare these approaches with existing methods in data augmentation.

Answer: We thank the reviewer for the valuable comment. We include more baseline methods for generating data and measure the data quality of each method in comparison with our design with multi-agent debate on 1000 examples randomly sampled from Alpaca. We report this in Appendix F. We see that our method consistently generates higher quality data than every other prompting method. We also include this table here:

Thank you for your thoughtful review! We respond to your points below.

Question: The supervised data seems composed of questions and answers given by the multi-debate framework, rather than multi-agent debate demonstrations. It does not make so much sense to me as there are incorrect answers in the answers as well, right? (Plz correct me if any falsehood) Have the authors tried using answers given by human annotators?

Answer: This is correct. We construct our dataset by extracting the question and the final round response since our findings show that responses in later rounds are more accurate than responses in earlier rounds of multi-agent debate. We show the results comparing DebateGPT-3.5 with a GPT-3.5 fine-tuned on the entire debate demonstration. We randomly select 100 examples from two evaluation datasets, GSM and MMLU, respectively. In the table below, we see that that fine-tuning on the final response is superior to fine-tuning on the entire debate demonstration. This is because when we fine-tune using the entire debate demonstration, the fine-tuned model is confused by the incorrect information in the early rounds of debate. Another reason is that the length of some demonstrations exceeds the context window length of GPT-3.5 and the conversations in later rounds of debate are discarded. Thus the method we proposed in the paper that utilizes the most accurate final debate results achieves better performance.

The results have been added to Appendix H of the paper and are included below.

Multi-agent debate can generate incorrect answers like any automatic generation method. However, our goal in this work is to avoid human annotation, which is expensive, time-consuming, and hard to scale, while constructing datasets (See paragraph 1 of our introduction in our paper) . Based on this, our paper shows that multi-agent debate can generate high quality data in an efficient manner and this high quality data can lead to stronger fine-tuning results as shown in the paper and the results above.

Question: Human-in-loop is so expensive, utilizing other methods including debate might be a panacea, but need to carefully investigate. Does the multi-agent debate framework apply to the evaluation in the experiments?

Answer: We assume the reviewer is asking whether multi-agent debate can be added as an evaluation baseline. To be clear, multi-agent debate is only used for data generation in our paper. Our evaluation only uses a single agent the same as baseline methods for fair comparisons. However, multi-agent debate can also be used for more accurate evaluation which is not the focus of this paper.

Question: The confidence score sounds interesting, however, the reliability of the scalar given by LLM is still worth doubting (an empirical concern from the reviewer), has the author done any faithfulness investigation on that?

Answer: This is a good point! We have conducted a preliminary investigation into how reliable the confidence score given by the LLM is. Based on our results, LLMs usually have a confidence score > 90 and are even confident in incorrect answers. However, in multi-agent debate, we find that the agents will adjust their response based on the response from other agents, and responses with higher confidence scores are more likely to be accepted by the other agents.

To better understand the reliability of our confidence score, we design a small human evaluation. We sample 100 examples from our dataset. For each example, we had the evaluator analyze the quality of the response. The evaluator then analyzed the confidence score in the final round. If the confidence score was low (< 95) and the quality of the response was low, or if the confidence score was high (>= 95) and the quality of the response was high, then we say that the confidence score reliably reflects the quality of the response. We calculated the percentage of examples where the confidence score reliably reflects the response quality according to our definition in the 100 examples. We find 82% of confidence scores in the last round of debate accurately reflected the correctness. Our human evaluation shows that the confidence scoring method in our paper can reflect the relative accuracy of the generated response, which is reliable enough as a reference when agents participate in multiagent debate.

Question: It could also be beneficial to discuss prior work on using multi-LLM agent cooperation to generate instruction finetuning datasets [1].

Answer: Thank you for your interest in our paper! We have added discussion of [1] to our related work section, highlighted in blue!

[1] Li, Guohao, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. "CAMEL: Communicative Agents for" Mind" Exploration of Large Language Model Society." NeurIPS 2023