Multiagent Finetuning: Self Improvement with Diverse Reasoning Chains

Q7: It is pretty costly to train multiple LLMs, especially considering the inference-time compute and resources required to serve N LLMs. A straightforward possible strategy to avoid training multiple LLMs while also maintaining diversity is to include an unique identifier (e.g. an ID) or a special token in the input for each agent. How does this strategy compare to finetuning multiple LLMs?

Thank you for this suggestion. We include an experiment where we avoid training multiple LLMs by feeding a simple unique ID at the beginning of each agent as a method for preserving diversity. Given a generation agent, we feed an ID of the form GEN1 where GEN refers to a generation agent and 1 refers to the first agent. Similarly, critic agents are fed IDs in the form CRIT1. We also include a small description of what generation and critic agents are meant to do at the beginning of the prompt. In general, we find this method comparable to debate. Results are shown in Appendix Section I. We paste results here as well: https://ibb.co/yYY46CZ.

Thank you for your feedback. Here I have some further questions.

The title “Multiagent Finetuning of Language Models” may imply a broader scope than the paper addresses

As mentioned in the original review comment, “Multiagent Finetuning of Language Models” can refer to settings like web agents or gaming where the LLM does not just perform reasoning but also decision-making.

How is a fair comparison with baseline methods established?

How many training tokens and how many rounds of debates are used in the fine-tuning process? A fair comparison should be ensured with a similar amount of training tokens or a similar number of reasoning trajectories.

“Single Agent” and “Multi Agent” are vague and unclear in this paper.

The notions of "agent" and "LLM" should be used rigorously in the context of the paper where multiple LLMs are involved and multiple agent types and debate agents are involved. Multiple agents can be supported by different LLMs or a single LLM. The notion of "single-agent" was confusing for me since the authors use a single agent for multiagent debate, though the actual case is using a single fine-tuned LLM for multiagent debate.

"How does varying the number of agents affect the performance of the proposed method?"

Additional experiments on the number of agents resolve my concerns regarding this question. Is there any discussion on the benefits and limitations of the approach if significantly larger amount of agents are used?

Q1: may imply a broader scope than the paper addresses. Multi-agent applications of language models can indicate a much broader range of settings besides reasoning tasks and multi-agent debate

We have included an experiment in Appendix D to address this, which shows our approach can be incorporated with LLM agents which are applied collaboratively rather than competitively as with debate. We find that multiagent finetuning improves performance over the collaborative settings as well and have updated the introduction with this point.

Q2: “Single Agent” and “Multi Agent” is vague and unclear in this paper.

We will make the distinction more clearer. Single Agent refers to finetuning a single agent and using that single finetuned agent at inference time. The data generation method for that agent could use multiagent debate or some other simpler method for prompting like chain-of-thought. The result is only a single finetuned agent that we can apply complex prompting methods at inference time such as multiagent debate. To answer your question specifically, this refers to the equivalent of one generation agent or standard finetuning using multiagent debate.

Q3: This work might not obey the standard training and evaluation procedure on GSM and MATH dataset, as only 500 examples are selected for training.

Thank you. Due to limited time, we have expanded the evaluation procedure on GSM to finetune on 1K examples and evaluate on 1K examples using LLaMA-3.

Q4: Line 77: How does the proposed approach “promotes diversification within the society of models”?

A key observation of multiagent debate and other multiagent prompting methods is that a single LLM can answer questions using multiple reasoning approaches that emerge from sampling on the probability distribution from the model. Given that the same base LLM can have different answers, the power of an approach like debate is the ability to generate diverse proposals, some of which lead to the correct answer, and integrate them by evaluating the correctness of approaches from other agents.

Multiagent finetuning preserves and emphasizes this diversity by separating the finetuning across each agent. Finetuning leads to a collapse of internal diversity. However, by finetuning each agent separately on the dataset that the agents correctly generated in the previous, we identify reasoning chains that lead to the correct answer and preserve these reasoning chains in individual agents. This feedback loop promotes specialization within the society of models. As an agent finetunes on its own specific responses that led to a correct answer, it can become an expert on specific problems. This leads to stronger specialization over iterations of finetuning. In a toy model mathematical model in Appendix Section G, we provide a simple illustration on how such mathematical specialization can happen.

Q5: How is a fair comparison with baseline methods established? Our baselines use different strategies that include (1) different prompting strategies over base, unfinetuned LLMs, (2) multiagent finetuning with simpler data generation methods than debate, and (3) finetuning methods like STaR. Base provides a comparison between all methods and using a single agent at inference time. Majority provides a comparison with a multiagent prompting approach where we simply take a majority vote over the agent responses. Similarly, Debate is the closest comparison to our approach since it uses a similar number of models at inference time. Finally, Majority FT and STaR are finetuning methods. Majority FT is a finetuning method that uses multiple agents, like our method. STaR is a state-of-the-art method for self-improvement.

Q6 How does varying the number of agents affect the performance of the proposed method?

Appendix Section F includes an experiment with 5 agents. We find that more agents leads to improved results overall.

We thank the reviewer for reading through our response! We address the additional comments below.

Q1: “Multiagent Finetuning of Language Models” can refer to settings like web agents or gaming where the LLM does not just perform reasoning but also decision-making.

Thank you for the feedback. We recognize that the title may be too broad. We will change the title to convey that our focus is primarily on self-improvement. Our new title is Self Improvement in Language Models through Multiagent Finetuning. We’re also happy to further change the title of the paper if the reviewers so desires.

Q2: How many training tokens and how many rounds of debates are used in the fine-tuning process?

For our method, we always used 2 rounds of debate. This is the same as the Debate baseline, which does not use finetuning. We compare the number of training tokens for generation agents and critic agents in our method in comparison to the number of training tokens used for Majority-FT. We evaluate this on the MATH dataset with Mistral-7B. For multiagent finetuning, a generation agent is finetuned on 55889 tokens and a critic agent is finetuned on 119645 tokens. In comparison, we use 353814 tokens with Majority FT. Majority-FT uses a majority vote over final responses and collects all responses that match the majority voted answer. Our method only selects one response per input example. Furthermore, generation agents are only finetuned on examples that have a final answer in the first round which matches the majority voted solution in the second round. This reduces the number of selected examples further in comparison to Majority-FT. So we finetune on significantly more tokens with Majority-FT in comparison with our method. We believe this shows that our comparison with other finetuning baselines is fair since we finetune on fewer tokens in our approach (with the total aggregated number of tokens across multiagent finetuned models similar to that of Majority-FT).

Q3: The notions of "agent" and "LLM" should be used rigorously in the context of the paper where multiple LLMs are involved and multiple agent types and debate agents are involved. Multiple agents can be supported by different LLMs or a single LLM.

We apologize for the confusion between “model” and “agent”. We have significantly revised the method section including section titles to make the distinction clear that a “LLM model” are the parameters that are trained, and an agent is constructed from the model. We have highlighted all changes in blue.

Q4: Is there any discussion on the benefits and limitations of the approach if significantly larger amount of agents are used?

This is a great point! Reviewer Tw8m asked a similar question as well. We indeed have an experiment illustrating how the approach works well with 5 agents in Appendix Section F, where additional agents lead to significant improvement across all tasks, and significantly outperforms all baselines, illustrating how we can use an even larger number of agents in our approach. Theoretically, Appendix Section G shows that more agents allow for better coverage of different specialization skills, outside of three used as part of the original analysis. In addition, prior work [1] has shown significant benefit from incorporating more agents in inference time approaches like multiagent debate.

However, a limitation of using more agents is added compute and memory complexity. For example, we had hardware limitations that prevented us from experimenting with additional agents on open-source models, as each additional model required additional GPU time to train. There are some initial ways of overcoming these limitations. A promising venture could include some of the smaller LLMs that have 1-2B parameters such as LLaMA-3.2. Memory optimization methods like quantization could also be applied to allow for more agents to be loaded in memory.

[1] Du et. al. Improving Factuality and Reasoning in Language Models through Multiagent Debate, ICML 2024.

We thank the reviewer for their thoughtful review. We address the comments below.

Q1: However, it would be helpful to clarify the specific objectives each role optimizes. Additionally, I suggest emphasizing that only two roles are used in this paper (generation and critic) to avoid confusion.

We restate the roles of our agents and their optimization procedure precisely here.

Generation Agent: The role of a generation agent is to generate accurate responses to input questions. These should rely on diverse reasoning chains to promote diversity. The finetuning procedure is as follows. Each generator agent is trained on a dataset curated from its response from the first round. For example, the $n$-th critic agent uses the text from $\{y_{1, n}\}$. We only keep data where the correct solution is achieved i.e. $y_{1, n} = \hat{y}$ is achieved as shown in Figure 2 in the main paper.

Critic Agent: The role of a critic agent is to provide accurate critiques to responses from other agents and use these responses to provide an updated answer. The finetuning procedure is as follows. Each critic agent is trained on a dataset curated from its own responses and the corresponding responses from previous rounds. For example, the $n$-th critic agent uses the text from $\{y_{1,n} \text{ to } y_{M,n}\}$. The critic agent might generate either correct or incorrect solutions. We only keep the data where the correct solution $y_{M,n} = \hat{y}$ is achieved, as shown in Figure 2 in the main paper. For these data points, if the response of the corresponding generator in the first round $y_{1,n}$ is wrong, we classify such data as negative data. Including this data allows the critic agent to develop the ability to correct answers even when the initial response is wrong. The remaining data, where both the generator's and the critic's responses are correct, are classified as positive data. We sample positive and negative data from these two sets and mix them together to train the critic agent.

Q2: find the claims made in Section 4.3 not very convincing

We address the three objections point by point.

(a) We agree! We re-run an extended evaluation on 1000 data points for all reported numbers. We find that our method significantly outperforms all baselines on 1000 held out data points when finetuning on a different dataset. We find that this performs significantly better than all other baselines. We paste results here: https://ibb.co/JKpZ7WJ.

(b) We apologize. This should have been included in the final result. We have added error bars to the figure to analyze for significance, please see Figure 11 in the new version of the paper. Our method has significant improvement when finetuning on MATH and testing on GSM.

(c) We include a new setting where finetune on the Arithmetic and test on GSM and report this in Appendix Section L. We find that this also performs significantly better than all other baselines. We past the result here: https://ibb.co/WnpGh4Z.

Q3: The limitations section mentions increased computational demands but does not quantify these trade-offs.

This is a fair point that we should have made clear. To run experiments we used a combination of eight H100 GPUs and four 40GB A100 GPUs. Running multiagent finetuning was difficult due to memory usage from loading in 6 finetuned models when using open source models. In general, this takes a total of 120 GB of GPU memory for Phi-3 and about 240 GB of memory for larger models like Mistral and LLaMA-3 if you load all six finetuned LLMs at once. In general, we found that inference time for 500 problems took between 12-16 hours depending on the size of the model when using multiple GPUs. When using one GPU but changing which agent is on GPU, inference time could take 24 hours. For open-source models, code was written in pytorch but designing models in JAX/FLAX may lead to further improvements in runtime. We expand our limitations section with this discussion.

Q5: Initial improvements in diversity We compare the diversity measures after one iteration of finetuning for Mistral using responses from the ablations in Table 2.

We find that methods using multiagent finetuning generally has more diverse responses than generating responses using a single finetuned agent. We can attribute the close diversity for Multiagent FT and Multiagent FT w/o summary to summarization having little effect on diversity but instead on incorporating information and accuracy.

Q4: If yes, can this method be integrated with DPO (a combination of query and human feedback preferences data) and PPO (a combination of query and reward signal data) approaches? Please provide a detailed discussion.

Yes, all agents are finetuned using SFT. This means that all agents could also be finetuned using DPO and PPO, as long as there are preferences between responses. One way to incorporate DPO and PPO would be to have critics generate preferences between responses or assign rewards in order to guide optimization. For example, in DPO, critic agents would assign preferences and these preferences can be leveraged to align generation agents’ outputs with the preferred response, optimizing for higher-quality outputs. PPO would have a similar mechanism, where reward signals from critic agents can be incorporated, allowing generation agents to iteratively refine their responses through reward feedback. This could allow the feedback to go beyond binary correctness and use more subjective metrics.

There are challenges with integrating DPO or PPO. Preferences or reward signals from critics may be noisier and less robust in comparison to explicit correctness labels used in SFT. This could lead to instability during training, which has been observed in prior work with PPO [1]. PPO, in particular, may require additional tuning to manage noisy signals increasing computational complexity compared to SFT. SFT offers simplicity, reliability, and efficiency, but DPO and PPO are useful for scenarios where labels aren’t available or when we want to optimize for more subjective metrics beyond correctness.

Q5: How is the Combination of datasets implemented in line 25 of Algorithm 1 pseudocode? What does the hyperparameter denote?

The hyperparameter term "w" refers to a weighting parameter used for sampling data from two different sets. In this context, "w" represents the proportion of data sampled from the first set, while "(1 - w)" represents the proportion of data sampled from the second set. This has been updated in the text.

Q6: How do the authors view the relationship between the number of agents and performance improvement, as well as the issue of number versus resource consumption?

We believe that incorporating more agents will lead to better performance. Empirically, this has been noticed in inference-time approaches like multi-agent debate where more agents leads to stronger results. Intuitively, our setting will also benefit from additional agents. More agents will lead to more specialization through finetuning as we cover more reasoning chains stored in the model. However, running experiments with many agents is challenging as it is very computationally expensive to train 5 or more separate models. We find in our experiments that there are significant result even with 3 agents and believe this trend with more agents.

In terms of performance of 5 agents reported in Table 4 of the paper, we note that our approach with multiagent finetuning does lead to a large boost in performance compared to baseline methods. The performance gains may look smaller than those than Table 1, but this is because we only report comparisons with the strongest baselines in Table 1, and because all methods already get much higher performance with more agents, giving less room for additional improvements.

Q7: The selection of baselines. From the setup of the baselines, this work seems more like an “A+B+C” approach.

We're not completely sure what the reviewer means by a "A+B+C" approach. Our approach gives a method in which we can directly specialize a set of language models so that they can coordinate together to solve complex reasoning tasks. There isn't really a specific "A" + "B" + "C" set-up in this approach (unless the reviewer refers to the individual components of multiagent debate) where we can directly swap a baseline approach for one component and maintain the other components of our approach (since other methods focus solely on either self-improvement or test-time inference). If the reviewer refers to the individual components of multiagent debate, we actually investigate the effect of individual components of our approach such as multiple models, debate, and separate verifier / critic models in Table 2 of the paper.

[1] Rui Zheng et. al. Secrets of rlhf in large language models part i: Ppo. arXiv, 2023.

We thank the reviewer for their thoughtful and constructive review. We address specific points below.

Q1: The paper exclusively uses mathematical language reasoning tasks, and each task is not particularly challenging.

Thank you! We include a more challenging setting by using all levels of the MATH dataset instead of levels 1-3, evaluating over LLaMA-3 (8B). We find that multiagent finetuning has significantly better performance than all baselines and has continuous improvement across iterations of finetuning. We compare with baselines here: https://ibb.co/fqppHjW and over iterations of finetuning here: https://ibb.co/qgJ0WwC.

We also include a result on a dataset related to factuality, MMLU. We find that multiagent finetuning has significantly better performance, achieving 68.80%. See results here: https://ibb.co/7r722kp.

Our proposed method outperforms baseline approaches not only on the more challenging math tasks but also on tasks beyond mathematics, such as those requiring factual accuracy.

Q2: The performance is not significant, especially considering the introduction of multiple large models collaborating. It may be worthwhile to quantify other metrics beyond answer accuracy, for example, the KL divergence of each LLM from the original LLM distribution?

We thank the reviewer for their valuable feedback. We believe that our method does demonstrate a significant improvement over the baselines in terms of answer accuracy as illustrated by our statistically significant results in Table 1 and other tables in paper.

As suggested by the reviewer, we have added an additional experiment computing the KL divergence of each LLM from the original LLM distribution. We have included results in the new Figure 10 in the paper and also an attach an image of the results here https://ibb.co/10FDvbW. Our approach obtains a significantly higher KL divergence from the original LLM distribution than single model fine-tuning. In addition, we have also added a new Figure 13 where we measure the likelihood of generated responses from one agent under other multiagent fine-tuned agents. In this figure, we find additional evidence of diversity in the agents over rounds of finetuning. This is shown here as well: https://ibb.co/FscDc97.

In addition, we would like to note that in the original submission, we already included evaluations of diversity on a broader range of metrics. For example, in Figure 3 we include embedding distance, in Figure 7 in Appendix Section C.2, we analyzed the KL divergence between our proposed method and the baseline method that uses single-agent fine-tuning. Additionally, in Figure 6 of Appendix Section C.1, we evaluated the consensus of the generated results. Our method demonstrated greater consensus and diversity compared to the baseline across all these axes.

Q3: In other words, diversity is only brought about by the quantity of agents. The diversity cannot be theoretically guaranteed, nor can it be conceptualized and defined for specific agents after it appears.

One way to conceptualize the findings in this work is to think of a toy setting where we apply an LLM on a synthetic task with subtasks -- A, B, C -- with an initial distribution of 33/33/33 among the subtasks. We can activate the heterogeneous characteristics among agents in this framework where the agents are finetuned on their own outputs, naturally specializing them in tasks the agents excel at.

Consider a scenario with 3 agents. If we expose each agent to an individual subtask of specialization, this guarantees agent specialization. This is because we only finetune each agent on answers it gets correct using multiagent debate. Over iterations of finetuning, the process leads to self-improvement where the agent becomes specialized in task A, B, C. Theoretically, this process should converge to a state where each agent exhibits 100% on the respective subtasks. This shows that diversity and task-specific heterogeneity are naturally induced without additional intervention. We provide a mathematical derivation of this phenomenon in Appendix Section G.

One question is how subtasks manifest. This happens from internal model specialization. LLMs are probabilistic. They can respond to questions in different ways using different reasoning chains that are derived based on sampling. The ability to specialize is useful for multiagent inference methods like debate where different reasoning chains can be used to find the answer to a question. Our finetuning method encourages individual agents to concretely discover subtasks -- A, B, C -- and finetune to enhance specialization.

Thanks for your response, which has addressed most of my concerns. I now have primary questions regarding the diversity among agents. Firstly, I am curious about the mechanisms through which diversity among agents is achieved. The paper does not seem to have an explicit mechanism for the definition and assignment of agents’ roles. I would appreciate it if the authors could provide a specific example to illustrate this. Secondly, I would like to delve deeper into the origin of this diversity. For instance, in the case of five agents, does their diversity stem just from the increased number of agents, or is it a result of collaboration among the agents? Considering my initial rating was between 4 and 5 (the highest possible at the time), I will now add one point, without changing my level of confidence, making my temporary score a 6.