Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learning

Thank you for your valuable feedback and time spent! We’re glad you found our idea original and likely to influence other work to build on it, our paper easy to follow, and the observations from the multi-objective RL perspective interesting. We would like to address your concerns below.

R4-1 Missing Baselines

We appreciate your concern regarding the need for more convincing baselines. In response, we have added an additional baseline, Elastic Reset(ER). The original ER paper also tested using the IMDB dataset. We adopted the experimental setup from the paper with parameters set to decay=0.995, reset_freq=17, and eta=0.001, successfully replicating their results (ER-17). Moreover, we conducted comparisons with two other sets of reset frequencies. The specific experimental results are as follows:

Furthermore, due to the reset mechanism, the training curve of Elastic Reset (ER) exhibits a peak-like shape, unlike the stable state observed in our case.

R4-2 Insufficient Benchmarks

Thank you for the reminder. We will amend the statement "across diverse reward function landscapes, encompassing both subjective and objective categories" in the revised manuscript.

Regarding the benchmarks, we believe they are currently persuasive enough. Reviewer Syvc mentioned that "the experiments are adequate and effectively support the proposed method," and Reviewer F4Ro also noted that we have utilized modern and relevant benchmarks and models.

However, we understand your concerns regarding the benchmarks. Therefore, we have included an additional, widely used dataset, Anthropic-HH. On this dataset, CORY shows its superior ability to balance task reward and KL divergence than all the baselines (PPO, Elastic reset, REINFORCE). We hope this can addresses your concerns. Training curves are in Figure 4 in the additional PDF.

R4-3 Self-improvement Method

In the domain of LLM fine-tuning, there exist some self-improvement methods, such as SPIN [1] and Iterative DPO [2]. However, these approaches are offline algorithms in supervised manner, where self-improvement stems from making more comprehensive use of the dataset.

In contrast, RL as an online algorithm, is optimized on data generated by itself. In the MARL field, the success of AlphaGO and OpenAI Five has validated the effectiveness of the self-improvement among a multi-agent system that diverse data are generated through competition/cooperation among multiple agents. The problem we are concerned with is how to make the RL fine-tuning of LLM benefit from this self-improvement in multi-agent system [3, 4]. Therefore, CORY introduces a completely different self-improvement mechanism into the fine-tuning of LLMs.

[1] Chen, Zixiang, et al. "Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models." Forty-first International Conference on Machine Learning.

[2] Yuan, Weizhe, et al. "Self-Rewarding Language Models." Forty-first International Conference on Machine Learning.

[3] A social network for AI. Nat Mach Intell 5, 1175 (2023). https://doi.org/10.1038/s42256-023-00769-4

[4] Duéñez-Guzmán, Edgar A., et al. "A social path to human-like artificial intelligence." Nature Machine Intelligence 5.11 (2023): 1181-1188.

R4-4 Hyperparameter Choice

The parameter settings for fine-tuning GPT2 followed the default configuration in TRL for the IMDB dataset, while the fine-tuning for LlaMA2 primarily adhered to the guidelines established by StackLlama. In pursuit of a fair comparison, all parameters were carefully chosen to balance the stability and performance of PPO. And a grid search was conducted over learning rates and eta coefficients within the ranges lr[1e-6, 1e-5, 1e-4] and eta[1e-3, 1e-2, 1e-1, 0.2, 0.3] respectively, to select the parameters that yielded the most stable training for PPO. Given CORY's robustness to parameter variations (as evidenced by the results of adjusting learning rates and kl coefficients presented in Appendix E), the parameters for PPO, with the exception of the learning rate, were directly applied to CORY. In the GSM8K dataset, we adjusted the learning rate for CORY. Important parameters include lr and kl_coef, while parameters such as mini_batch, ppo_epoch are considered to be of relative lesser importance.

R4-5 PPO Hyperparameter Concern

Yes, it is acknowledged that a strictly on-policy PPO is equivalent to A2C. However, the hyperparameter 'Epoch' in our hyperparameter table does not refer to PPO epochs. In our experiments, the ppo_epochs were actually set to 4, which aligns with the standard configuration of PPO. We will update our parameter table to prevent similar misunderstandings.

R4-6 Feedback on the Presentation

Thanks for your suggestion! Regarding your first point, our paper already contains clear experiments related to "policy optimality" (Sections 4.1 & 4.2), "resilience to distribution collapse" (Sections 4.1 & 4.2), and "robustness during training" (Appendix E). We will explicitly add these descriptions to the experimental section of our paper.

As for your second point, we will include definitions of the ">=" symbol in the revised manuscript.

Thank you for your positive remark and insightful feedback! We’re glad you found our method viable for improving LLM RL fine-tuning, our paper is easy to follow, and modern and relevant benchmarks are used in our experiments. Below, we provide individual responses addressing your comments.

R3-1 Limitation Section

Thank you very much for your suggestion. We will carefully use the discussion regarding Moore's Law and will directly incorporate the discussion on computational power consumption into Section 6 of the revised manuscript.

R3-2 Reward Signal in RL Fine-Tuning

From a reinforcement learning perspective, the N steps to generate a sentence (assuming padding is included) constitute an episode. For PPO, at the end of the N steps, gradient ascent is used to optimize the token-level policy in the direction of maximizing the return for those N steps.

R3-3 KL in RL Fine-Tuning

This is a profound question. KL divergence measures the distance between the current LLM’s token-level policy and the pre-trained token-level policy. The pre-trained token-level policy is quite delicate. Deviating too much can lead to the destruction of the language modeling in the pre-trained model, that is, distribution collapse. For instance, if the probability of the eos_token in the token-level policy is too low compared to the pre-trained one, it could prevent sentences from ending, or result in the repetition of the same word until the maximum output length is reached. Distribution collapse can largely be detected by KL divergence, and experimental results also reveal a strong correlation between them.

To avoid distribution collapse, current RL fine-tuning often needs to be early-stopped before reaching a catastrophic level of KL divergence. This places RL fine-tuning in a rather awkward position.

Overall, the benefits of maintaining a low KL can be summarized as follows:

1. Stable training without collapse: This means that RL can explore more data, increasing the chances of finding better policies.
2. Efficient improvement: This means achieving as much performance improvement as possible with as few changes as possible to the reference policy.

We kindly ask the reviewer to read and respond to our rebuttal. During the rebuttal phase, we conducted new experiments that we believe address all the concerns raised regarding the paper and may merit an increase in the score. The experiments can be summarized as follows:

1. New Baselines: Introduced two new baselines: REINFORCE and a strong baseline, Elastic Reset [1] (see attached PDF Figure 3).
2. New Benchmark: All baselines have been compared on the Anthropic HH benchmark (see attached PDF Figure 4).

If there are any outstanding issues, we would appreciate the opportunity to respond before the discussion period concludes. Thank you.

[1] Noukhovitch, Michael, et al. "Language model alignment with elastic reset." Advances in Neural Information Processing Systems 36 (2024).

Thank you for your positive remark and insightful feedback! We’re glad you found our idea is both novel and compelling, our presentation is well-organized and clear, and our experiments are adequate and effectively support the proposed method. Below, we provide individual responses addressing your comments.

R2-1 Theoretical Analysis

Our paper has already provided detailed mathematical modeling for LLM RL fine-tuning (Section 2), token-level PPO (Appendix B), and CORY (Appendix C), along with an analysis from the perspective of multi-objective RL (Section 3.2). However, we are still willing to offer a theoretical analysis of why CORY works from the perspective of game theory.

As shown in Figure 5 in the supplementary PDF, the training phase of CORY can be modeled as an extensive-form two-player game. This is a game of perfect but incomplete information, meaning that the observer is aware of all historical information, but the two agents do not know any specific utility functions. The relationship between the utility functions of the two LLM agents determines the type of game. In CORY, the utility function format $U=R1+R2$ corresponds to a fully cooperative game, while the format $U=R1-R2$ corresponds to a zero-sum fully competitive game.

It is easy to prove that the policy combination corresponding to $U_{\text{obs}} \equiv U_{\text{pio}} \equiv 0$ is a Nash equilibrium of the zero-sum game, where $R_1 \equiv R_2$, meaning that the rewards of both agents are the same under any query. They tend to find policies that are similar in performance but not optimal during training, which is also verified by subsequent experiments (in R2-4).

Conversely, the design of CORY cleverly utilizes the characteristics of the global optimal solution. Assuming that for any task query, there exists an answer that maximizes the task reward, it can be proven that 1. The game has Pareto optimums (constructive proof), and 2. Under the Pareto optimums, both agents receive the same and maximum rewards. However, there are two extremes among all the Pareto solutions. One is both the observer and the pioneer independently generate the optimal response (which is the true global optimum), and the other is the observer completely relies on the pioneer, merely repeating its optimal response. Thanks to the design of role-exchange and collective reward, the policies of the two agents will avoid evolving towards the second extreme and instead evolve towards the first extreme. Thanks, we will add this section to the appendix.

R2-2 Other RL Fine-tuning Baselines

Thank you for your suggestion. We have added two baselines into our analysis: REINFORCE and Elastic Reset. Here is the results of IMDB.

The results indicate that CORY has the ability to maintain a stable balance between the two targets. The detailed training curves are shown in Figure 3 in the supplementary PDF.

R2-3 Emergent Communication

Prior work has discovered that defining communication protocols as vocabularies enables agents to spontaneously emerge their own languages during cooperation. Furthermore, incorporating human labels into the learning process of communication policy can facilitate the emergence (0 to 1) of communication policy that utilize natural language [1,2]. CORY is equivalent to a two-player cooperative game with unidirectional communication grounded in natural language. The communication policies are pre-trained LLMs. The LLM/communication policy has already undergone one emergence during pre-training. By casting the RL fine-tuning of the LLM into a multi-agent context, we anticipate a secondary emergence (1 to 100) of the LLM.

[1] Lazaridou, Angeliki, Alexander Peysakhovich, and Marco Baroni. "Multi-Agent Cooperation and the Emergence of (Natural) Language." International Conference on Learning Representations. 2022.

[2] Graesser, Laura, Kyunghyun Cho, and Douwe Kiela. "Emergent linguistic phenomena in multi-agent communication games." 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019. Association for Computational Linguistics, 2019.

R2-4 Total Reward Design

This is indeed an interesting question. We have noticed that Reviewer 7TR2 also raised a similar issue. Due to the space constraints of the rebuttal, please refer to R1-1 for details on the relevant experimental setup. The results indicate that competitive scenarios significantly underperform cooperative ones. Furthermore, under complete competition (zero-sum game settings), the agents trained to a state of low KL divergence but also low performance (similar but suboptimal), which aligns with our previous game-theoretic analysis.

R2-5 More LLM Agents

Yes! introducing more agents could lead to performance improvements, which represents an exciting direction for future work. We look forward to the participation of more researchers in this area. We have already developed preliminary concepts, which fundamentally concerns the design of communication topologies. There are several simple communication topologies that could be explored, including linear (chain-like), mesh (fully connected), and randomly connected. Recent research [3] has uncovered scaling laws of agent numbers in the collaboration among multiple Large Language Models (LLMs). It also discussed several other insight communication topologies for LLMs (e.g. tree-like, radial).

[3] Qian, Chen, et al. "Scaling Large-Language-Model-based Multi-Agent Collaboration." arXiv preprint arXiv:2406.07155 (2024).

Thank you for your detailed review and valuable feedback! We’re glad you found our proposed method interesting and the writing clear. We would like to address your concerns below.

R1-1 Dual-LLM Setup and Multi-LLM Learning

There has been some work studying the collaboration of multi-LLM (including dual-LLM setup) to accomplish a shared task [1], and we will add this to our reference. To our knowledge, CORY is the first work employing a multi-agent framework for the RL fine-tuning of LLM. As highlighted by the reviewer, the dynamics of competition and cooperation among agents are crucial in multi-agent learning framework.

To investigate these dynamics, we modified the original collective reward of CORY from $R_{self} + R_{other}$ to an $R_{self} + λ * R_{other}$ format. By adjusting the value and sign of $λ$, we could represent varying degrees of competition and cooperation. Additionally, to mitigate the impact of reward magnitude on training, we normalized the reward values. As demonstrated in Figure 1 in the attached PDF, across both the GSM8K and IMDB datasets, the task rewards in competitive settings were significantly lower than those in cooperative settings.

Beyond experimental analyses, we believe that training multiple LLMs in a cooperative setting can be modeled as MARL problems with a shared team reward. Research in the MARL domain on credit assignment (e.g., VDN[2],QMIX[3]) could further enhance the training levels of multiple LLMs. This represents a promising direction for future development!

[1] Wu, Qingyun, et al. "AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation." ICLR 2024 Workshop on Large Language Model (LLM) Agents.

[2] Sunehag, P., et al. "Value-Decomposition Networks For Cooperative Multi-Agent Learning Based On Team Reward." Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems. 2018: 2085-2087.

[3] Rashid, T., et al. "QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning." International Conference on Machine Learning. PMLR, 2018: 4295-4304.

R1-2 MoE Baseline

We agree that it is crucial to fairly compare the total model size as well as the number of models. Regarding the model size, our paper has already conducted ablation studies, comparing GPT2-l (770M) + CORY with GPT2-xl (1.5B) + PPO in Section 4.3, and LLaMA2 7b + CORY with LLaMA2 13b + PPO in Appendix E. The experiments demonstrate that the advantages of CORY do not come from an increase in training parameters.

As for expanding the number of LLMs involved in training, we designed a simple baseline inspired by the MoE approach: aggregating the outputs of two models and selecting the output with a higher estimated value based on the Q-values. The experimental results on the IMDB dataset are as follows:

As you mentioned, we found that using MOE to aggregate responses from multiple LLMs can effectively enhance task rewards, resulting in a 9% improvement compared to PPO. However, by comparing with CORY's results, we discovered that CORY achieves a 23% improvement. This indicates that CORY provides a greater enhancement in performance because it significantly boosts the capability of individual LLM agents through collaborative training. These findings indicate that the key to the effectiveness of the pioneer-observer structure lies in facilitating interactions between models.

R1-3 Fig.2 Explanation

We sincerely appreciate your valuable suggestion and apologize for any confusion caused. Regarding Fig. 2(c), the values of eta from left to right indeed are 1e-5, 1e-4, 1e-3, and 1e-2. We will ensure to annotate this clearly in the revised figure for better understanding.

As for the sub-optimal frontiers, these actually represent the testing KL and reward, which we will explicitly clarify in the revised manuscript to avoid any ambiguity.

R1-4 Possible Heterogeneous Multi-LLM Training

This is indeed a fascinating question. Although the paper emphasizes that CORY is a plug-and-play alternative to RL fine-tuning (requiring only a duplicate of the model to be finetuned, without the need for extra auxiliary models during training), the applicability of CORY's core ideas in heterogeneous multi-LLM finetuning is a topic worthy of exploration.

To this end, we conducted experiments on the GSM8K dataset involving heterogeneous LLM training with LLaMA3-8B, LLaMA2-7B, and GPT-2. As shown in Figure 2 in the attached PDF, in the (LLaMA2-LLaMA3) combination, LLaMA2's training curve exhibits stability similar to that of original CORY, with its training KL values (consistently less than 1) significantly lower than those of single-PPO (which peak at 10). This suggests that the "knowledge transfer" mechanism can effectively alleviate the optimization pressure on LLaMA2 at the KL end. However, in the (LLaMA2-GPT2) combination, due to the insufficient capabilities of the GPT2 model, which produces many erroneous or empty responses, it fails to alleviate any training pressure on LLaMA2, resulting in a KL curve for LLaMA2 that closely mirrors the trend of single-PPO. The weaker model hinders the training of the stronger model, a similar outcome can also be observed in the (LLaMA2- LLaMA3) combination. It is evident that in a heterogeneous setting, CORY's underlying mechanisms still function, but the selection of heterogeneous LLMs and their interactions pose significant challenges, making this an issue worth further investigation.

We kindly ask the reviewer to read and respond to our rebuttal. During the rebuttal phase, we conducted new experiments that we believe address all the concerns raised regarding the paper and may merit an increase in the score. The experiments can be summarized as follows:

1. Dual-LLM Setup: Evaluated both cooperative and competitive settings (see attached PDF Figure 1).
2. Heterogeneous Setup: Implemented two different LLMs in CORY to assess the effectiveness of the proposed mechanism (see attached PDF Figure 2).
3. New Baselines: Introduced two new baselines: REINFORCE and a strong baseline, Elastic Reset [1] (see attached PDF Figure 3).
4. New Benchmark: All baselines have been compared on the Anthropic HH benchmark (see attached PDF Figure 4).

If there are any outstanding issues, we would appreciate the opportunity to respond before the discussion period concludes. Thank you.

[1] Noukhovitch, Michael, et al. "Language model alignment with elastic reset." Advances in Neural Information Processing Systems 36 (2024).