Self-playing Adversarial Language Game Enhances LLM Reasoning

We appreciate the suggestions and comments from Reviewer 1AP3. To address the reviewer's concerns:

1. About why reasoning ability improves by training within the game: We provided a brief explanation about why self-playing adversarial games can improve the LLM's reasoning in the abstract and the introduction sections, that in adversarial language games such as Adversarial Taboo, both attacker and defender need to have high-level language capacities in terms of expression, understanding, and knowledge for the reasoning of the target word. In our opinion, the adversarial language game for reasoning improvement is an analog to sports competition for fitness. For example, a person can play basketball, tennis, or running competition to strengthen his/her body. We imagine that in the future, LLMs can also strengthen their capability by self-playing many different kinds of language competitions, of which Adversarial Taboo can be one candidate.

2. We began our experiments with negative advantages but failed to obtain the desired self-play performances. That is the reason why we use the ReST method which filters the negative advantages in the current paper draft. Since our experiments with negative advantages failed (the performances are even worse than the base models), we did not report them for we haven't figured out whether the failure with negative advantage is because of our RL implementation or the instability of negative policy gradients. The one thing we can make sure of is that without negative advantages the SPAG framework works effectively. We will recheck our implementation of negative advantages and provide a more comprehensive discussion about the negative advantages in the next revision.

3. About the PPO objective: Although in recent RLHF papers, the PPO (or more precisely, the LLM alignment objective) is with regularizer KL(\pi || \pi_ref), in the original TRPO[2] and PPO[1] paper the regularizer is KL(\pi_ref || \pi ). I think the term KL(\pi || \pi_ref) is more convenient for DPO[3] to derive the closed-form solution for the optimal policy.

Reference: [1] Proximal Policy Optimization Algorithms, 2017 [2] Trust Region Policy Optimization, 2015 [3] Direct Preference Optimization: Your Language Model is Secretly a Reward Model, 2023

Thanks for the reply. Regarding the second point about negative advantages, how would the method perform with negative advantages filtering, but other data (such as data from no self-play scenarios) were used instead?

We thank reviewer R1Zh for the constructive comments and suggestions. To address the reviewer's concerns:

1. About the evaluation on general tasks: we agree with the reviewer that evaluating general language capacities is important to the SPAG models. We actually have reported the general language understanding evaluation of SPAG models on the Massive Multitask Language Understanding (MMLU) dataset in Table 1&2, which might give the reviewer an overview of the general capacity of SPAG models. Besides, the uniform performance gains of SPAG on various reasoning benchmarks also reflect the generalization ability of SPAG models, since these benchmarks are based on diverse reasoning challenges. Moreover, the SPAG models obtain uniform reasoning gains just by self-playing the adversarial language game without any training in the reasoning training sets (many reasoning benchmarks have their own training sets but we didn't use any of them). This setup also supports the generalization ability of our SPAG method.

2. About the IM+AlpacaSFT baseline performance: we agree that the IM+AlpacaSFT model can achieve a comparable performance with SPAG-1 model. However, the IM+AlpacaSFT has already shown the upper-bound performance of Alpaca-SFT on imitation-learned models. In contrast, SPAG models can continuously be enhanced through iterative self-play and RL training processes. Besides, we did notice that IM+AlpacaSFT outperformed SPAG with Baichuan-2 as the base model on the Mutual benchmark. Due to the limited effort, we did dive deep to find the reason for this observation in the current draft. One potential reason might be the Mutual dataset has a similar domain or text pattern to the text in the Alpaca set. We will make more detailed analysis on the Mutual benchmark in the next revision.

3. About the Figure 1: We are sorry about the confusion caused by the notations in Figure 1. We will add more explanations and descriptions to the notations in Figure 1 in the next revision.

We appreciate reviewer ib6k's detailed comments and review. To address the reviewer's concerns:

1. About RL results:

we did utilize the MDP/RL formalisms to introduce our SPAG objective in equation (13), which is a policy-gradient-based off-policy RL loss. Then we did conduct the RL training with this SPAG loss (eq.13) in an offline scheme on the imitation-learned LLMs. For each LLM, we iteratively conducted three-epoch RL training, denoted as SPAG-1, SPAG-2, SPAG-3. The RL results of these SPAG models are reported in Table 1, Table 2, and Figure 4. From the RL experimental results, we observed clear performance gains in each RL iteration (SPAG-1 $\rightarrow$ SPAG-2 $\rightarrow$ SPAG-3) in terms of LLM reasoning and game-playing.

2. About the contribution of imitation learning:

although imitation learning on GPT self-play traces improves the LLM reasoning uniformly on benchmarks, we did not count these performance gains as the contribution of our paper. The reason for conducting imitation learning is that the original open-source LLMs (LLaMA-2 and Baichuan-2) have limited instruction-following capability to obey the game rules of Adversarial Taboo. Without imitation learning, it is hard for us to collect legal self-play game traces of open-source LLMs. By cloning the behavior of GPT-4, both LLaMA-2 and Baichuan-2 learn to play the adversarial game legally. At the same time, the reasoning abilities of these imitation-learned models also increase naturally because of the SFT from GPT-4 as a much better LLM.

However, the imitation-learned models are the baselines to show the effectiveness of the proposed SPAG method. We iteratively conduct three-epoch RL training based on the self-play traces of the imitation-learned models and continuously observe the performance gains on reasoning benchmarks (SPAG-1 $\rightarrow$ SPAG-2 $\rightarrow$ SPAG-3 in Table 1&2). These results support the effectiveness of the proposed SPAG method. In these rollout & RL processes, the only supervision data we used is the Alpaca dataset, which helps to preserve the general instruction-following ability of LLMs.

To ablate the impact of the Alpaca SFT data, we also continuously train the imitation-learned models with the Alpaca set and show their performances in Table 1&2 (denoted as IM+AlpacaSFT), which have lower performances than SPAG-trained models (SPAG-1 can be regarded as IM+AlpacaSFT+winning-selfplay). This ablation study also supports the effectiveness of the proposed SPAG compared with continuous SFT with the same amount of SFT data (Alpaca).

To sum up, we agree with the reviewer that imitation learning of GPT-4 increases the reasoning of LLaMA-2 and Baichuan-2. However, this process only provides us with better baseline models and does not weaken the effectiveness of the SPAG method, since all the SPAG-trained models are based on these imitation-learned models.

3. About the Reviewer's Question:

We agree that the GPT-4 self-play data on the adversarial taboo game can be regarded as a rich conversational dataset for supervised learning. However, as discussed above, we do not count the performance gains from GPT-4 imitation into the contributions of our paper. After the imitation learning stage, all the taboo conversation data is collected synthetically from the model outputs. The only additional supervision is the Alpaca dataset. As mentioned above, we have conducted ablation studies (imitation model+Alpaca vs imitation model+Alpaca+SPAG) to show the effectiveness of SPAG.

For the comparison with 20Q, we agree that the target sizes of the two games are different. However, the target sizes are entangled with the game designs, e.g. GuessMyCity can only use city names as targets. Although target sizes are different, we have made efforts to control other experiment factors. In Appendix Section E, we have used similar self-play sampling sizes (20K for imitation and 10K for self-play RL training) for non-adversarial games. From the perspective of supervision, we have used the same size of imitation data (20K GPT-4-played games) and the same SFT data (Alpaca). All other conversation data are sampled by the learning models themselves aligned with the concept of reinforcement learning. Therefore, the performance gaps between the adversarial taboo and non-adversarial games can support the efficiency of this adversarial game design.

We express many thanks for the reviewer's supportive comments. For the reviewer's concerns:

1. About the second model choice: we choose Baichuan-13B as the second base model mainly because the full self-play experiments (imitation learning, SPAG-1, SPAG-2, SPAG-3) of one base model are already heavy enough especially with limited computational resources. So we give this opportunity to a totally different open-source LLM to LLaMA-2-7B, with different model sizes, major languages, and even company regions. Baichuan-2 is pretrained with more focus on Chinese language scenarios by a Chinese startup. In this setup, we can test the universality of the proposed SPAG method with totally different LLMs. Fortunately, SPAG has shown LLM reasoning improvements on both model bases.

2. About N>2 models: I think it can be imaginative and potential to have more than two LLMs engaged in the language games. These LLMs can play the Adversarial Taboo or other multi-player language games. I believe with elaborate game designs, the LLMs' capacity can be further enhanced via self-play, just like our human activities in society. It can be a very interesting direction for future work.

3. About the performance of SPAG transferred to other LLMs: as mentioned above, we chose two different LLMs to test the proposed SPAG method and observed satisfied reasoning improvements, which has supported the universality of our method. However, it remains unclear whether the SPAG method has its upper bound since the open-source LLMs' reasoning abilities are continuously getting better. But we can still design more challenging self-play games and conduct SPAG for LLMs to practice and reinforce. For example, we can let two LLMs implement game AIs to play the AI Game Programming Competitions, which could be a more hard-core SPAG to enhance LLMs' coding and reasoning.