Efficacy of Language Model Self-Play in Non-Zero-Sum Games

Thanks for your review! We would like to make a couple of clarifications:

Self-play has been known to be effective in non-language-based cooperative settings like MPE [1], SMAC [2], GRF [3], etc., so it is expected to work in language-based cooperative settings

We respectfully disagree with this claim. In particular, past work (including the work cited here) has shown that self-play in cooperative settings can lead to models which collaborate well with themselves, but not to models which collaborate well with humans. See [4] for further discussion on this point

Instead of just presenting these empirical findings, a more important question is to understand why language model self-play works in some tasks but fails in others, and how to make it work in more general tasks.

For sure, this would be a great contribution. Our best guess is that self-play works in this setting because LLMs can be finetuned with relatively few samples, avoiding the overfitting described in [4]; however, you are correct that it would require experiments on additional domains to justify this hypothesis. Because language model self-play is a computationally expensive method, however, this would be a relatively difficult set of experiments to run. We view our work as more of a “proof of existence” that vanilla self-play can be used to train LLMs which can better collaborate with humans, which has not been shown before.

This is more like a case study and it would require evidence on more tasks with more LMs and training for more rounds to support the authors' claim that "self-play leads to large improvements in both the cooperative and semi-competitive settings" (L047-048).

We don’t believe this is an overclaim – we aren’t claiming that self-play leads to large improvements in all cooperative and semi-competitive settings, only in the ones we studied.

More models, more training, more tasks

We totally agree that all of these directions are valuable. Unfortunately, as discussed in the response to R2, the total cost for all computational experiments in this paper (excluding human data collection) is about $6000, making it intractable to run large-scale experiments on additional models and tasks.

If the performance can decrease with more training rounds, is it possible that, with more self-play rounds, the model will still diverge to uninterpretable language?

This is totally possible. As with many training methods, it’s possible to overfit, and the assumption is that you should conduct early stopping before this divergence occurs.

[4] Collaborating with Humans without Human Data. https://arxiv.org/abs/2110.08176

Thanks for the positive review! In response to your concerns:

What could be the underlying reasons for the observed performance decline in cooperative games involving humans after 8 to 10 rounds? Why is this trend not present in other game settings?

We found that our model began to overfit after the sixth iteration of cooperative self-play. In particular, in certain settings, the model would exhibit an “inverse proposal” strategy where it described one set of objects and then formally proposed the complementary set. As a result, the model’s scores increased in self-play, but the human-LM scores declined. In the semi-competitive setting, overfitting doesn’t occur until a later iteration.

What alternative environments/games could be explored to strengthen the study's findings?

One challenge here is finding games which can be naturally converted into cooperative and competitive versions. Possible options include the fruit trading game from [1], or CaSiNo in [2], although both are structurally very similar to the Deal or No Deal domain. However, we did not run experiments on these additional domains due to cost.

Would expanding the study to include more competitive and cooperative levels provide a more comprehensive understanding of the impact of cooperation on self-play efficacy?

This is definitely possible! We did not explore this direction due to cost and resource constraints, and because the cooperative and semi-competitive results were relatively similar. However, as noted in Section 2, these experiments can straightforwardly be run by changing the value of \lambda.

[1] States as Strings as Strategies: Steering Language Models with Game-Theoretic Solvers. https://arxiv.org/abs/2402.01704v2 [2] CaSiNo: A Corpus of Campsite Negotiation Dialogues for Automatic Negotiation Systems. https://arxiv.org/abs/2103.15721

Thank you for your positive review! To answer your questions:

Can you further elaborate why the performance in iteration 3 is worse in Table 1? Do you have any examples showcasing the overfitting?

It’s hard to give definitive answers, but we can provide a best guess based on our qualitative evaluation of the dialogues. The main reason seems to be that models trained on GPT-4 outputs are overly agreeable, and after a few rounds of self-play training these models become pushovers. For example, in this dialogue, the model accepted a deal worth zero points:

User: [message] I need the hat and ball and at least one book
Model: [message] I value the books at 0, so I'm happy to let you have one or all of them. Since you need the hat and the ball, would it be agreeable if I take the remaining books? How many points are the hat and the ball worth to you? This information can help us make a fair division.
User: [message] deal, you get two books and I get two books, the hat and the ball
Model: [message] That works for me. I'll take two books, and you'll take two books, the hat, and the ball. Shall we proceed with finalizing this proposal? Or is there anything else you'd like to adjust or discuss?
User: [propose] Proposal made. You must now respond with a proposal of your own.
Model: [propose] (2 books, 0 hats, 0 balls)

What do you think is the reason why open source LLMs cannot improve their performance with your method?

In short: language model self-play only works when models are above a certain baseline level of performance. At the time of experimentation, most open source models lagged significantly behind closed source models, especially on agent tasks [1]. During the first round of self-play, we found that open source models obtained scores of 0 in almost every game, resulting in a lack of data for finetuning. This result parallels other research in self-improving LLMs, which has also found that strong base models are necessary to see gains from self-improvement [2].

How long does it take to finetune one round/iteration?

About one hour, using GPT-3.5 via the OpenAI API.

In response to your other concerns:

The study exclusively focuses on language models and does not incorporate any reinforcement learning (RL) baselines for comparison. Including RL-based models could have provided a broader benchmark, highlighting the unique contributions of self-play for LLMs while also revealing potential strengths or weaknesses relative to established RL techniques.

Could you please elaborate further on what type of RL baselines you have in mind here? Ultimately, the task we consider is a grounded dialogue game, so we do need a model that is capable of generating language. While that language model could be trained with more sophisticated RL techniques (e.g., DPO), we chose to focus on filtered behavior cloning as we wanted the simplest possible experimental setup, in order to avoid possible confounds.

The self-play data generation and subsequent finetuning require substantial computational resources, resulting in high training costs.

For sure, this is absolutely true. The approximate cost of running experiments for this paper (including exploratory experiments, but not counting human evaluation) was about $6000. This is the main reason we did not experiment with a wider range of domains or models. However, as time goes on, we anticipate that the cost of language model finetuning and inference will decline, making self-play and related approaches more feasible. Additionally, high training costs may sometimes be permissible in industry, e.g., if a model only needs to be trained once but will be deployed to millions of users.

[1] AgentBench: Evaluating LLMs as Agents. https://arxiv.org/abs/2308.03688 [2] Autonomous Evaluation and Refinement of Digital Agents. https://arxiv.org/abs/2404.06474

Thanks a lot for your review! It is really thorough and has a lot of actionable feedback. We have incorporated your suggested references, as well as your comments about game dynamics, into the revised paper. To address a couple of your concerns:

This is a bit underwhelming and entirely hinges on the fact that your deal or no deal game is difficult to play using prompts alone. I am not convinced of this. I am especially not convinced that a better prompt could solve the cooperative game with GPT-3.5.

We’d like to clarify that DoND is actually somewhat hard for LLMs to play. To make this point, we ran additional experiments with our base models on 10 different cooperative and semi-competitive prompts (20 prompts total), devised by the authors. For each prompt, we ran self-play for 100 games; across all prompts, the average self-play score was below 1.2. We added a brief description of this experiment to Appendix B.

We think the reasons for this are twofold: (1) language models like GPT-3.5 struggle with agent tasks, especially without few-shot examples. This is reflected in other agent benchmarks, e.g., [1]; and (2) the fact that proposals are private in DoND means that it is difficult for models to accidentally “stumble into” good behavior. Models need to exhibit good understanding of the task, basic dialogue and math reasoning skills, and the ability to submit formal proposals correctly, to have a reasonable chance at obtaining a nonzero score.

There should be no communication in a fully competitive game

We generally agree with this point, and we’ve updated our paper in a couple places to reflect it. You are correct that between rational agents, a 2p0s game does not involve any communication (this is partially why the zero-sum results are currently shoehorned into the appendix).

However, the Crawford & Sobel (1982) result does not mean that it is impossible to train models which could in principle use communication to achieve higher scores in competition with humans. Consider, for example, a heads up (i.e., two player) poker game: it’s common for players to speak, and especially if one player is a pro and one player is an amateur, some information can legitimately be gained from this conversation, even though it only tends to benefit the stronger player.

Given that (1) we’re not training models to convergence, (2) these models have a language model prior, and (3) we’re ultimately interested in the performance of models in collaboration or competition with humans, we think it makes sense to keep the results for the zero-sum condition, provided with a disclaimer that communication isn’t useful between fully rational agents.

We also want to provide a couple of clarifications:

The competitiveness of DoND depends on the specific items and preferences each player has over the items. A game with λ=1 can be highly competitive if both players have similar values for the same items, highly cooperative if they value different items, or a mix of the two.

Perhaps there is a slight mistake here? With λ = 1, both players’ rewards are identical (R_1 = X + 1Y = 1X + Y = R_2), therefore the game is fully cooperative. We do agree that this statement holds true when λ = 0, however, and added average Pareto-optimal scores for the semi-competitive settings as you requested

As well, the that your game is "cooperative, competitive, or anything in between" in your abstract is not necessarily true.

We believe this is shown in the “game objectives” paragraph of Section 2. The game is fully cooperative when λ = 1 (because the rewards are the same), fully competitive when λ = -1 (because the rewards sum to zero), and can be smoothly interpolated in between these settings by changing the value of λ. Please let us know if you have further followup questions here!

[1] AgentBench: Evaluating LLMs as Agents. https://arxiv.org/abs/2308.03688

To answer some questions:

why do errors occur in 22% of cooperative games but only 1% of semi-competitive? (section 6.1)

We don’t have a clear intuition for why there’s a discrepancy here, but almost all of these errors are of the same type. In the cooperative setting, we found that models often output both a message and a proposal at the same time; when this happened, the environment would throw an error message, giving the model a chance to correct itself. However, in 22% of cases, the baseline model failed to correct itself, resulting in the game aborting. (Note: the reported %s are abort rates; the error rates are higher for both conditions, cf. Figure 10.) Interestingly, we found that this error persisted even across a range of prompts for the cooperative setting.

how can you be sure that human-AI performance is not bolstered by humans compensating for your AI playing badly?

This is totally possible, and in fact probably true: we suspect that this explains the high initial score for the baseline cooperative model w/ humans (cf. last paragraph of Section 5.1). However, since the performance of the human+LM team still improves over the course of self-play training, we don’t believe that this contradicts our main claims.

Why do you include lying about an item preference value in your errors for 6.2? Lewis et al argued that this lying could be advantageous / effective deception.

We include this because (1) it is fundamentally difficult to distinguish whether a model is lying strategically or simply hallucinating, and (2) we found during manual annotation that lying about item preference values almost always leads to failed proposals (i.e., scores of zero). In other words, we do not observe cases of GPT-3.5 (or its finetuned variants) using lying productively.

A fundamental assumption of this paper is that the DoND game cannot be played by modern language models. Through a small amount of effort, I think I have put this assumption in doubt and I am very unsure about the soundness of this paper. The authors really need to address this point.

Here is an anonymized jupyter notebook where I create a DoND setup and successfully get cooperative DoND negotiation using GPT-4o mini https://nbviewer.org/urls/file.io/tAjuZj7zVzxo EDIT: see link below

The differences from your setup are

1. I used a slightly modified prompt to indicate the game is cooperative and added a tiny amount of chain of thought to improve getting the final proposal in the right format. I also explain what "score" is better than the authors' original prompt
2. I use GPT-4o mini instead of GPT-3.5 turbo

Overall, GPT-4o mini achieves ~60% optimality with minimal prompt tuning in around 2 hours of total work. This is not better than the authors' final self-play result but the story is no longer sound.

These results imply that either the authors assumed a bad prompt (I did a very small amount of prompt tuning) or GPT 4o-mini solves all issues from GPT-3.5. GPT4o-mini was released July 18, well before the submission deadline, and I'm not really sure what is the point of this research direction if a newer models + better prompt does so well. You can make the point that self-play training is easier than prompt engineering but you don't make this point, you claimed it was essentially impossible to play this game with prompting. There is still a point to be made and I still like the research direction but I think the point being made needs to be clear and sound.

Remaining Concerns

• The story is still a jumble.

• Thank you for the average scores. Can the authors also add the maximum scores for the semi-competitive setting. A line showing the average and top baselines on Figure 1 would be useful.

• The references to Cao et al and Noukhovitch et al feel minimal if the point you're trying to make is about successful negotiation in a competitive game

• The game is specifically not "anything in between [cooperative and competitive]". The reason for explaining the sampling of values is to specify that it is specifically some level of competitiveness and that you do not provide a general method to get any level of competitiveness (as Noukhovitch et al do).

Overall, I do not feel my concerns are addressed and perhaps I have bigger concerns now that I see GPT 4o-mini can play the cooperative DoND game.