Competing Large Language Models in Multi-Agent Gaming Environments

We appreciate your efforts in reviewing and your recognition of our paper's rigorous exploration of factors influencing LLM behavior. We will address your concerns one by one.

The paper does not sufficiently clarify which specific cognitive or decision-making abilities of LLMs are being measured through this benchmark. While the Nash equilibrium is a logical choice for evaluating decision rationality, the paper does not articulate whether the benchmark truly reflects the LLMs’ capacities in areas such as long-term planning, adaptability, or ethical decision-making. The discussion of which kind of ability can be measured through this benchmark.

We appreciate the feedback. All the eight games require some common abilities, such as arithmetic ability, the ability to understand environments, game rules, and historical results, the ability to understand others’ intentions (ToM ability), and strategic reasoning (critical thinking).

There are some distinct abilities for each game:

• K-level reasoning: Guess 2/3 of the Average;
• Mixed strategy adoption: El Farol Bar;
• Risk management: Divide the Dollar; Sealed-Bid Auction;
• Long-term planning: Battle Royale; Pirate Game;
• Ethical reasoning and Altruism: Public Goods Game; Diner’s Dilemma;

We have added the discussion in the introduction at Line 107, Page 1.

The paper misses an opportunity to delve into the reasons behind the LLMs’ decisions. Different models may exhibit distinct decision-making tendencies based on their training data and architectures. For instance, GPT-4’s tendency toward cooperative behavior, as seen in the Public Goods Game, may stem from its RLHF process, which favors kindness and prosocial actions. Understanding these motivations could offer valuable insights into model alignment and performance tuning. One section of analyzing the reasoning behind the decision is the need for a better explanation of the various performances of LLMs. Could you explain why some LLMs, which are typically considered more advanced, do not perform well on this benchmark? For example, models like GPT-4 generally excel in many NLP tasks but underperform in certain games. Could this be due to the influence of ethical considerations embedded in their training (such as RLHF) that conflict with the purely rational strategies required to achieve Nash equilibrium?

Thank you for the good suggestion. We agree that training data or different RLHF processes can affect models’ tendency. However, the base models (before instruction tuning) do not have conversation ability, thus are unable to conduct GAMA-Bench evaluation. Therefore, to bypass the value alignment on LLMs, we use the jailbreak technique, specifically, the CipherChat [1]. Previous work showed that this jailbreak method can make GPT-4 show more negative traits [2]. To evaluate whether LLMs will have behavioral changes after value alignment, we evaluate GPT-4o before and after CipherChat using betraying games, Public Goods Game and Diner’s Dilemma.

Before CipherChat, GPT-4o achieves 90.91±2.72 and 10.7±8.3 on Public Goods Game and Diner’s Dilemma. After CipherChat jailbreaking, it achieves 88.55 ± 2.38 and 7.5±4.1 on the two games. We can observe a performance drop. This is because GPT-4o is tuned to be risk-averse. In the Public Goods Game, they think “if no contribution is made to the public pot, then I will have no loss.” In Diner’s Dilemma, they think “choosing the cheaper dish will cost me less money.” The same conservative strategy appears in the El Farol Bar Game, where GPT-4 performs badly. It tends to stay at home instead of risking going to the bar. After jailbreak, it can use more risky strategies of contributing less in the Public Goods Game and selecting the expensive dish in the Diner’s Dilemma.

The experiment and discussion is added in the Appendix in Page 37.

[1] GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher, In ICLR, 2024, Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, Zhaopeng Tu.

[2] On the Humanity of Conversational AI: Evaluating the Psychological Portrayal of LLMs, In ICLR, 2024, Jen-tse Huang, Wenxuan Wang, Eric John Li, Man Ho Lam, Shujie Ren, Youliang Yuan, Wenxiang Jiao, Zhaopeng Tu, Michael R. Lyu.

We appreciate your efforts in reviewing and your recognition of our extensive experiments and our clear presentation. We will address your concerns one by one.

The author mentioned that most previous benchmarks incorporates classic settings that LLM has seen during training. Yet most games proposed in GAMA-Bench seem to also be taken from classical work. For instance, I prompted the game definition of all 3 cooperative games to GPT-4o and it is able to recognize which famous game it is. The authors could improve on this by proposing variants and provide concrete evidence on no test set leakage by comparing LLM performance on variants vs. original games.

We appreciate your effort in trying GPT-4o to recognize the games. We acknowledge that, with the vast training data, state-of-the-art LLMs know well about the definition, and even the Nash equilibria of classical games. However, we would like to highlight that: (1) there is a distance from theory to practice. Knowing the theory does not necessarily imply doing well in practice. (2) Models fail to perform well in generalized settings, even in recognizing. For example, when prompting the guessing game with the ratio (2/3) changed to 2, GPT-4o answers:

User: What's the Nash equilibrium of "2 guessing game" such that the ratio is 2 instead of 2/3?

GPT-4o: The Nash equilibrium for this game is: $x_i = 0 \forall i$.

The conversation is accessible through this anonymous link: https://chatgpt.com/share/673c5be2-20bc-800a-b6f2-125170fba777. However, the Nash equilibrium of this setting is the maximum number, which is 100.

Language model are not designed to solve games, nor provide concrete mathematical calculation which is required to compute NEs. If the goal of this work is to understand LLM's ability to adapt in dynamic, multi-agent systems, I would encourage the authors to design additional experimental settings where partial actions by other agents are revealed. For instance, it would be more informative to see how models will adjust in "Guess 2/3 of the Average" game knowing there is a weak/malicious model that is likely to output a large number, compared to simply letting the model work its way to the NE. An example could be a series of rounds where the LLM is given information about other players' previous moves or tendencies, and then measure how quickly it adapts its strategy.

Thank you for the good suggestion. We acknowledge that current LLMs are weak in math calculations. Additionally, our framework provides LLMs with the chance to learn from historical data and adapt themselves in the environments. We have investigated LLMs’ performance when playing against some fixed strategies in Page 36: (Appendix G) LLM vs. Specific Strategies. First, we let one player consistently bid an amount of 91 golds in the Divide the Dollar game, compelling all other participants to bid a single gold. Additionally, we examine agents' reactions to a persistent free-rider who contributes nothing in the Public Goods Game. We find that agents lower their bids in the Divide the Dollar game in response to a dominant strategy. Contrary to expectations, in the Public Goods Game, agents increase their contributions, compensating for the shortfall caused by the free-rider. Conclusion: LLMs can adapt their strategies dynamically to the environment.

However, it is an implicit setting for experiments described above. That is, we do not tell the players about others’ fixed strategies. Therefore, we further design experiments where we explicitly tell the player about others’ fixed strategies using the Guess 2/3 of the Average game.

• In setting (a), a player is told that others are smart and will always choose 0 the Nash equilibrium.
• In setting (b), a player is told that others are smart, but is not told that others will always choose the Nash equilibrium.
• In setting (c), a player is told that others are stupid and will always choose random numbers. The experiment uses GPT-4o.

Results show that:

• (a) GPT-4o selects 0 in the first round, and keeps selecting 0 for the remaining rounds.
• (b) GPT-4o does not select 0 in the first round, but converges to 0 quickly in a few rounds.
• (c) GPT-4o’s selection is quite random, suggesting that it cannot infer the optimal decision of 50 * 2/3 = 33 since the average of others’ random selections is 50.

The experiment is added in Appendix G in Page 36.

We appreciate your efforts in reviewing our work and your recognition of our unique approach to generalizability. We will address your concerns one by one.

The case could be made stronger for "LLMs being good at Game Theory" == "LLMs useful for decision making". The intro "jumps" from "decision-making poses a significant challenge for intelligent agents" to Game Theory. It's slightly strengthened by noting that your "framework evaluates LLMs’ ability to make optimal decisions". In the conclusion you mention "Our findings reveal that GPT-3.5 (0125) demonstrates a limited decision-making ability on γ-Bench", which is great. Overall, I think your paper points in the direction and does well, but this introduction is a bit jarring.

Thank you for this insightful suggestion. We add two sentences to explain why evaluating Nash equilibria in game theory models can reflect individuals’ decision-making ability in the introduction at Line 43, Page 1:

“Many real-world scenarios can be modeled using Game Theory (Koller & Pfeffer, 1997). Furthermore, individuals’ ability to achieve Nash equilibria reflects their capacity in decision-making (Risse, 2000).”

Here is a brief introduction to the two supporting references:

Daphne Koller and Avi Pfeffer. Representations and solutions for game-theoretic problems. Artificial intelligence, 94(1-2):167–215, 1997. This paper delved into modeling complex real-world scenarios like arms control through simplified game-theoretic forms.

Mathias Risse. What is rational about nash equilibria? Synthese, 124:361–384, 2000. This paper explored rationality in Nash equilibria and the reflective processes preceding strategic decisions.

"We leveraging GPT-4 to rewrite our default prompt" -> 'We leverage'

Solved at Line 409, Page 8.

"The game has a PSNE where all players", "a PSNE but presents an MSNE, " -> PSNE and MSNE is defined in the appendix but acronyms are used in the main body.

Solved at Line 146 and Line 155, Page 3.

"The model produces average numbers of 34.59, 34.59" -> intentionally the same numbers?

Yes, we intentionally put the same numbers here. The model obtains the same average score using $R=\frac{1}{2}$ and $R=\frac{2}{3}$. Please also see Table 8 at Page 34.

"We first focus on open-source models, including OpenAI’s GPT-3.5 (0613, 1106, and 0125), GPT-4 (0125), and Google’s Gemini Pro (1.0, 1.5)." -> closed source

Solved at Line 477, Page 9.

RW: "Other than papers listed in Table 3 on evaluating..." -> this should present the table better. "We present a comparison of studies using LLMS..." It currently feels like a passing note, but is stronger than that. Also, link this in your statement contributions in the introduction.

We appreciate the feedback. We have improved our presentation in the paper, particularly in the sections of related work and introduction.

Solved in the related work at Line 499, Page 10:

“Evaluating LLMs through game theory models has become a popular research direction. An overview on recent studies is summarized in Table 3. We find: (1) Many studies examine the PSNE on two-player, single-round settings, focusing on the Prisoner's Dilemma and the Ultimatum Game. (2) Varying temperatures are employed without discussing the impact on LLMs' performance.”

Solved in the introduction at Line 122, Page 3:

“We provide a comprehensive review and comparison of existing literature on evaluating LLMs using game theory scenarios, as summarized in Table 3. The review includes key aspects such as models, games, temperature settings, and other game parameters}, highlighting our emphasis on the multi-player setting and the generalizability of LLMs.”

RQ4: Qwen-2 seems to do great in all but Diner and SBA which seem very out of place and there's no mention of this result or why it could've happened. What happened here?

Thank you for pointing out this phenomenon. Many LLMs show this similar behavior in Betraying games, including Qwen-2 and GPT-4-0125. They all achieve relatively good performance on Public Goods Game, but perform badly on Diner’s Dilemma and SBA. We believe that these models are tuned to be risk-averse. In the Public Goods Game, they think “if no contribution is made to the public pot, then I will have no loss.” In Diner’s Dilemma, they think “choosing the cheaper dish will cost me less money.” In SBA, they think “the utility will not be negative as long as the bid is less than the valuation,” thus not taking the risk to give a much lower or higher bid. The same conservative strategy appears in the El Farol Bar Game, where GPT-4 and Qwen-2 perform badly. They tend to stay at home instead of risking going to the bar.

The explanation is added to Line 482, Page 9.

Thank you for the follow-up question. This is a very interesting one.

At the beginning, we also thought that the CipherChat [1] method will make the model less prosocial, make it selfish, and increase the scores on betraying games. However, we observe that GPT-4o's scores on the Public Goods Game and the Diner's Dilemma decrease instead (90.91±2.72 to 88.55±2.38 and 10.7±8.3 to 7.5±4.1). We believe that it is because OpenAI improves the value aliment on GPT-4o, defending against CipherChat. Following [2], we conduct the test on model's negative traits using Dark Triad Dirty Dozen. Results of GPT-4o before and after jailbreak are listed below:

We find that instead of the results of GPT-4 increasing its scores after jailbreak reported in [2], we find that GPT-4o decreases in these negative traits instead. Therefore, it may not show negative characteristics like selfishness.

Therefore, we believe that jailbreak currently affects more on the model's risk-management behavior. GPT-4o is tuned to be risk-averse. After our jailbreak, it tends to take riskier strategies. In the Public Goods Game, it tries to contribute more money to the public pot, which is opposite to the optimal decision, thus decreasing the score. It is the same for the Diner's Dilemma. The jailbroken GPT-4o can take riskier strategies, choosing the costly dish and spending more money.

New discussion has been added in Appendix H, Page 37.

[1] GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher, In ICLR, 2024, Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, Zhaopeng Tu.

[2] On the Humanity of Conversational AI: Evaluating the Psychological Portrayal of LLMs, In ICLR, 2024, Jen-tse Huang, Wenxuan Wang, Eric John Li, Man Ho Lam, Shujie Ren, Youliang Yuan, Wenxiang Jiao, Zhaopeng Tu, Michael R. Lyu.

We appreciate your effort in reviewing and your recognition of the systematic experiments we conducted. We will address your concerns one by one.

My main concern is that, from a conceptual/technical point of view, the contribution and novelty of this work are somewhat limited. The authors discussed related literature limited to two players or actions and motivated their focus on the multi-player setting. However, the study of multiple LLM agents in gaming is not novel ([1]). I think a more detailed and thorough comparison with previous related benchmarks is needed. [1] Sahar Abdelnabi, Amr Gomaa, Sarath Sivaprasad, Lea Schönherr, and Mario Fritz. LLM-deliberation: Evaluating LLMs with interactive multi-agent negotiation games, 2023.

We deeply appreciate the insightful feedback. Studying LLMs through game-theoretic models is an active area of research. We acknowledge that while most existing papers focus on two-player or two-action games, there are still papers focusing on multi-player, multi-action settings. However, when it comes to classical game theory scenarios such as the Prisoners’ Dilemma, previous studies only focused on the two-player version, while in our GAMA-Bench, we use the multi-player version, the Diner’s Dilemma.

Additionally, we would like to highlight that models fail to perform well in generalized settings, even in recognizing. For example, when prompting the guessing game with the ratio (2/3) changed to 2, GPT-4o answers:

User: What's the Nash equilibrium of "2 guessing game" such that the ratio is 2 instead of 2/3?

GPT-4o: The Nash equilibrium for this game is: $x_i = 0 \forall i$.

The conversation is accessible through this anonymous link: https://chatgpt.com/share/673c5be2-20bc-800a-b6f2-125170fba777. However, the Nash equilibrium of this setting is the maximum number, which is 100. These variants of the classical game theory models are not systematically studied in previous literature.

For Abdelnabi et al., 2023 [1], they design negotiation games involving six parties with distinct roles and objectives to evaluate LLMs’ ability to reach agreement. The paper is discussed in the related work at Line 526, Page 10.

The single metric that measures how far the LLM agent's behavior is from the Nash equilibrium cannot fully evaluate the performance of LLM agents in decision-making.

Thank you for pointing out this issue. We acknowledge that decision-making is a broad concept including various aspects. However, we believe evaluating Nash equilibria in game theory models can reflect individuals’ ability in decision-making (especially making the optimal decisions) because: (1) Many real-world scenarios can be modeled using Game Theory (Koller & Pfeffer, 1997), and (2) Individuals’ ability to achieve Nash equilibria reflects their capacity in decision-making (Risse, 2000).

Here is a brief summary on the two supporting references:

Daphne Koller and Avi Pfeffer. Representations and solutions for game-theoretic problems. Artificial intelligence, 94(1-2):167–215, 1997. This paper delved into modeling complex real-world scenarios like arms control through simplified game-theoretic forms.

Mathias Risse. What is rational about nash equilibria? Synthese, 124:361–384, 2000. This paper explored rationality in Nash equilibria and the reflective processes preceding strategic decisions.

We have improved the writing of our motivation in the introduction at Line 43, Page 1.

Some of the insights are not very surprising in light of the prior literature (such as it is possible to improve the performance of GPT3.5 through CoT).

This is a good point. We acknowledge that CoT has been proven effective under multiple downstream tasks. However, CoT is not always effective, as found by [1]. Therefore, verifying whether CoT can also work in multi-agent gaming environments becomes important. Our findings suggest that CoT in the scenario works as expected, providing a deeper understanding about LLMs and their behaviors.

[1] To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning. In arXiv 2409.12183. Zayne Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, Greg Durrett.