Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4

Thank you very much for time and effort to providing us with more references and we will discuss with them in the final version to improve our paper.

First, we agree with you that randomness plays a significant role for imperfect information game evaluation. To improve the randomness of our experiments, we already designed two types of experiments to compare Suspicion Agent with baselines:

1. Reducing the randomness of Random Seeds:

   Our agent plays against different baselines for 100 games utilizing varying random seeds for each game. This tactic is intended to dampen the stochastic variability introduced by the random seed settings.

2. Reducing the randomness of Card Strength/Position: (which is also used in the recommended AIVAT [1])

   Given the limited games, the distributions of strong hand and weak hands are not equal for each player in limited game rounds. To ensure a fair comparison, it was crucial that both the Suspicion-Agent and the opponent model experienced the same card strength across an equal number of games. To achieve this balance, we employed a strategic approach: initially, the Suspicion-Agent was placed in position 0 for the first set of 50 games. Subsequently, we reproduce these 50 games with same cards as the first one but switched the positions, placing the Suspicion-Agent in position 1 and the baseline model in position 0. This method allowed us to maintain an equal card strength for both the Suspicion-Agent and the baseline model over a total of 100 games, thereby enabling a more accurate evaluation of each model's performance.

Given these two designs, we believe that we can alleviate the effect of randomness in imperfect information games. Given the results in Table 1 and Table2 in the main paper: Suspicion Agent achieves superior performance over learning-based baselines except CFR+ in all 100 random games, 50 games in position 0 and 50 games in position 1. They are the clear evidence to show the advantages of our suspicion agent.

Lastly, we would like to highlight that our study is not only the first to enable LLM for imperfect information games but also the first to consider the interaction between two agents by introducing the theory of mind using LLM. We believe our work sets a valuable baseline and will serve as an inspiration for future research in this area. We respectfully hope that you reconsider your decision in light of these contributions and the innovative aspects of our study.

[1] Burch, Neil, et al. "Aivat: A new variance reduction technique for agent evaluation in imperfect information games." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.

First of all, I appreciate the author's response. Having reviewed both the response and the updated paper, the author effectively highlights the gap between the Suspicion-Agent and Nash equilibrium, partially addressing a concern of mine. Consequently, I have adjusted my rating to 5.

However, I cannot assign a higher score as I believe the current evaluation results remain less reliable. In imperfect information games, randomness plays a significant role. Achieving statistically significant results necessitates a substantial number of evaluations. With only 100 games conducted thus far, the randomness is excessively high, making it challenging to derive statistically meaningful conclusions.

While I acknowledge the considerable costs associated with the Suspicion-Agent due to the GPT API expenses, it is imperative to conduct more evaluations for more reliable results. I suggest the author consider incorporating some variance reduction techniques [1,2] to decrease the number of evaluations required. I encourage the author to further optimize the experimental section to obtain more statistically significant results for the next submission. I believe it will be a strong submission for the next conference.

[1] White, Martha, and Michael H. Bowling. "Learning a Value Analysis Tool for Agent Evaluation." IJCAI. 2009.

[2] Burch, Neil, et al. "Aivat: A new variance reduction technique for agent evaluation in imperfect information games." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 32. No. 1. 2018.

Thank you for your time and effort in sharing critical feedback regarding our work. We provide the following response to address your concerns about results limitations, and thus respectfully hope you can consider the response to the final decision.

1. the evaluation of the method is conceptually flawed.

A: We acknowledge the concerns you raised regarding the experiments in our paper, and the omission of Nash equilibrium, a pivotal concept in evaluating algorithms in game settings. To address this, we added the relevant discussion in our paper, added the concept to the main paper and conducted additional experiments, specifically focusing on the CFR+ algorithm. Given the codebase of RLCard (https://github.com/datamllab/rlcard), we further train the CFR+ algorithm for 2000000 iterations and play our suspicion agent with CFR+, getting the following results:

which indicates the current LLM (without training examples and fine-tuning) cannot beat algorithms that reach Nash equilibrium when the iterations are large enough. Therefore, we have moderated our claims in the paper. Our revised findings demonstrate that our LLM may not surpass Nash Equilibrium strategies. As a preliminary experimental study, Our focus is not on achieving Nash Equilibrium directly with the LLM but on exploring the design of an effective prompting system that can unlock the potential of LLMs in these complex game environments. Our results highlight that Suspicion-Agent can adaptively alter its behavior patterns when facing different opponents. This ability to effectively compete against simpler learning-based methods, which utilize reinforcement learning, is a noteworthy contribution to the field. Our research provides a foundation for further exploration into how LLMs can be utilized and improved for strategic decision-making in imperfect information games.

In addition to the new results, as the first work to design the prompting system to enable LLM for imperfect information games, we still believe our proposed methods and the preliminary public experimental study and data are valuable for the community.

1. We experimentally demonstrate that even with pre-trained data, LLM itself cannot play various imperfect information games well as Table 3 shows ((-72 no ToM vs +24 second-order ToM)). Then we introduce the first prompting system to enable large language models like GPT-4 to play various imperfect information games using only the game rules and observations without any extra training, this may inspire more LLM-based subsequent works.
2. Leveraging the theory-of-mind capabilities of large language models, our work is the first to demonstrate that a simple prompting system can enable GPT-4 to outperform some learning-based algorithms like DMC and NFSP, even without specialized training or examples. However, the further experiments prove that it still cannot beat algorithms that can achieve Nash-Equilibrium such as CFR+. In addition, we also qualitatively evaluate the potential of LLM into the different imperfect information games, which may inspire more subsequent works.
3. We comprehensively identify and discuss the current limitations of employing large language models in imperfect information games, contributing valuable insights for the further AI-Agent research.
4. We are preparing to make all our code and interactive data publicly available. We hope that this will advance the community's understanding of the capabilities of large language models, particularly GPT-4. We also hope this will catalyze the development of more efficient models in the field of imperfect information games.

Thank you for your time and effort in sharing critical feedback regarding our work. We provide the following response to address your concerns about results limitations, and thus respectfully hope you can consider the response to the final decision.

1. The game on Leduc Hold’em is too specific. It is possible that the winning strategy of playing Leduc Hold’em was discussed online and then used to train LLMs.

A: We agree that the pre-trained knowledge is helpful for LLM. However, the pre-training itself cannot enable LLM to perform imperfect information well. As Table 3 shows, the simple prompting without ToM prompts is not able to achieve promising results, (-72 no ToM vs +24 second-order ToM). This is a strong evidence to show the effectiveness of our prompting system and the potential of LLM in imperfect information games.

2. the evaluation of the method is conceptually flawed.

A: We acknowledge the concerns you raised regarding the experiments in our paper, and the omission of Nash equilibrium, a pivotal concept in evaluating algorithms in game settings. To address this, we added the relevant discussion in our paper, added the concept to the main paper and conducted additional experiments, specifically focusing on the CFR+ algorithm. Given the codebase of RLCard (https://github.com/datamllab/rlcard), we further train the CFR+ algorithm for 2000000 iterations and play our suspicion agent with CFR+, getting the following results:

which indicates the current LLM (without training examples and fine-tuning) cannot beat algorithms that reach Nash equilibrium when the iterations are large enough. Therefore, we have moderated our claims in the paper. Our revised findings demonstrate that our LLM may not surpass Nash Equilibrium strategies. As a preliminary experimental study, Our focus is not on achieving Nash Equilibrium directly with the LLM but on exploring the design of an effective prompting system that can unlock the potential of LLMs in these complex game environments. Our results highlight that Suspicion-Agent can adaptively alter its behavior patterns when facing different opponents. This ability to effectively compete against simpler learning-based methods, which utilize reinforcement learning, is a noteworthy contribution to the field. Our research provides a foundation for further exploration into how LLMs can be utilized and improved for strategic decision-making in imperfect information games.

3. The current experiment setting is not fair because LLMs used the online information when playing the game, but CFR cannot use this information. That is, LLMs exploit more information while playing the game.

A: As described before, the pre-training itself cannot enable LLM to perform imperfect information well. By contrast, the key is how to prompt LLM to leverage the prior knowledge As Table 3 shows, the simple prompting without ToM prompts is not able to achieve promising results, (-72 no ToM vs +24 second-order ToM). This is a strong evidence to show the effectiveness of our prompting system and the potential of LLM in imperfect information games.

4. The comparison results among different algorithms for Game Coup and Texas Hold’em Limit should be presented.

A: Limited by the budgets, we just showcased it in the original paper. Following your suggestions, we evaluate Coup with 5 people, each people can play it in 5 games. The average of winning rate of our proposed method against human being is 72%, which also proved the effectiveness of the proposed method.

Thank you for your time and effort in sharing critical feedback regarding our work.

1. the rounds per game is not large and the context window may be able to handle several full games without compressing or discarding any history.

A: Good question. To address this, we have incorporated a summarization module in our system. This module is designed to concisely summarize the progress of each game, thereby efficiently managing the use of context space. This approach not only optimizes the use of the context window but also highlights the significance of the long context window capability in Language Learning Models (LLMs) for AI agents.

2. Most of the templates are given, so the GPT-4 is thinking in the given logic

A: Good point. Even without predefined templates, GPT-4 has the capability to generate reasonable analyses, showcasing its second-order Theory of Mind (ToM) abilities. However, one of the inherent challenges with LLMs like GPT-4 is their tendency towards producing hallucinations—outputs that might seem plausible but are factually inaccurate or inconsistent. To mitigate this issue, we introduced templates into our system. These templates serve as a guiding framework, helping to steer GPT-4's responses towards more accurate and relevant analyses by providing a structured approach to processing and responding to information. The effectiveness of this method is evident in the results, where the use of templates has significantly alleviated the problem of hallucinations, leading to more reliable and coherent outputs from the model.

3. Is there a study on the intermediate responses from GPT-3.5 and GPT-4, about correctness in calculations, on the distribution of opponent's hand, the probability of winning rate, expected rewards, etc.?

A: In response to your question, we have conducted a qualitative evaluation focusing on the correctness of calculations in the intermediate responses generated by GPT-3.5 and GPT-4. Our findings indicate that GPT-3.5 shows limited proficiency in reasoning and calculation. On the other hand, GPT-4 demonstrates a notable improvement in this area, achieving approximately 80% accuracy in calculations relevant to playing games. This improved performance in GPT-4 can be attributed to its more advanced algorithms and larger training dataset, which enhance its ability to process and analyze complex game situations that involve probabilistic and numerical reasoning.

4. If the above calculation values of GPT-3.5 is usually incorrect, then would checking and correcting them be effective in leveraging its behavior?

A: Good Point. While addressing the calculation inaccuracies of GPT-3.5 could potentially improve its performance, it's important to consider that the model's limitations extend beyond just calculation errors. GPT-3.5 also exhibits challenges with reasoning abilities, particularly in comprehending and applying complex game rules. This aspect significantly impacts its capability to effectively play imperfect information games

5. Is there a necessity in constructing or thinking with 3rd-order (or even higher) ToM, while playing the game?

A: This is an intriguing question. The necessity of constructing or utilizing third-order (or higher) Theory of Mind (ToM) in game-playing scenarios has been a topic of discussion in previous research [1,2]. These studies suggest that ToM beyond the second-order tends to become less effective. This reduced effectiveness is primarily attributed to the increased complexity involved in calculating third-order (or higher) ToM. This increase in complexity can lead to diminishing returns in terms of strategic advantage, especially in practical gaming scenarios. Players or AI agents attempting to anticipate an opponent's beliefs about another player's beliefs (which is the essence of third-order ToM) often face considerable challenges in accurately processing and utilizing this information effectively. Thus, while higher-order ToM may provide theoretical insights into more complex strategic thinking, its practical application in game-playing contexts is limited by these computational and cognitive constraints.

Thank you for your time and effort in sharing critical feedback regarding our work. We provide the following response to address your concerns about results limitations, and thus respectfully hope you can consider the response to the final decision.

1. the evaluation of the method is conceptually flawed.

A: We acknowledge the concerns you raised regarding the experiments in our paper, and the omission of Nash equilibrium, a pivotal concept in evaluating algorithms in game settings. To address this, we added the relevant discussion in our paper, added the concept to the main paper and conducted additional experiments, specifically focusing on the CFR+ algorithm. Given the codebase of RLCard (https://github.com/datamllab/rlcard), we further train the CFR+ algorithm for 2000000 iterations and play our suspicion agent with CFR+, getting the following results:

which indicates the current LLM (without training examples and fine-tuning) cannot beat algorithms that reach Nash equilibrium when the iterations are large enough. Therefore, we have moderated our claims in the paper. Our revised findings demonstrate that our LLM may not surpass Nash Equilibrium strategies. As a preliminary experimental study, Our focus is not on achieving Nash Equilibrium directly with the LLM but on exploring the design of an effective prompting system that can unlock the potential of LLMs in these complex game environments. Our results highlight that Suspicion-Agent can adaptively alter its behavior patterns when facing different opponents. This ability to effectively compete against simpler learning-based methods, which utilize reinforcement learning, is a noteworthy contribution to the field. Our research provides a foundation for further exploration into how LLMs can be utilized and improved for strategic decision-making in imperfect information games.

2. Do you consider only two players games?

Yes, as a preliminary proof-concept study, we only consider two players games to make the problem simplified, and we would like extend it into multi-agent (more than 2) in the subsequent works.

3. Why do you make a distinction between 0 and 1 position?

a. motivation of making a distinction between 0 and 1 position?

Due to budget constraints, we were unable to conduct large-scale experiments like previous methodologies which involved over 100,000 games. (However, the distributions of strong hand and weak hands are not equal for each player in limited game rounds)To ensure a fair comparison, it was crucial that both the Suspicion-Agent and the opponent model experienced the same card strength across an equal number of games. To achieve this balance, we employed a strategic approach: initially, the Suspicion-Agent was placed in position 0 for the first set of 50 games. Subsequently, we re-conduct these games but switched the positions, placing the Suspicion-Agent in position 1 and the baseline model in position 0. This method allowed us to maintain an equal card strength for both the Suspicion-Agent and the baseline model over a total of 100 games, thereby enabling a more accurate evaluation of each model's performance.

b. Way to make it:

To ensure consistency and reproducibility in our experiments, we used the same seed to establish the gaming environments. This approach allowed us to replicate the card distributions in different runs, facilitating the testing of the Suspicion-Agent in varying positions.

4. Additional optimistic variants:

A: Thanks for your suggestions, we add the discussion about these works to the updated version.