Learning to Discuss Strategically: A Case Study on One Night Ultimate Werewolf

Thank you very much for your positive comments.

Response to W1: The transferability of proposed algorithms and proofs to different games or game-like environments like financial markets is not demonstrated.

Thank you for your comment. As this work is initially motivated by our analyses of the ONUW game, we mainly focus on the ONUW game in the experiments. We recognize that the financial markets is an interesting and highly practical environment for demonstrating our agents, but it needs relevant knowledge to construct agents for this scenario. However, since the agent framework we proposed is modular, we believe it can be transferred to various scenarios with similar discussion-driven dynamics like bargaining in the financial markets, as long as we have certain knowledge or data to train a corresponding discussion policy.

Response to W2: I have not found any examples of the discussions between the LLM players in the manuscript. Further, the code is not provided by the authors.

Thank you for your constructive comment. We made further human analyses based on the discussion logs between LLM players. However, due to the page limits in the global rebuttal, we can only provide one analysis in the PDF file. More analysis results and discussion examples will be updated in the revision. Also, the anonymized link to our code has been sent to the Area Chair in a separate comment as required by the conference guidelines.

Response to W3: The proposed model is seemingly limited by the predefined discussion tactics and does not allow learning or discovery of alternative discussion tactics.

Thank you for your comment. As discussed in our work, the predefined six discussion tactics were intended to simplify the problem and facilitate the learning process, due to the lack of a standardized classification for discussion tactics and adequate human player data in games like ONUW. While this approach does constrain the agents to choose from a predefined set of tactics, it also provides a structured framework for understanding the dynamics of discussion. However, as part of future work, we consider it interesting to explore more flexible and adaptive methods that allow for the discovery of new discussion tactics as well.

Response to W4: It would be good to have a random choice baseline and compare the proposed method to its performance.

Thank you for your constructive comment. We have added an LLM-based agent whose discussion policy is to randomly choose discussion tactics as a new baseline (named Random). Please refer to the global response for the results.

Response to W5: About the number of negotiation rounds.

Thank you for your feedback. Actually, although the ONUW game has only one Night, one Day, and one Voting phase, there could be many rounds of discussions during the Day phase where players are allowed to discuss, depending on the game settings. And in our settings, we allowed players to have three rounds of discussions. Please refer to the PDF file in our global response for the detailed game process. Moreover, we consider our discussion policy can adapt to multiple rounds of discussions since it relies on the current state of discussion rather than a specific number of rounds.

Response to Q1: The reason for choosing offline RL.

Thank you for your comment. The main reason we chose offline RL is the slow interaction with LLMs, especially the interaction rate limits of GPT-4. Online RL requires agents to interact with the environment in real-time to collect data for training. However, as some parts of our agent framework are LLMs and each inference of LLMs takes a few seconds, it is almost impossible for our agents to train the discussion policy in an online manner. So we decide to collect game data first and then adopt offline RL for training.

Response to Q2: How is the classification into six discussion tactics performed: manually or by an 0-shot LLM?

The classification of players' discussion tactics is performed by a 0-shot LLM.

Response to Q3: About the "human-side" or "villager-side".

Thank you for pointing it out. The "human-side" is the name that reference [28] used to represent the "villager-side", oppositing to the "werewolf-side". As it might cause confusion for readers, we have replaced it with "villager-side" in our latest version to keep consistency with other parts of the paper.

Response to Q4: What is the reasoning behind selection of this specific number 1536 for state embeddings?

Thank you for your insightful comment. The 1536 state dim is actually decided by the model we selected for state embeddings. In our experiments, GPT's text-embedding-ada-002 model is adopted to encode players' observations and beliefs into state embeddings. We believe it is well pretrained and could effectively represent the semantic information of the original text. We have not performed a hyperparameter sweep across different state dims as it is not the main scope of our work, but it is an interesting suggestion for future work to explore the impact of varying the state dim.

Response to Q5: What is the total cost of the experiments described in the paper?

Thank you for your comment. For each evaluation in our experiments, we repeated the game 30 times to obtain the final results. Since we conducted experiments on one three-player game (Sec 6.2), two five-player games in fixed settings (Sec 6.3), and one five-player game in a random setting (Sec 6.4), there are 50 sets of results. However, the cost of each agent version (such as ReAct and LLM-instructed) varies a lot due to their frameworks, and the Gemini API was free when we conducted our experiments, so we can only estimate the total cost be around 200.

Thank you very much for your insightful review.

Response to W1: Differences from related work [1].

Thank you for your comment. We have discussed the differences between our work and related work [1] (line 80-82). Here are more detailed differences. And these will be added to the main paper in a future version.

The main differences between our work and [1] lie in the model architecture and the training process. In related work [1]'s agent model, a set of action candidates is generated and the RL policy is trained to select one action among these candidates. However, the RL policy in our framework (i.e. discussion policy) is directly integrated into the thinking and decision-making process of agents. So the RL policy in [1] can be seen as an adjustment for the action distribution of the LLM-based agent, while our discussion policy is part of the agent's strategy. As for the training process, related work [1] adopts a population-based RL training method to adapt to the diverse actions of various players. We believe the discussion policy should be generic and invariant across players, so it is trained in a single-agent manner. Finally, the experimental results of our agents in both three-player and five-player games indicate good scalability, which is not explicitly shown in [1].

Response to W2: It is possible that players in the training dataset appear again during testing, resulting in less convincing results.

Thank you for your feedback. In fact, most agents in our experiments are different from those used for generating game logs for training. Specifically, the players in the training dataset only utilize the GPT-4-based LLM-instructed agent. On the contrary, the players in our experiments have various settings, including ReAct, Belief, LLM-instructed, and RL-instructed, while adopting Gemini or GPT-4 as the backend LLM in different experiments. For example, all players in Figure 4 use Gemini as the backend LLM to avoid the potential appearance of players in the training dataset.

We believe the results are convincing since these game logs only reflect the preferences of GPT-4 which are supposed to be different from Gemini. Also, the differences between agents' versions have an impact on players' performances as shown in our experimental results, so we can regard them as different players.

Response to W3: About typos.

Thank you for pointing out the mistakes. We have fixed these typos in our revision.

Response to Q1: About the dimensions of players' beliefs.

We appreciate the reviewer for pointing it out, as it might confuse readers who are not familiar with game theory. Each belief in our theorem corresponds to a probability distribution over a specific information set. Therefore, the dimension of belief is actually the number of states in its corresponding information set. According to the game rules, players should vote for other players simultaneously in the Voting phase, which can be seen as a normal-form subgame. However, the game tree is used to describe the extensive-form game, so we need to transform the Voting phase into the extensive form for better demonstration:

• In our game tree (Fig 2 and 5), we assume Player 2 votes after Player 1 and Player 3 votes after Player 2.
• For Player 1, since it does not know which action Player 3 (original Robber) takes at night, there are 3 potential states in its information set.
• For Player 2, it does not know Player 1's voting choice as they vote simultaneously in fact, and it also does not know Player 3's action at night, so there are 6 potential states in its information set in the game tree considering Player 1 votes before it.

Therefore, the different belief dimensions are actually due to the manually set "pseudo" voting order. But it has no impact on the final derived equilibria results.

Response to Q2: Detailed explanation of why the NashConv values of GPT-4-based agents are higher than Gemini-based agents.

The NashConv value is defined as $\text{NashConv}(\pi)=\sum_i\left[R^i(\text{BR}(\pi^{-i}), \pi^{-i}) - R^i(\pi)\right]$, which represents how much utilities players can gain by deviating to their best responses. It is notable that players' gains in NashConv actually relate to other players' strategies $\pi^{-i}$, so the NashConv value is a result of all players' strategies.

Let us still take the three-player game as an example. In this game, Player 1's action space is $A^1=\{V2, V3\}$, Player 2's action space is $A^2=\{V1, V3\}$, and Player 3's action space is $A^3=\{(NS, V1), (NS, V2), (S1, V1), (S2, V2)\}$. We ignore the $(S1, V2)$ and $(S2, V1)$ in $A^3$, since they are respectively dominated by $(S1, V1)$ and $(S2, V2)$ (this result is stated in Appendix D.1). Considering the following two strategy profiles:

1. $\pi_1$: Player 1 and Player 2 always vote for Player 3, while Player 3 adopts a random strategy. In this case, $\text{NashConv}(\pi_1) = 1/2+1/2+0 = 1$.
2. $\pi_2$: All players adopt random strategies. And in this case, $\text{NashConv}(\pi_2) = 1/2+1/2+1/4=5/4$.

We can see that even though the deviation gains of Player 1 and Player 2 are the same in $\pi_1$ and $\pi_2$, the deterministic strategies of Player 1 and Player 2 in $\pi_1$ make Player 3 can no longer find a better response, resulting the lower NashConv value comparing to $\pi_2$.

When we analyze the evaluation logs of the experiments in Section 6.2, we find that when employing Gemini, the two Werewolves (Player 1 and Player 2) are more likely to both vote for Player 3, while the GPT-4-based agents are more random. It suggests that the Gemini-based agents' strategy profile is closer to $\pi_1$, so its NashConv value is lower than the GPT-4-based agents in the same settings.

References:

[1] Zelai Xu, et al. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.18940, 2023.

Thank you very much for your positive feedback.

Response to W1: About the oversimplification and manual discretization of the discussion tactic space.

Thank you for highlighting this limitation. As acknowledged in our work, the six discussion tactics are manually identified and simplified. This is primarily due to the lack of a standardized classification for discussion tactics and adequate human player data in games like ONUW. But the manual discretization of discussion tactic space provides a structured framework for understanding the dynamics of discussion during the game, and the experimental results have to some extent demonstrated the effectiveness of learning a discussion policy based on these tactics to improve the discussion ability of LLM-based agents. And as part of future work, we consider it interesting to explore techniques to automatically identify the discussion tactics during the gameplay as well.

Response to W2: The generalization is still limited to the ONUW game.

Thank you for your comment. Since the manually selected discussion tactics are specific to the ONUW game, the trained discussion policy is naturally limited to this game. However, the framework we proposed is modular and can be adapted to various scenarios with similar discussion-driven dynamics. For example, if we can somehow collect widely used bargaining tactics, then we could train a bargaining policy in the same way and apply it to our framework to construct an LLM-based agent for bargaining.

Response to Q1: How did the authors determine the specific categories for discussion tactics?

Thank you for your comment. The specific categories for discussion tactics were determined mainly through the inspiration from prior research on argumentation [1, 2] and analyzing human choices when playing similar games. And to highlight the potential deception in the ONUW game, we further divide these tactics into honest and deceptive ones. We have not yet explored automated methods for discovering these categories, but we highly acknowledge it is an interesting avenue for future research, as it could enhance the automation level of the LLM-based agent constructions and might discover unexpected tactics among humans' discussions.

Response to Q2&W3: Did the authors consider including any human participants in their trials?

Yes, we consider it would be interesting to include human players to play with the agents as well. As the main topic of our work is to demonstrate the importance of discussion in the ONUW games and to improve the discussion ability of LLM-based agents, we did not consider the impact of human participants in our experiments. However, we are planning to add a human interaction interface to our project code, which will be updated soon.

Response to Q3: How do the authors imagine the role-switching dynamics of ONUW impacted their results? What might change were the roles fixed?

Thank you for your insightful comment. As we analyzed in Section 4 and the game tree, the role-switching dynamics of ONUW do significantly impact the game's outcome, players' utilities, and the strategies employed by players. And it is the role-switching dynamics that make the three-player ONUW game possible. Imagine if roles were fixed in the three-player case, then two Werewolves would know the player left must be on Team Village and vote it out together, which would make the game meaningless.

Also, we believe that the reasoning difficulty brought by the role-switching dynamics is one of the key challenges that reflect the abilities of different language models. If roles are fixed, the ability gaps between different models in the experimental results would be narrowed down, and the need for strategic deception and counter-strategies would be reduced, possibly leading to simpler discussion policies.

References:

[1] Bolin Lai, et al. Werewolf among us: A multimodal dataset for modeling persuasion behaviors in social deduction games. arXiv preprint arXiv:2212.08279, 2022.

[2] Winston Carlile, et al. Give me more feedback: Annotating argument persuasiveness and related attributes in student essays. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 621–631, 2018.

Thank you very much for your valuable feedback.

Response to Q1&W1: Detailed comparison with related work [1] and [2].

Thank you for your constructive comment. We have discussed the differences between our work and related work [1] (line 80-82). Here are detailed differences between our work with related works. And these will be added to the main paper in a future version.

• Differences from Xu et al. (2023) [1]: The main differences lie in the model architecture and the training process. In related work [1]'s agent model, a set of action candidates is generated and the RL policy is trained to select one action among these candidates. However, the RL policy in our framework (i.e. discussion policy) is directly integrated into the thinking and decision-making process of agents. So the RL policy in [1] can be seen as an adjustment for the action distribution of the LLM-based agent, while our discussion policy is part of the agent's strategy. As for the training process, related work [1] adopts population-based RL training method to adapt to the diverse actions of various players. We believe the discussion policy should be generic and invariant across players, so it is trained in a single-agent manner. Finally, the experimental results of our agents in both three-player and five-player games indicate good scalability, which is not explicitly shown in [1].
• Differences from Wu et al. (2024) [2]: From the perspective of the model architecture and usage, RL in related work [2] is utilized to train the Thinker module, which focuses on handling complex logical analysis and strategic planning in specialized tasks, while the RL policy in our work (i.e., the discussion policy) mainly focuses on enhancing the discussion abilities of LLM-based agents. In the training process, the input and output of the RL policy in [2] are structured language features. But the input in our discussion policy is the embedding of current observation and belief and the output is the chosen discussion tactic. Meanwhile, as the context generated by the Presenter in [2] is supposed to be consistent with Thinker's produced policies, the agent framework in [2] barely considers the significance of speaking strategically (i.e. being honest or deceptive), which is actually the motivation and core of our work.

Response to Q2&W2: The human evaluation for agents' performance.

Thank you for your comment. We understand the importance of human evaluations in assessing the quality of AI's performance and its ability to mimic human-like strategy discussions. We have conducted several human evaluations and analyses based on the game logs. However, due to the page limits in the global rebuttal, we can only provide one typical analysis in the PDF file. More results will be added to the appendix in the revision.

Response to W3: Detailed information of the dataset.

Thank you for your comment. We have provided the process of data collection, the dataset statistics and analysis in Appendix E.1. The dataset used for offline RL consists of 120 game logs (containing 1800 discussion turns) from the five-player ONUW game. And for each log, it includes the entire discussion history, players' discussion tactics, the initial and final role assignment, voting results, and game outcomes.

Response to Q3: Can the authors opensource the dataset?

Thank you for your feedback. We highly acknowledge the importance of opensourcing the code and dataset of our work. The anonymized link to our code has been sent to the Area Chair in accordance with the conference guidelines. And right now we are cleaning our dataset, which is scheduled to be opensourced after the rebuttal period.

Response to Q4: Clarification on offline RL implementation.

Thank you for your comment. Our offline RL training details can be found in Appendix E.2. In our implementation, we first use GPT's text-embedding-ada-002 model to encode players' observations in the dataset into state embeddings, and then adopt these embeddings for further offline RL training. The reason we adopted a frozen encoder is that GPT's embedding model is well pretrained and we believe it could effectively represent the information of the original text in the semantic space. Also, related work [1] adopts prior embeddings before training. However, we agree with the reviewer's point that fine-tuning the embedding model may help further improve the performance of the agents. We will clarify this in the revision.

References:

[1] Zelai Xu, et al. Language agents with reinforcement learning for strategic play in the werewolf game. arXiv preprint arXiv:2310.18940, 2023.

[2] Shuang Wu, et al. Enhance reasoning for large language models in the game werewolf. arXiv preprint arXiv:2402.02330, 2024.