Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game

W3: The paper lacks explanations on how credit assignment is handled in a multi-agent setting. Additionally, it does not specify the reward structure. The impact of the hyperparameter 'N', which represents the number of generated actions, on the results is also not discussed.

We have added descriptions of reward design and credit assignment in Appendix D.2 of our revised paper.

• Winning reward: all winners +100, all losers -100.
• Werewolf killing reward: if the Werewolves successfully kill a player at night, the Werewolves +5, the Villagers (Seer, Doctor, Villagers) -5.
• Seer seeing reward: if the Seer successfully identifies a Werewolf at night, the Werewolves -2, the Seer +2.
• Doctor saving reward: if the Doctor successfully saves a player at night, the Werewolves -5, the Doctor +5.
• Voting result reward:
  • If a Werewolf is voted out, the Werewolves -5, the Villagers (Seer, Doctor, Villagers) -5.
  • If a non-Werewolf is voted out, the Werewolves +5, the Villagers -5.
• Individual voting reward:
  • If the current player votes for a Werewolf, the Werewolves -1, the current player +1.
  • If the current player votes for a non-Werewolf, the Werewolves +1, the current player -1.
  • If the current player chooses not to vote, no additional reward for any player.

We also added an ablation on hyperparameter N = 2, 3, 4, 5 in Appendix E.6. The table below shows that increasing N will produce more actions, but the diversity increased by newly added action is small (distance < 0.2) when N = 4, 5. This is because sometimes the number of possible actions is small. For example, when there are only 2 Werewolves and 3 other players alive (a common situation on the second night), the Werewolves only have 3 possible actions and N = 4, 5 will not introduce diversity in the action candidates.

We extend our sincere gratitude for your feedback and hope our answers have addressed your concerns. Your support is invaluable to us and we genuinely hope that our efforts merit a raise in your rating.

Thank you for your constructive comments and questions! We are encouraged to see your positive assessment of our method's effectiveness and robustness. We hope the following responses can address your concerns.

W1: The approach mainly combines prompt engineering with RL, specifically tailored for the Werewolf game. It's unclear how this would inspire or be applicable to other tasks.

Our approach has three components: (1) deductive reasoning, (2) diverse action generation, and (3) population-based RL training. We believe our framework of using these three components is general enough to be adapted to other social deduction games. The main changes would be specific prompt designs but the general framework stays the same. Specifically, while some prompts used in the first component are carefully designed for the Werewolf game, the prompts for generating diverse actions, and the population-based reinforcement learning method in the second and third components are general to be applied in other tasks like social deduction games (e.g., The Resistance: Avalon [1]) and board games with negotiations (e.g., Diplomacy [2]).

Take The Resistance: Avalon as an example. To apply our method in this game, we need to specialize the deduction template in the first component and maybe use additional prompting techniques like [1]. We also need to change the prompt in the third component to generate the agent pool for this game. Then our method can be applied without changing the diverse action generation and RL training components. We have added a discussion on our method's application to other tasks in Appendix H.1 of our revised paper.

[1] Wang, Shenzhi, et al. "Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation." arXiv preprint arXiv:2310.01320 (2023).

[2] Meta Fundamental AI Research Diplomacy Team (FAIR)†, et al. "Human-level play in the game of Diplomacy by combining language models with strategic reasoning." Science 378.6624 (2022): 1067-1074.

W2: The paper does not clearly justify the need for using RL for action selection. What is the added benefit of the extra training cost incurred by using RL?

The motivation for using RL is that LLM alone cannot produce the optimal action distribution and is likely to be exploited by adversarial opponents in zero-sum games. By contrast, RL can learn the optimal action distribution and produce less exploitable policies. To intuitively show the benefit of using RL, we compare the action distributions of agents with and without RL policy in three cases and the discussion can be found in Section 5.4 of our revised paper.

Consider the case when it is the first night and the Werewolves need to decide who to kill. The agent without RL policy has a high probability to choose player_0 (0.38) and a low probability to choose player_5 (0.01). An adversarial Doctor can take advantage of this pattern by always saving player_0 and achieve a 0.38 success rate. In comparison, our agent with RL policy produces a uniform distribution and no Doctor can achieve a success rate higher than 0.14. Please see Section 5.4 and Figure 4 in our revised paper for a detailed analysis on two more cases.

Q4: A cognitive architecture with short-term and long-term memory is worth considering in implementations, baselines, and related work.

We agree with the reviewer that there are potential ways to further improve the performance of our agent by using better prompting techniques or agent architecture. However, the main focus of this work is to combine LLM with RL to build strategic language agents, and we pay less attention to prompt design and agent architecture. Moreover, in the Werewolf games, the whole game history can fit in the LLM's context window (4k) most of the time, so we defer the Werewolf-specific short-term and long-term memory design to future work.

Q5: Why is self-attention used on the action embeddings? A much more natural approach is just to learn a Q value function.

The key goal of the population-based RL training component is to learn a policy to select the best action from the candidates generated by the diverse action generation component. Using a self-attention policy or a Q value function are implementation choices, both ways are possible. We did try to learn a Q function in our early attempt but found the performance improvement was not significant. This may be because value-based methods could have unsatisfying results in multi-agent environments with diverse behaviors [5]. Therefore, we turned to the current self-attention implementation and used PPO to train the policy.

[5] Fu, Wei, et al. "Revisiting some common practices in cooperative multi-agent reinforcement learning." arXiv preprint arXiv:2206.07505 (2022).

Q6: Exp 5.1 round-robin tournament.

The last row (Ours) is bolded to show that our agent achieves the highest win rates as the Villagers. The last column (Ours) is underlined to show that our agent also achieves the lowest lose rate as the Werewolf. We have added more explanation in our revised paper.

We have updated Figure 3 to report both mean and std. For LLM-based agents like "Vanilla" and "Concurrent", the randomness only comes from the LLM. For methods with RL policy like "Atomic" and "Ours", we added two more runs with different seeds and the randomness comes from both the LLM and the RL policy.

We agree with the reviewer that regret or exploitability would be a better metric in theory. However, in practice, computing regret or exploitability requires finding the best response to a fixed policy, and it is impossible to get the exact best response in complex n-player games like Werewolf. Instead, cross-play and human evaluation are commonly used to evaluate performance [1, 2].

The agent within the same team uses the same method. For example, the first row means the Villagers are all Vanilla agents, and the first column means the Werewolves are all Vanilla agents.

[1] Jaderberg, Max, et al. "Human-level performance in 3D multiplayer games with population-based reinforcement learning." Science 364.6443 (2019): 859-865.

[2] Meta Fundamental AI Research Diplomacy Team (FAIR)†, et al. "Human-level play in the game of Diplomacy by combining language models with strategic reasoning." Science 378.6624 (2022): 1067-1074.

Q7: Exp 5.2 human evalution.

We have added results on the win rate in each round in Appendix E.8 of our revised paper. As shown in Table , the win rate of human players gradually increases as they played more games against other agents, while stay relatively unchanged against our agent. This shows our agent is more robust to adversarial opponents.

We would like to clarify that the data in Table 1 is the win rate of human players, and a smaller value means a better performance of the evaluated agent. Therefore, our agent w.o. training and diversity performs better than the vanilla agent.

As we mentioned in the paper, each participant played with 6 AI agents of the same type. We divided the 80 participants into 4 groups, each group corresponding to one AI agent type. In addition, participants in each group are divided into two roles, the Villagers or the Werewolves. So effectively the 80 participants are divided into 8 categories. Human players are divided randomly and each category has 10 players. The other 8 values in Table 1 is produced by replacing the human player with our agent and playing agaisnt the same other AI agents. These data are used to compare the performance of human players and our agents.

For agent just w.o. diversity, the LLM only produces one action, and there is no need to use the RL policy to select from the action set. Therefore it is the same as the agent w.o. training and diversity.

Q8: Exp 5.3 zero-shot transfer. we have updated Table 2 to report both mean and std.

We genuinely value your dedication to reviewing our paper and believe our detailed responses have addressed your concerns. We would really appreciate it if you could consider raising the rating of our work based on our responses.

W3: All problems/concepts of diversity are punted to just asking the LLM to be diverse. No guarantees of diversity or notions of what kind of diversity.

The objective of our diverse action generation component is to generate a set of strategically different actions. Although there are many ways to define what is diversity, they may not align with human's intuition and produce undesired behaviors. In fact, the best way to decide if the actions are diverse is to use human evaluation. In our method, LLM is used as a human proxy to decide what actions are diverse in the Werewolf game and we find simply asking the LLM to "propose strategically different actions" is effective and produces desired results. We have added two qualitative examples in Appendix C.6 of our revised paper.

To quantitatively validate this claim, given an action set $A$, we can define the increased diversity introduced by a new action $a$ as $\mathrm{div}(a, A) = \mathrm{min}_{a'\in A} \|e(a) - e(a')|$, i.e., the minimum Euclidean distance between its embedding and existing actions' embeddings. A larger value means that the new action $a$ is different from any actions in the set $A$. As a reference, the distance between “kill player_i” and “kill player_j” is about 0.36 when $i \neq j$. We consider $N = 2, 3, 4, 5$ and evaluated the increased diversity introduced by the last action. As shown in the table below, our method produces actions that are different from existing actions. The increased diversity is smaller when $N = 4, 5$ because in some cases the number of available actions is smaller than 4 or 5. We use $N = 3$ in our implementation and this result has been added in Appendix E.6 of our revised paper.

W4: The SelfPlay algorithm isn't well described and takes many changes from existing algorithms without analyzing their impact.

We have added a section on the implementation details of population-based training in Appendix D.3 of our revised paper. We also added an ablation to compare our agent with a self-play agent without using the agent pool (trained with itself and its checkpoints). The table below shows that our agent achieves higher win rates. In general, adding more agents with different styles to the agent pool will make the final agent more robust. This result has been added in Appendix E.2 of our revised paper.

Q1: Why is reliability on a 1-10 scale?

The 1-10 scale is just one reasonable implementation choice, it is also possible to use 1-5 scale or 1-100 scale. The focus of our design is to combine LLM with RL to build language agents with strategic play. The detailed prompting choices are not optimized as long as they produce reasonable results.

Q2: Are all of the four attributes generated by the deduction LLM necessary? Is there any data on ablations of this information?

We added an ablation on the four attributes by gradually adding role, confidence, evidence, and reasoning and comparing their win rates against our agents with all four attributes. The table below shows that all attributes improve the performance and are necessary. This result has been added in Appendix E.3 of our revised paper.

Q3: Why are reliability and confidence separate and somehow being treated as additive/substitutive?

We would like to clarify that reliability is determined by confidence.

• If a player is deduced to be a Werewolf, then their reliability = 11 - confidence. In other words, if a player is deduced to be a Werewolf with high confidence, then we should not trust their statement and their reliability should be low.
• If a player is deduced to be a Seer, Doctor, or Villager, then their reliability = confidence. That is, if we think a player is likely to be a non-Werewolf, then we should trust their statements.

Thank you for your review and valuable feedback! We provide the following explanation can address your concerns.

W1: The motivational claims about the advantages of this benchmark are tenuous.

We would like to clarify that our work focuses on LLM-based agents and aims to improve their strategic reasoning ability. Comparing to existing environments for LLM-based agents, the Werewolf game has the following two challenges.

1. decision-making with deceptive information. We didn't cite papers to support our claim on "Prior work on LLM-based agents …" because there is little work that evaluates these agents in deceptive environments by the time of our submission. This claim is validated by our own results in Figure 3 and Table 1 where our agent with deductive reasoning achieves stronger results than the vanilla LLM-based agent. We have also added [1] (which was published after our submission) to support the claim in our revised paper.
2. strategic play in mixed cooperative-competitive games. Most works on LLM-based agents focus on single-agent or cooperative tasks. The mixed cooperative-competitive property of Werewolf is a key difference from existing environments like ALFWorld.

For the claim that "it is impossible to achieve strong play in Werewolf without communication", we would like to emphasize that communication is critical in Werewolf to share information and deduce hidden roles. Without communication, there is no way for the Villagers to know who is the Werewolf. To validate this claim, we added agents w.o. communication as baselines and the table below shows that no communication greatly hurts the performance. The result has been added in Appendix E.4 of our revised paper.

[1] Wang, Shenzhi, et al. "Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplation." arXiv preprint arXiv:2310.01320 (2023).

W2: Hidden role games are analogous to ad hoc teamwork and opponent-policy belief/likelihood modeling and these are not discussed nor used as potential baselines.

We would like to clarify that the key problems in hidden role games and ad hoc teamwork are fundamentally different.

• The key problem in hidden role games is to identify who are the teammates [2]. Once the teammates are correctly identified, collaboration with them is relatively easy.
• In comparison, the key problem in ad hoc teamwork is to collaborate with new teammates without prior coordination [3]. The agents do not need to identify their teammates and the challenges lie in zero-shot coordination.

For opponent modeling, we have discussed in the Related Work section about DeepRole [2] which combines deductive reasoning with CFR to reason about joint beliefs. In our revised paper, we have included additional relevant papers [1, 4] (published after our submission) that use opponent modeling in social deduction games or design LLM-based agents.

[2] Serrino, Jack, et al. "Finding friend and foe in multi-agent games." Advances in Neural Information Processing Systems 32 (2019).

[3] Mirsky, Reuth, et al. "A survey of ad hoc teamwork research." European Conference on Multi-Agent Systems. Cham: Springer International Publishing, 2022.

[4] Guo, Jiaxian, et al. "Suspicion-Agent: Playing Imperfect Information Games with Theory of Mind Aware GPT-4." arXiv preprint arXiv:2309.17277 (2023).

W8: Motivation for studying and improving the performance of agents in an environment that encourages deception and lying by agents is lacking.

The motivation for studying and improving the performance of agents in an environment with deception and lying lies in the pursuit of advancing AI capabilities in complex, human-like scenarios. Werewolf, as a social deduction game, provides a challenging and dynamic environment to explore the limits of language communication and strategic thinking. Our goal is not to promote deception per se but to improve agents' ability to recognize deceptions and stay robust against adversarial opponents. In reality, human society and AI-generated content are full of deceptive or misleading content. Understanding and countering deception in AI systems is crucial to prevent malicious use with harmful intent.

W9: No ethics statement.

We have added a section on the ethics and societal impact in our revised paper. Please see Appendix A for detailed discussions.

W10: Quantitive results for emergent behaviors.

We have added results on the average occurrence of four emergent behaviors and compared our agent with three other agents. The result in Figure 7 of our revised paper shows that our agent produces complex behaviors like bluffing and sacrificing more often than other agents.

W11: Analysis of the failure cases.

One failure case is Werewolf agents could unintentionally reveal their true identity in discussion. This is because the Werewolf agents need to first reason as a Werewolf and then pretend to be a non-Werewolf in discussion. This mismatch in their thoughts and words could make the LLM get confused and accidentally speak out their thought as a Werewolf. Fortunately, the probability of such a failure case is significantly reduced by generating multiple diverse actions and using an RL policy to choose the best one. Please see Appendix H.3 in our revised paper for a detailed analysis of more failure cases.

We would like to express our appreciation for your constructive review. We have carefully addressed each of your concerns and believe that our responses demonstrate the value of our proposed approach. We kindly request you reconsider the rating for our submission and genuinely hope our efforts will warrant a higher evaluation.

W4: What are the generalization settings that the models are tested for? Can the model generalize to new forms of the game? What are the limits?

The methods are tested for generalization to unknown teammates and opponents. During training, our agent does not know the teammates or opponents in test time and only plays with its past version and agents in the predefined agent pool. In test time, our agent is evaluated with players that are not seen in training like human players and other baselines.

For generalization to new forms of the Werewolf game, the first two components in our method (deductive reasoning and diverse action generation) can be directly generalized to new settings like changing the number of players, adding new roles, and changing the winning conditions, with slight changes in the prompt.

The RL training method can also be directly used in different settings, but the policy needs to be retrained for a new setting to achieve the best performance. We expect an RL policy trained under one specific game setting to generalize to similar settings like adding or removing one player, but we do not expect it to generalize to games with significant changes like changing the winning conditions. To evaluate the transferability of a trained RL policy to slightly different game settings, we considered a 6-player and an 8-player Werewolf game and compared agents with and without the RL policy trained on the 7-player game. The table below shows the RL policy can generalize to the 6-player and 8-player settings and improve the performance. We have added the results and discussions in Appendix H.2 of our revised paper.

W5: Can the LLM itself be used as a reward model as in [1, 2] to choose actions?

The two mentioned works use LLMs in very different ways and we will discuss them separately. [1] does not use LLM as a reward model. Instead, it directly uses LLM as an agent and proposes several prompting techniques to improve its strategic reasoning ability. Their method requires recursive searches over the finite game tree and is not applicable in our setting because the action space in Werewolf is prohibitively large if not infinite (the agent can say anything in discussion).

[2] uses LLM as a reward model to solve the reward design problem when the reward is hard to specify. The example in their paper is to generate a reward function for a "versatile" negotiator. However, the reward design in our setting is clear and simple: the winners get +100 reward and the losers get -100 reward. We do use simple reward shaping as described in Appendix D.2, but the designs are straightforward and there is no need or advantage to use LLM for reward design.

[1] Gandhi, Kanishk, Dorsa Sadigh, and Noah D. Goodman. "Strategic Reasoning with Language Models." arXiv preprint arXiv:2305.19165 (2023).

[2] Kwon, Minae, et al. "Reward design with language models." arXiv preprint arXiv:2303.00001 (2023).

W6: Analogously a discussion on an alternative method where an LLM is used as a reward model to train a separate agent is needed, what are the advantages of an LLM agent?

First, the pretrained LLMs can be directly used in the agent for natural language understanding and communication, which is critical in the Werewolf game. By contrast, training a separate agent from scratch for natural language communication would require a huge amount of data, even when an LLM is used as a reward model.

Second, as discussed in the previous question, the reward in Werewolf is simple and clear. There is no need to use LLM for reward design.

W7: The similarities to prior work like Cicero diminish the novelty claims. The key difference seems to be using language for reward estimation.

We would like to clarify that our method is very different from Cicero.

• Cicero first uses the RL policy to choose from the game-predefined action set and then uses the selected actions as input for LLM to generate action-conditioned dialogue.
• Our method first uses LLM to propose diverse action candidates and then uses the RL policy to select from these actions.

The difference comes from the fact that Diplomacy is a board game with predefined actions. By contrast, the actions in the discussion phase of Werewolf can be any natural language and can not be predefined.

We also implemented the "Atomic" baseline which is inspired by Cicero and compared it with our agent. Results in Figure 3 show that our agent achieves better performance.

Thank you for your time and constructive comments! We appreciate your acknowledgment of our proposed environment and behavior analysis. In response to your questions, we provide the following explanation.

W1: More implementation details are needed for the multi-agent RL algorithm.

We have added a section on the implementation details in Appendix D of our revised paper. The section includes detailed discussions on the attention policy architecture, environment reward, MARL hyperparameters, and population-based training setup. We are willing to provide more explanations if any details are not clear.

W2: Additional evaluation of the MARL method would strengthen the results, such as multiple training runs and assessing cross-population transfer.

We have added two more runs to train the RL policy and report the mean and std in Figure 3 of our revised paper. The result remains the same: our agent achieves the best performance. For cross-population transfer, we are not sure about the meaning of this evaluation. We are happy to add this evaluation if you could provide more explanation.

W3: Lack of baselines: The justification for the particular method is lacking. What about alternate forms of prompts and decomposing reasoning? Are there ablations of the method that can help understand where the performance gains are coming from?

To the best of our knowledge, there is no prompting technique specialized for social deduction games like Werewolf other than the included concurrent work [3] by the time of our submission. Therefore, we used a vanilla LLM agent and the concurrent work as our baselines.

For comparison with general prompting techniques, we added CoT [4], ReAct [5] as our baselines and compared them with our agent using deductive reasoning only (no diverse action generation and RL training). The table below shows our prompting design achieves the highest win rates against our agent both as the Villagers and as the Werewolves. This result has been added in Appendix E.1 in our revised paper.

The ablation of the three components in our method is shown in Table 1 using human evaluation. All three components contribute to the performance gain. Specifically, comparing to the Vanilla agent, adding the deductive reasoning component leads to 7% performance improvement as the Villagers and 5% improvement as the Werewolves (evaluated agents are the column player and the data is the win rate of the row player). Adding the diverse action generation and population-based RL training components also further improve the performance. In addition, we have added a round-robin tournament evaluation between our agents and its three ablated versions. The updated results are in Figure 6 of our revised paper.

[3] Xu, Yuzhuang, et al. "Exploring large language models for communication games: An empirical study on werewolf." arXiv preprint arXiv:2309.04658 (2023).

[4] Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in Neural Information Processing Systems 35 (2022): 24824-24837.

[5] Yao, Shunyu, et al. "React: Synergizing reasoning and acting in language models." arXiv preprint arXiv:2210.03629 (2022).

Q3: Does the author consider changes in the same game but different settings?

For generalization to new forms of the Werewolf game, the first two components in our method (deductive reasoning and diverse action generation) can be directly generalized to new settings like changing the number of players, adding new roles, and changing the winning conditions, with only slight changes in the prompt.

The RL training method can also be directly used in different settings, but the policy needs to be retrained for a new setting to achieve the best performance. We expect an RL policy trained under one specific game setting to generalize to similar settings like adding or removing one player, but we do not expect it to generalize to games with significant changes like changing the winning conditions. To evaluate the transferability of a trained RL policy to slightly different game settings, we considered a 6-player and an 8-player Werewolf game and compared agents with and without the RL policy trained on the 7-player game. The table below shows the RL policy can generalize to the 6-player and 8-player settings and improve the performance. We have added the results and discussions in Appendix H.2 of our revised paper.

Q4: Does the selection of the RL policy raise concerns about consistency?

The consistency in agents' behavior is achieved by both the LLM and the RL policy. The past actions and statements are used as input for the LLM, therefore, the LLM knows its previous behavior and most of the proposed action candidates are consistent. However, it is possible that the LLM produces inconsistent statements. In this case, the inconsistent statement is not a good action and the RL policy learns to assign it a low probability to avoid sampling it. Please see Appendix H.3 in our revised paper for more discussions on potential failure cases.

We genuinely value your dedication to reviewing our paper and believe our detailed responses have addressed your concerns. We would really appreciate it if you could consider raising the rating of our work based on our responses.

Thank you for your valuable feedback! We are encouraged to see your appreciation of our proposed method and experiment results. For your constructive questions, we hope the following response can address your concerns.

W1: A limited number of tests and a narrow selection of baseline comparisons.

To the best of our knowledge, there is no prompting technique specialized for social deduction games like Werewolf other than the included concurrent work [1] by the time of our submission. Therefore, we used a vanilla LLM agent and the concurrent work as our baselines.

For comparison with general prompting techniques, we added CoT [2], ReAct [3] as our baselines and compared them with our agent using deductive reasoning only (no diverse action generation and RL training). The table below shows our prompting design achieves the highest win rates both as the Villagers and as the Werewolves. This result has been added in Appendix E.1 in our revised paper.

In addition, we would like to emphasize that prompt design is just one of our three components. The second component (diverse action generation) and third component (population-based RL training) use LLM with RL and can be combined with any new prompting techniques.

[1] Xu, Yuzhuang, et al. "Exploring large language models for communication games: An empirical study on werewolf." arXiv preprint arXiv:2309.04658 (2023).

[2] Wei, Jason, et al. "Chain-of-thought prompting elicits reasoning in large language models." Advances in Neural Information Processing Systems 35 (2022): 24824-24837.

[3] Yao, Shunyu, et al. "React: Synergizing reasoning and acting in language models." arXiv preprint arXiv:2210.03629 (2022).

W2: Description of the experimental part is not detailed enough and some details are missing.

We have added a section on the implementation details in Appendix D of our revised paper. The section includes detailed discussions on the attention policy architecture, environment reward, RL training hyperparameters, population-based training setup. We are willing to provide more explanations if any details are not clear.

W3: RL requires both strong language understanding capabilities to comprehend the intentions behind all possible actions, and the ability to solve reasoning tasks. This dual demand can potentially result in low learning efficiency or pose challenges in the learning processes

We would like to clarify that the strong language understanding capability comes from the LLM. The action embedding produced by LLM can be regarded as a compact representation of the intention in the semantic space. The RL policy directly uses these embeddings as input and only needs to learn the optimal action distribution to improve the strategic reasoning ability. The RL policy in our experiment is trained for 50M environment steps using 4 RTX4090 GPUs and does not have challenges in learning efficiency. Please see Appendix D of our revised paper for more training details.

Q1: Details for human evaluation experiments.

Human players take the same text input as our agent, which includes their ID and hidden role, their Werewolf teammate (if any), their secret night actions (if any), the announcement, the discussions, and the voting results. Please see Appendix E.7 in our revised paper for an example input for human players.

Currently, we only use text as input because Werewolf is a language game and the words of players contain the most important information. It would be an interesting future direction to consider more input like the expression and tones.

Q2: The zero-shot transfer experiments doesn't prove the RL policy's potential applicability to various reasoning tasks. Can the author provide more evidence to verify this, such as transferability between different games?

We would like to clarify that the zero-shot transfer experiment aims to show the trained RL policy's applicability to unseen LLMs, not different tasks or games. The results show that the RL policy trained on one LLM can be directly combined with other LLMs to improve their performance in the Werewolf game. Since the RL policy is trained only under this specific game, it cannot be transferred to different games. However, we believe our framework itself can be adapted to other social deduction games, with the deductive deduction component reasoning about game history, the diverse action generation component generating multiple actions, and the population-based RL training component choosing from the generated actions in a strategic fashion.

Thank you for your time and feedback!

We are committed to addressing any issues to ensure the clarity and reproducibility of our results. Could you please provide more details or specific aspects that require further elaboration? We are more than happy to provide additional information and clarifications to enhance the reproducibility of our result. Your insights are crucial to improving the quality of our work, and we want to ensure that all necessary details are readily available.

Thank you for your appreciation of our work and thoughtful comments! We are heartened to see your recognition of our work’s novelty and the positive assessment of our experiment results. We hope our responses can address your concerns.

W1: Mischaracterisation of the Cicero algorithm.

Thank you for your explanation. However, we think we have the same understanding of Cicero as yours. In the second line on page 3 of our submission, we discussed the difference between Cicero and our method as follows.

The main difference between Cicero and our method is that they have a predefined atomic action set and select the action first to generate languages. By contrast, the actions in our methods are natural languages generated by LLMs during the game and our agent selects an action after these languages are produced.

The "Actomic" baseline in Section 5.1 is similar to Cicero in that it also first uses an RL policy to choose from predefined actions and then uses the LLM to generate action-conditioned statements.

W2: Human benchmarking results and conclusions about robustness: The authors should make it clearer earlier on in the paper that the robustness results are limited to the single-human setting, and discuss the limitations of this choice in the results section.

We agree with your suggestions and have updated the Introduction and Experiment sections of our paper accordingly. In our case, since LLMs are pretrained on a large amount of human data, they can be regarded as human proxies and produces human-like behaviors [1]. We have provided example game logs of our agents in Appendix I to show that the evaluated agents’ behaviors are similar to humans’. Therefore, the single-human setting is a reasonable evaluation in our case. We agree with your comments on evaluation with multiple humans and have added a discussion on the limitation of single-human evaluation in Appendix H.4 of our revised paper.

W3: There is no ethics statement accompanying this paper.

We have added a section on the ethics and societal impact in our revised paper. Please see Appendix A for detailed discussions.

W4: There are some missing literature citations that it would be good to include.

Thank you for your suggestion. We have included these two citations in the Related Work section and discussed their relation to our method in our revised paper.

We extend our sincere gratitude for your feedback and hope our answers have addressed your concerns. Thank you again for your support and appreciation of our work.

I thank the authors for their response. A few points of clarification:

W1: mischaracterisation of the Cicero algorithm.

We do not believe that the authors have taken on board our advice. More specifically the following claim is false:

"[Cicero selects] the action first to generate languages"

It is not true that Cicero selects the action and only then generates language. In fact, Cicero interleaves action selection and language generation, and the action selection is conditioned on the previous conversation, as it must be for the conversation to be relevant to the strategy. The authors should update this statement in any camera-ready version.

It is also not true that their "atomic" baseline is similar to Cicero. Their atomic baseline is a pure RL agent trained on atomic actions, whereas Cicero has both atomic actions and a conversational element. The authors should remove this statement in any camera-ready version.

The difference between their agent and Cicero is therefore more subtle and should be clarified much more carefully. Specifically, in their agent, RL has as its action space the set of language utterances, whereas in Cicero, RL has as its action space the set of atomic actions in the game. In other words, they apply RL within the conversational component, which Cicero does not.

W2: Human benchmarking results and conclusions about robustness.

I disagree with the authors that their agents produce "human-like" behaviours. It is not sufficient to eyeball game transcripts as in Appendix I, since this is not a scientifically rigorous process. The justification that this evaluation is strong because the LLMs are "human-like" is simply not backed up with empirical evidence.

It would be much better for the authors to point out that this is simply a starting point, and that it is important to evaluate in the multiple human case before drawing strong conclusions. In fact, pointing out a limitation in such a stark way is the hallmark of good science. As it is, the authors have weakened their paper by continuing to overclaim the significance of the single-human evaluation.

Overall, I feel that the authors have failed to take on board my feedback, and these two concerns over their claims remain. If I could, I would lower my score by one point (two points seem too harsh). So please consider my new score to be a 7.