Multi-agent KTO: Enhancing Strategic Interactions of Large Language Model in Language Game

Thank you for your valuable feedback and insightful comments on our paper. We truly appreciate the time and effort you dedicated to reviewing our work. We address your comments in detail as below.

---

Q1: Generalization Beyond Werewolf game.

A1: We appreciate this insightful comment on the generalization of our approach. In this work, we demonstrate that through SFT and Multi-agent KTO, LLMs can effectively learn the complex strategies required for the Werewolf game. We believe that social deduction games share significant similarities in their underlying strategic spaces. As mentioned in Appendix J, we hypothesize that with minor fine-tuning, our LLM-based agent can effectively generalize to other games, such as Avalon. We aim to validate this in future work.

Additionally, we plan to include experimental results from more complex game settings, such as 10-player games with Guard/Hunter and 12-player games, to demonstrate the generalization capability of our approach across different player numbers. The detailed results can also be found in our response to Review 3tA5's Q&A-1.

---

Q2: On computational cost: the multi-agent play and KTO refinement loop may entail significant computational cost and large sample requirements. These overheads are not fully characterized.

A2: Thank you for raising this important detail. Taking the largest scale training (MaKTO-72B model) as an example: the training involved 289 9-player Werewolf games. To ensure effective sampling, each game included at least three SFT-72B models. The remaining participants in these games included GPT-4o-mini (180 appearances), GPT-4o (180 appearances), Claude-3.5 (180 appearances), Mix model (162 appearances), and Qwen2.5-SFT-14B (296 appearances). This process yielded 20,674 MaKTO training samples, comprising 12,225 desirable data samples (59.1%) and 8,449 unacceptable data samples (40.9%). Overall, the API calls were affordable.

---

Q3: Impact of Agent Pool Diversity and Strength: how MaKTO's performance varies if newer LLMs are swapped into the pool or if GPT-4o is removed, and whether there's a relationship between pool diversity/strength and final outcomes.

A3: This is an excellent suggestion for an ablation study. We conducted an experiment where we removed GPT-4o from the agent pool and trained a MaKTO-14B-w/o GPT4o model for comparison. The head-to-head win rates are shown below:

From these results, we observe that MaKTO-14B, trained with GPT-4o in its agent pool, achieves a higher average win rate (0.540) compared to MaKTO-14B-w/o GPT4o (0.518). This suggests that the presence of stronger, more diverse agents like GPT-4o in the training pool contributes to learning more robust and effective strategies, leading to better overall performance for MaKTO. We also hypothesize that the diversity and strength of the agent pool has a positive correlation with the final outcomes of MaKTO model. Your suggested ablation study is very interesting. Our current experiment of simply removing GPT-4o is preliminary. We plan to conduct more comprehensive experiments, such as controlling the number of agents in the pool and comparing performance across different model sizes.

---

Q4: How language choice impacts strategy learning, win rates, and detectability.

A4: All behavior cloning, multi-agent play, KTO refinement, and evaluations are conducted in Chinese-only. While the experiments were in Chinese, our human evaluators did not report any issues with GPT-4o's fluency in Chinese. The detectability of GPT-4o in the Turing-style test was primarily due to its behavioral patterns and voting strategies, rather than linguistic proficiency. As described in Line 210, GPT-4o's speaking style was often overly conservative, hesitant to make bold accusations even when the situation seemed clear to expert human players. Its voting behavior also differed, frequently leading to abstentions or votes for non-critical roles, unlike human players who would typically take a decisive stance in crucial moments (e.g., when two players claim to be the Seer). These strategic and behavioral differences were the main factors influencing its detectability. Detailed behavior of GPT-4o can also be found in the response to Reviewer uAsG Q&A-3.

---

Q5: Performance ablation of different training stages.

A5: Thank you for this excellent suggestion to quantify the contribution of each stage. We can provide a clear breakdown of the performance gains from our Behavior Cloning (SFT) stage and the subsequent MaKTO (SFT + KTO refinement) stage, compared to the base instruction-tuned model.

Here are the head-to-head win rates against various agents:

From the table, we can clearly see the contributions:

• Base Model (Qwen2.5-Instruct-72B): As a raw instruction-tuned model, it shows an average win rate of only 24%, indicating a significant gap when playing against API-based models and our fine-tuned 14B models.
• Behavior Cloning (SFT-72B): Training with our collected expert data via SFT significantly boosts performance. The SFT-72B model achieves an average win rate of 57%, representing a 33% improvement over the base Qwen2.5-Instruct-72B model (0.570 - 0.240). This demonstrates the critical importance and quality of our collected expert dataset, enabling the SFT model to even outperform API-based models like GPT-4o and Claude in head-to-head matches.
• MaKTO Refinement (MaKTO-72B): Building upon the strong SFT baseline, our MaKTO algorithm further refines the model's capabilities. The KTO-72B model achieves an average win rate of 65.3%, which is an 8.3% improvement over the SFT-72B model (0.653 - 0.570). This demonstrates that even on top of high-quality behavior-cloned data, our KTO-based refinement loop provides substantial additional gains, contributing to the overall superior performance.

Thank you for your valuable comments on our paper. We truly appreciate the time and effort you dedicated to reviewing our work. We will address your specific comments and concerns below.

---

Q1: Generalization Limitations Due to Human Prior Knowledge. It is unclear whether it can be directly transferred to other types of SDGs.

A1: We appreciate this insightful comment on the generalization of our approach. The "think-before-respond" format in our dataset indeed embeds fundamental strategic insights for social deduction games. Our experiments demonstrate that by training on this dataset, the LLM effectively learns these core strategies and develops strong reasoning abilities specifically for Werewolf. We posit that social deduction games share substantial similarities in their strategic spaces. Therefore, we believe that with minor fine-tuning, our approach could effectively generalize to other SDGs, such as Avalon. We plan to validate this in future work.

Additionally, we plan to include experimental results from more complex game settings, such as 10-player games with Guard/Hunter and 12-player games, to demonstrate the generalization capability of our approach across different player numbers. The detailed results can be also found in our response to Review 3tA5's Q&A-1.

---

Q2: Applicability of Selection Methods to Online MARL and Discovery of Novel Strategies: the selection methods in Table 10 could apply to online MARL and whether online learning could discover strategies different from human experts.

A2: We agree these are highly relevant points. First, directly optimizing LLMs with online Reinforcement Learning (RL) in a multi-agent, dynamic, and long-utterance conversational environment is non-trivial and computationally demanding. To our knowledge, no existing work has achieved direct LLM optimization in such complex multi-party dialogue settings.

Second, we did incorporate an online RL-based baseline: the Mix model[1] -- a Cicero-like policy model, which combines Online RL policy model with an LLM as an expressor (i.e. RL+LLM). In the policy model of Mix, both utterances and actions are formatted into structured inputs (e.g., "Player X labels Player Y as Z identity"). Its reward function incorporates both the ultimate game outcome (win/loss) and round-based correct actions (e.g., correctly voting for a werewolf, seer identifying a werewolf, similar to the heuristic-based rewards in our Table 10). It uses an online MARL training approach, similar to AlphaStar. The LLM then serves as an expressor to articulate the learned policies into natural language utterances. As shown in Table 1 and Figures 3-6, MaKTO consistently outperforms the Mix model on average.

Regarding the question --- "Whether online learning can discover effective strategies different from those of experts," we found an interesting observation in Figure 3 (we simply described in lines 169-171). The strategies learned through online learning do indeed differ from human expert strategies. While the Mix model achieved a strong head-to-head win rate as a Villager, its strategy as a Werewolf was often overly aggressive, leading to a significantly lower Werewolf win rate. For example, we observed a pattern where all three Werewolves would simultaneously claim to be the Seer in one round of day-time discussion. In such a case, if the true Seer gained the trust of most villagers, the identities of all three Werewolves would immediately be exposed.

Due to space constraints, our detailed description of the Mix model's training was brief (only Line 160 in the main text now). We will add a more detailed description of this Online RL + LLM algorithm in the Appendix. Directly applying MARL methods to optimize LLMs remains a challenging research topic that we are actively exploring, though it falls beyond the primary scope of this paper.

---

Q3: Whether GPT-4o's performance in the Turing-style detectability test is attributable to the choice of the Chinese language.

A3: While our experiments are conducted in Chinese, the performance of GPT-4o and its detectability in the Turing-style test are not due to linguistic fluency issues. Based on feedback from our human players, GPT-4o's utterances are fluent and natural in Chinese. The detectability is more from its distinct **speaking style and voting behavior(( compared to human players (as noted in Line 210).

Specifically, in terms of speaking style, GPT-4o tended to be overly cautious. Even in situations where, to an experienced human player, the identity of a Werewolf was quite obvious, GPT-4o would often refrain from boldly accusing a player, preferring to state the need for further observation before confirming a suspicion. Regarding voting behavior, GPT-4o often abstained from voting or voted for a non-critical role. For example, if two players simultaneously claimed to be the Seer (a critical situation where human players would typically vote for one of them to express their stance), GPT-4o might abstain due to uncertainty or vote for another player it suspected, rather than engaging directly with the crucial "Seer" claims. These behavioral differences, not language fluency, were the main factors in its detectability.

---

Q4: Experiments with other open-source LLMs and Influence of foundational model capabilities.

A4: Yes, in the early stages of our experiments, we did SFT train based on Llama-3.1-8B/70B-Chinese-Chat models [2]. Here are the results:

We observe that, for the same SFT, Qwen performed slightly better than Llama-Chinese models in the Chinese context. This observation played a key role in our decision to select Qwen as our base model.

Furthermore, we find that larger model sizes significantly influenced performance. Larger models inherently has stronger instruction-following capabilities and fewer hallucinations. These are crucial for complex game dynamics. For instance, if a Villager model hallucinates or provides incorrect contextual information during its speech, it's more likely to be perceived as a Werewolf, directly reducing the Villager team's win rate. The enhanced reliability and contextual accuracy of larger foundational models therefore directly contributed to the overall performance improvements we observed.

---

[1] Wu, Shuang, et al. "Enhance reasoning for large language models in the game werewolf." arXiv preprint arXiv:2402.02330 (2024).

[2] https://huggingface.co/shenzhi-wang/Llama3.1-8B-Chinese-Chat

Dear Reviewer uAsG,

Apologies for the direct message, and thank you again for your valuable and detailed review of our paper.

We are writing to follow up on the rebuttal we submitted last week. In our response, we have included clarifications on the generalization, comparison with the Online RL method and explanation on GPT-4o's behavior in the detectability test. We also provided experimental results of finetuning other open-source LLMs (Llama3.1) We hope our response has sufficiently addressed your concerns.

As the discussion period deadline is approaching, we would be very grateful for the opportunity to discuss any final questions you may have. We are available to provide any further clarification needed. Thank you!

Best regards,

Submission 11638 Authors

Thank you for your valuable feedback and insightful comments on our paper. We truly appreciate the time and effort you dedicated to reviewing our work. Here are the responses to your concerns.

---

Q1: Minor novelty in algorithm.

A1: We appreciate your detailed assessment of our algorithmic novelty. We believe our core contribution and novelty lie in proposing an end-to-end training paradigm that unifies natural language communication and action decision-making within the language space, directly optimizing LLMs for complex social deduction games. Prior works on Werewolf, including those you cited [3, 4, 5], typically decouple these two aspects: using LLMs for communication (natural language action space) and a separate policy network for action decision-making (formalized action space). This decoupling often introduces a semantic gap between communication and action. In contrast, our approach significantly mitigates this gap by integrating both communication and action within a unified natural language space. To the best of our knowledge, our work is among the first to propose and implement such an end-to-end paradigm for this class of social deduction games.

Furthermore, we would like to clarify that we view MaKTO not as a simple application of KTO, but as an extension of KTO. While KTO was originally designed for preference alignment based on human feedback, we extend its principles to the much more complex domain of multi-agent strategic language games. MaKTO demonstrates how the core idea of KTO can be generalized beyond simple alignment tasks to reinforce strategic reasoning in interactive, language-based environments. We believe that this extended framework is a valuable contribution and holds promise for other multi-agent tasks.

---

Q2: Ambiguous Motivation from Wittgenstein's Language Game Theory, specifically question the mapping between "rules" in Figure 1(c) and "intention" in Figure 1(d).

A2: We apologize if our initial explanation of the motivation was unclear. A crucial insight from Wittgenstein's Language Game Theory is the unity of language and behavior: language games aren't just about speaking; they encompass language interwoven with various actions. Language functions through two-way interaction between people and with the world. This directly informs MaKTO's design: we aim to unify the modeling and optimization of both behavior (in-game actions) and communication (utterances) and to train our model end-to-end by interacting with diverse agents to receive varied feedback. This holistic approach is a key differentiator from prior works [3, 4, 5].

In contrast, previous work, which often decouples intentions and language for modeling, aligns more with Wittgenstein's earlier "picture theory" (Fig1a) where language is merely a representation—in this game context, a representation of intention. Regarding Fig.1, our intention is to convey that language, behavior, rules, intentions ... are interconnected components that should be optimized holistically from interactive data. There isn't a direct one-to-one mapping between "rules" in (c) and "intention" in (d). We will refine the introduction section to ensure our contribution, novelty, and motivation are clearer to readers.

---

Q3: Comparison with DPO.

A3: We understand the reviewer's interest in a comparison with DPO. While it might seem straightforward to construct "prompt-chosen-reject" pairs for DPO, doing so effectively for complex multi-agent, long-horizon games with with sequential decision-making like Werewolf is non-trivial. For example, in a single "night action - discussion - vote" cycle in a 9-player game, there are typically 21 decision turns. Even sampling just two responses per turn would lead to $2^{21}$ possible preference expansions for that single cycle. A full game averagely spans three such night-day cycles, resulting in a staggering $2^{63}$ potential outcomes to enumerate and construct preferences from, which is computationally infeasible. The impact of a single-turn preference of one single agent player can be easily mitigated over a long game sequence, leading to inefficient sampling when creating DPO training data.

Despite this challenge, we followed your suggestion and conducted an experiment with a simplified DPO setup. While heuristic-based construction of DPO training data is simpler for atomic actions (e.g., "Villager voting for a werewolf" as chosen vs. "Villager voting for another villager" as reject), it's much harder for the speech in natural language space. Nevertheless, we attempted to construct some DPO training data for speech data. For staged vote-based speech (e.g., "Werewolf speaking and being voted out"), we selected the utterance of a Werewolf who was not voted out later in the game as the "chosen" utterance. For verifier-based speech, we modified numbers in utterances that conflicted with observable facts in the gameplay to create "rejected" samples.

With such setup, we trained a DPO-14B model and compared its performance with GPT-4o, SFT, and MaKTO. The results are as follows:

From the table, you can see that DPO's win rate is not as good as MaKTO's. This is because Werewolf demands highly nuanced language capabilities, and constructing effective "prompt-chosen-reject" samples for DPO in the complex language space is challenging.

Furthermore, DPO's training stability was inferior to KTO's. DPO frequently experienced training failures and tended to generate long, repetitive, or unparseable text. We specifically used DPO-ftx to achieve stable training. Given these challenges, we believe that KTO is a more suitable and robust approach for this complex domain.

---

Q4: Missing Related Works.

A4: Thank you for pointing out these relevant work. We will add them to our related work section. As [5] and [6] were published very recently, we will incorporate these valuable works into our revised paper and discuss their different approaches. For instance, we will clarify that [6] focuses on no-press Diplomacy, a setting with less complex communication, while [5] utilizes clustering on embeddings, which differs from our method of optimizing directly in the natural language space.

---

Q5: Title Clarification: suggesting replacing "Reinforcing" with "Enhancing" in the title to avoid confusion with reinforcement learning.

A5: We agree that "Reinforcing" might create an unintended association with traditional RL algorithms. To more accurately reflect our contribution and avoid any potential misunderstanding, we will revise the title to use "Enhancing".

Dear Reviewer Je77,

Apologies for the direct message, and thank you again for your valuable and detailed review of our paper.

We are writing to follow up on the rebuttal we submitted last week. In our response, we have clarified the paper's novelty and motivation and added an experimental comparison with DPO. We hope our response has sufficiently addressed your concerns.

As the discussion period deadline is approaching, we would be very grateful for the opportunity to discuss any final questions you may have. If there are remaining issues, We are available to provide any further clarification needed. Thank you!

Best regards,

Submission 11638 Authors

Dear Reviewer Je77,

Thank you again for your time and valuable feedback on our paper. We are writing to gently follow up on the rebuttal we submitted several days ago. We wanted to ensure our responses were clear and to see if there are any remaining points that require further clarification. We would be very grateful for the opportunity to discuss any issues before the discussion period concludes.

Best,

Submission 11638 Authors

Thank you for your follow-up. We are pleased that our responses on motivation (Q2) and other points (Q4, Q5) have resolved your concerns.

Q1: We would like to elaborate on the methodological contribution of MaKTO below:

First, from KTO to MaKTO, our work makes two key findings: (1) training with a diverse pool of agents is crucial for robustness in a multi-agent environment, and (2) intermediate, stepwise rewards are effective. We understand your point and agree that these concepts have been partially observed or mentioned in other related works, which we also acknowledge in our paper (e.g. Line 311). However, our key contribution here is that we are the first to systematically and comprehensively demonstrate and validate these findings within the context of the KTO for complex language games.

Second, we believe the disagreement is on the definition of methodological novelty. You are correct that the components of MaKTO (population-play, reward shaping, KTO) are established concepts, and our contribution is NOT in inventing these components. More importantly, we empirically validate that this specific combination of components is crucial for successfully training an interactive agent for social language games. As prominent works [1, 2] have highlighted, such games are a vital testbed for advancing an LLM's strategic and linguistic capabilities. Our work takes this recognized challenge and provides a simple yet effective implementation on the Werewolf testbed, demonstrating its practical significance. As a paper of application track, we believe our framework has significant value to the language game AI community (a contribution that also includes our annotated dataset).

---

On Q3: The Comparison with DPO.

Thank you for acknowledging the difficulty of optimizing DPO in long-horizon games. We would like to further clarify that even with a stepwise approach, it is non-trivial to construct (context, chosen, rejected) speech pairs for DPO.

To illustrate: in Table 10, a Seer's speech that results in receiving more than half of the villagers' votes is considered as "unacceptable". But how would we construct a DPO pair for this? E.g. for this game snippet:

• After Night 1, Player 4 (a villager) is dead.
• Player 1 (Werewolf): I'm the Seer. I checked Player 3; he's a villager. This helps us narrow down the options...
• Player 2 (Seer): I am the real Seer! Player 1 is a fake. He is a werewolf. I checked Player 3, and he is also a werewolf. Believe me, we have found two werewolves. Let's vote out Player 3! ...
• Player 3 (Werewolf): "Player 2 is the liar! He's a werewolf trying to frame me. I'm a simple villager. Player 1 must be the real Seer. How could two werewolves expose themselves so quickly?..."
• ... (other players' speeches are omitted.)
• Vote Outcome: The majority of players believe Player 1 is a seer and vote out Player 2 (the real Seer).

In this case, the real Seer's speech (Player 2), despite being factually correct according to the game state, is an unacceptable sample for KTO because it failed to persuade the team and led to a disastrous outcome. Now, for DPO, we have the rejected sample (Player 2's speech). But the critical question is: what is the corresponding chosen speech given the same context? There is no objectively "correct" or "winning" speech to construct. This ambiguity in creating a chosen sample for strategic and persuasive language makes a simple replacement of KTO with DPO problematic.

"Stepwise" data does not necessarily imply that effective preference pairs can be easily generated. Despite this challenge, our simplified DPO experiment in the rebuttal has shown that it performed worse than MaKTO and suffered from training instability. This aligns with findings in other works [3,4,5] that note DPO's drawbacks.

---

We hope this deeper dive has clarified our contribution. Our work provides a practical and robust framework for a challenging problem, and we respectfully ask that you might reconsider your evaluation in light of these more nuanced explanations. Thank you again for your review, which has pushed us to better articulate the foundations of our work.

---

[1] Wen, Ying, Ziyu Wan, and Shao Zhang. "Language Games as the Pathway to Artificial Superhuman Intelligence." arXiv preprint arXiv:2501.18924 (2025).

[2] Schaul, Tom. "Boundless socratic learning with language games." NeurIPS Workshop on Language Gamification (2024).

[3] Azar, Mohammad Gheshlaghi, et al. "A general theoretical paradigm to understand learning from human preferences." International Conference on Artificial Intelligence and Statistics. PMLR, 2024.

[4] Pal, Arka, et al. "Smaug: Fixing failure modes of preference optimisation with dpo-positive." arXiv preprint arXiv:2402.13228 (2024).

[5] Saeidi, Amir, et al. "Insights into alignment: Evaluating dpo and its variants across multiple tasks." arXiv preprint arXiv:2404.14723 (2024).

Thank you for your valuable feedback and insightful comments on our paper. We truly appreciate the time and effort you dedicated to reviewing our work. In the following, we address your comments in detail.

---

Q1: Adding Online RL as a Baseline, noting that rule-based rewards make it feasible within the paper's context.

A1: We appreciate this suggestion. However, Directly optimizing Large Language Models (LLMs) with online RL in complex multi-agent, long-utterance conversational environments is non-trivial and computationally intensive. To the best of our knowledge, there is no existing work that directly optimizes LLMs in such complex multi-party dialogue game scenarios. While we define "desirable actions", these are not strictly rule-based rewards that guarantee a win or loss. For instance, as shown in Table 10, a Seer identifying a Werewolf is a desirable action, which generally increases win probability. However, as illustrated in the case in Appendix H, there are cases where the Seer doesn't identify any Werewolf, yet the Villager team still wins through their deductions.

It is also important to note that we do include an "Online RL" baseline in our experiments: the Mix model [1]. This Cicero[2]-like policy model, which combines Online RL policy model with an LLM as an expressor. Its results are presented in Tab.1 and Fig.3-6. In the RL training for the policy model, we format utterances and actions into structured inputs (e.g., "Player X labels Player Y as Z identity") and employ a MARL approach as in AlphaStar[3] for policy modeling, using the LLM for speech generation. As mentioned in Figure 3 and lines 168-170, while the Mix model shows a good overall average win rate, it tends to expose the Werewolf team more easily. We believe the Mix model serves as a relevant Online RL baseline. Due to space constraints, the description of this baseline was concise in the main text, but we we will expand on its detailed description in the revised paper.

---

Q2: Generalization Evaluation: Simply replacing the Guard with the Hunter is insufficient to evaluate generalization ability; recommending evaluating under settings with a different number of players.

A2: This is an excellent suggestion, and we agree that broader generalization testing is crucial. We have conducted additional experiments with three new, unseen game settings, which were not present in either our behavior data or KTO training data:

• 10-player game: Seer, Witch, Guard + 4 Simple Villagers + 3 Werewolves
• 10-player game: Seer, Witch, Hunter + 4 Simple Villagers + 3 Werewolves
• 12-player game: Seer, Witch, Hunter, Guard + 4 Simple Villagers + 4 Werewolves

Win rate in a 10-player game with Guard:

Win rate in a 10-player game with Hunter:

Win rate in a 12-player game:

Our experiments show that MaKTO consistently achieves a better winning rate compared to the SFT model across all three new settings without any additional training. The performance improvement stems from MaKTO's ability to enhance the model's underlying common reasoning capabilities and strategic skill release through interactive feedback.

---

Q3: Detailed Data Comparison between desirable and undesirable training data.

A3: Here are the detailed stats of the expert-annotated data:

The purpose of the "SFT w/ desirable" experiment was to investigate whether filtering out "unacceptable" responses would lead to better performance compared to training SFT on the full dataset. Our findings indicated that, on average, training only on desirable actions actually resulted in worse performance than training on the full SFT dataset and then doing MaKTO training .

---

Q4: Whether MaKTO is effective to apply to smaller models, beyond Qwen2.5-72b.

A4: Yes, MaKTO's effectiveness is demonstrated on smaller models as well. In our paper, we provide performance evaluations for the 14B model size in Table 1, Figures 3-4, and Tables 4-8. Specifically, our ablation study in Section 3.6 confirms that the MaKTO algorithm is effective and performs well at the 14B model scale.

---

Q5: Comparable Methods, eg. [4].

A5: We appreciate you bringing up this work. The paper you cited is indeed relevant and has been mentioned in our introduction and related work sections. Unfortunately, as the authors have not open-sourced their code, we were unable to reproduce their results or conduct a direct comparison. However, as discussed in A1, we have compared our approach against another important baseline: the Mix model [1]. This is a Cicero-like model that integrates RL with an LLM.

---

[1] Wu, Shuang, et al. "Enhance reasoning for large language models in the game werewolf." arXiv preprint arXiv:2402.02330 (2024).

[2] Meta Fundamental AI Research Diplomacy Team (FAIR)†, et al. "Human-level play in the game of Diplomacy by combining language models with strategic reasoning." Science 378.6624 (2022): 1067-1074.

[3] Vinyals, Oriol, et al. "Grandmaster level in StarCraft II using multi-agent reinforcement learning." nature 575.7782 (2019): 350-354.

[4] Xu, Zelai, et al. "Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game." In Proc. of ICML2024.

Dear Reviewer 3tA5,

Apologies for the direct message, and thank you again for your valuable and detailed review of our paper.

We are writing to follow up on the rebuttal we submitted last week. In our response, we have provided a clarifications on the comparison with the Online RL method and other comparable methods. We have also included the experimental results on generalization, data statistics of desirable and undesirable training data, and results of smaller models. We hope our response has sufficiently addressed your concerns.

As the discussion period deadline is approaching, we would be very grateful for the opportunity to discuss any final questions you may have. We are available to provide any further clarification needed.

Best regards,

Submission 11638 Authors

Dear Reviewer 3tA5,

Thank you again for your time and valuable feedback on our paper. We are writing to gently follow up on the rebuttal we submitted several days ago. We wanted to ensure our responses were clear and to see if there are any remaining points that require further clarification. We would be very grateful for the opportunity to discuss any issues before the discussion period concludes.

Best,

Submission 11638 Authors

Dear Reviewer 3tA5,

Thank you for your follow-up. We are very pleased to hear that our detailed explanations regarding the generalization evaluation and data comparison have successfully addressed your concerns. We appreciate you taking the time for this important discussion on what appears to be the remaining fundamental issue: the novelty, i.e. core contribution.

While we agree that advancing multi-agent RL is a promising research direction, NOT every paper on agent learning must have this as its primary objective. We respectfully ask that our work be evaluated on its stated contributions: the proposal of a novel unified training paradigm for language game agents and the strong empirical results that validate its effectiveness. We believe a rejection based on our paper not solving a different set of research problems CANNOT be fully justified.

With this context in mind, here are our specific clarifications on the paper's novelty and methodology:

1. On Methodological Novelty and Scope: Our paper's primary goal is not to solve the foundational challenges of multi-agent training, and we have never claimed to propose a "genuine multi-agent system" in the traditional MARL sense in the paper. Instead, our core contribution focuses on a different but crucial problem: how to effectively train a single, unified policy that can jointly handle both language communication and strategic decision-making in complex social deduction games. This is a significant difference from existing literature [1,2,3], which typically trains communication and decision-making policies in two separate stages. As our experiments show, our unified approach demonstrates superior performance.

2. The MaKTO framework is a deliberate methodological choice to address this specific challenge efficiently. As we noted (Lines 114-118), the huge action spaces in multi-agent dialogues lead to sparse trajectory sampling, making online RL algorithms difficult to train. We were encouraged that your initial review recognized this contribution, describing our work as* an extension of KTO to a "multi-agent setting" with strong performance* (Strength 1).

3. On the "Multi-Agent" Paradigm: As for your mention that "its underlying paradigm essentially constitutes a single-agent reinforcement learning framework with data collection capabilities". Regarding the paradigm being a "single-agent framework," we would like to offer a different perspective. The key distinction between multi-agent and single-agent approaches often lies in whether the data collection process involves dynamic interactions among multiple agents or not. Our framework's data is generated precisely from such multi-agent interactions. This paradigm — training a centralized policy on data from multi-agent gameplay — is a common and highly successful training approach used in game AI research, including OpenAI Five[4] and AlphaStar [5]. Therefore, our method firmly presents a multi-agent approach.

We hope the above explanation can successfully clarify the core contribution of our work.

We again respectfully ask that our work be evaluated based on its stated contributions and its demonstrated success in this domain, rather than for not addressing a different set of research problems. Given these clarifications, we would be sincerely grateful if you would be open to reconsidering your evaluation. Thank you again for your time and for this important discussion.

---

[1] Meta Fundamental AI Research Diplomacy Team (FAIR)†, et al. "Human-level play in the game of Diplomacy by combining language models with strategic reasoning." Science 378.6624 (2022): 1067-1074.

[2] Shibata, Hisaichi, Soichiro Miki, and Yuta Nakamura. "Playing the Werewolf game with artificial intelligence for language understanding." arXiv preprint arXiv:2302.10646 (2023).

[3] Xu, Zelai, et al. "Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization." arXiv preprint arXiv:2502.04686 (2025).

[4]Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemysław Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. 2019. Dota 2 with large scale deep reinforcement learning. arXiv:1912.06680. Retrieved from https://arxiv.org/abs/1912.06680

[5]Vinyals, O., Babuschkin, I., Czarnecki, W.M. et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 575, 350–354 (2019). https://doi.org/10.1038/s41586-019-1724-z