A Generalist Hanabi Agent

Thank you for your thoughtful review and detailed feedback. We are pleased that you found the idea of casting Hanabi as a text-based game and using language-based representations intriguing, especially as a means to reason across diverse game settings. We also appreciate your recognition of R3D2's ability to achieve consistent policies and your interest in the potential of our generalist agent approach. You have raised some important concerns and questions, which we address below.

The paper is not able to have convincing results of the benefit of having a “generalist” agent, i.e. the benefit of going from R3D2-S to R3D2-M. If the generalist agent is given the same amount of data as the sum of the 4 single-variant agents, will it outperform the single agents?

We would like to remark that a major benefit of R3D2 is that it is the first one to even be able to play on different game-settings. OBL, OP and IQL all need to be retrained to accommodate different player-settings. R3D2-M allows us to play any game-setting competitively with the same agent, something that is simply not possible with the baselines. And, while R3D2-S can play on other game-settings thanks to its architecture and textual representation, its zero-shot performance is worse than R3D2-M.

Despite this, we have resumed the training of both R3D2-S and R3D2-M which, due to computational constraints, was trained for less iterations than the baselines. We expand upon this in the general response [About R3D2’s performance], but the gist is that, for 2-player settings, the performance of R3D2-S increases for both self-play (22.23 → 23.34) and intra-XP scores (19.66 → 23.23). This trend is also observed for R3D2-M, with 20.46 → 21.40 for self-play and 20.26 → 21.27 for intra-XP.

Symmetries are thus implicitly tackled by R3D2. Why is this the case?

R3D2 leverages a language-based representation, which implicitly models symmetries and is inherently compositional (Colas et al., "Language as a Cognitive Tool to Imagine Goals in Curiosity-Driven Exploration"). This allows R3D2 to learn generalizable strategies by marginalizing over symmetries, effectively covering many equivalent conventions.

we want policies that coordinates better with human-like policies (OBL) rather than inhuman policies (IQL) The high inter-AXP results are not super exciting

Thank you for raising this point. R3D2 focuses on coordinating with novel partners and game settings using only self-play. We not only have high inter-XP across agents but also, R3D2 achieves higher inter-XP scores with OBL agents while only having been trained with self-play, suggesting that it learns more human-like strategies, as OBL has been shown to coordinate well with humans. Benchmarking with human evaluations is an important direction, and we leave that as future work. Additionally, we want to mention our method is orthogonal to OBL agents. In future work, it would also be interesting to combine R3D2 and OBL.

What are the agents on the x-axis of Figure 6? Are they R3D2-k agents trained for each game variant? If so, why is the diagonal not the same as R3D2-SP?

Thank you for pointing this out! The numbers in the bottom part of the heatmap represent cross-play scores, and the x-axis corresponds to an outsider agent introduced to the game. To make it even easier, we made a new plot in the revised paper (Figure 4). We have now divided the plot based on the player setting, where each sub-plot represents a specific setting. The grey area highlights the cross-play results.

In summary, we have addressed your concerns by providing additional training iterations for R3D2, improving our explanations regarding the benefits of R3D2-M, and reformulating parts of the manuscript for clarity. We kindly ask you to reconsider your rating in light of these improvements and the novelty of our contributions. We believe our work opens exciting directions for future research and contributes meaningfully to the MARL community. If we have not addressed your concerns to satisfaction, we would be glad to provide additional details.

Thank you for the detailed explanation. Now I understand the results much better, despite quite some effort. I would encourage the authors to improve the presentation to emphasize the benefits.

I like that R3D2-3,4,5p models have a non-trivial performance in the 2-player setting despite being train on a single game setting.

I am a bit surprised that in n-player setting, xp(R3D2-M, R3D2-k) where k != n is not noticeably better than xp(R3D2-k, R3D2-k) because I would expect the R3D2-M to be more "in-distribution"?

I also like the new results on the GPT's performance on this task, although it is from other reviewer's comment.

For this question below, I think one experiment, such as training on 2,4,5 but play on 3 player-setting, is sufficient.

Is it possible to train the R3D2-M on 3 out of the 4 setting and play it in the remaining setting zero-shot?

I would like to increase my score to 8 to support the acceptance of this paper. Although I am more leaning towards somewhere between 6 and 8 but this paper has gotten so many reviews and I truly appreciate the authors effort to address them. So I pick the more encouraging side.

Thank you for your review and encouraging feedback. We are thrilled that you found the problem we tackle original and appreciated the novelty of our evaluation protocol, which we believe provides representative insights about the zero-shot capabilities of agents. Your recognition of the high quality and clarity of our paper is incredibly motivating, and we are grateful for your thoughtful insights.

You have made some valuable comments, which we address below.

XL land paper or Maestro (both citations which are missing in this work).

Thank you for these references. We have expanded the related work (Section 2, line number - 155 to 159) to discuss these works.

hard to disentangle the importance of the R2D2 style training, the RRN modification or the language based component underneath. Ablations would be appreciated to understand where the juice is actually coming from. Its really hard to work out why this method is working at all, is this just better domain randomisation, is there some particular underlying set of hinabi skills that generalise well over n players.

This is a very valid point. We have added additional ablation experiments (Section 6.2 and A.7) to assess the role of language in R3D2. We kindly refer to our general comment on the [Role of language and language models]. In summary, while a text representation provides some benefits for generalization, handling dynamic action spaces is equally crucial for successful transfer to novel partners.

I’d also like to just know if OBL agents in the language based environment generalize better.

This is a really interesting proposition. The goal of our work was to design a simple, scalable, and robust agent that requires no architectural changes across different player settings. Notably, R3D2 achieves higher inter-XP scores with OBL agents while using only self-play. However, our contributions can be seen as being orthogonal to OBL agents. We believe combining R3D2 and OBL would be a really interesting avenue for future work. Please refer to the general response [About R3D2’s performance] for more detailed discussion.

The results lack qualitative analysis and as such, its really hard to understand whats happening here or the significance of the work. Is the hope that the method has learnt via a larger set of environments / tasks to have more robust policies. If so what do these policies look like? Do they share underlying similar structures or methods? Could we also evaluate methods on OOD tasks (e.g 7 players hinabi or something not in the train?).

By proposing R3D2, we have created an agent capable of playing across different game settings, that can also generalize well across different types of agents. This is achieved through 2 main components: the incorporation of language to interact with the environment, and the design of a neural network architecture that allows for dynamic actions.

This allows us to propose 2 different setups for R3D2:

1. [Zero-shot Transfer] R3D2-Single (R3D2-S), which is trained on a single game-setting, and evaluated on others. We show that an R3D2-S agent trained in a 2-player setting can play in a 3,4,5-player setting reasonably well without additional training or architectural modification. This is also an Out of Distribution (OOD) Task (Figure 4 ), since the strategies required to win the game depend on the number of players.
2. [Multi-task MARL] R3D2-Multiple (R3D2-M), which is a single model trained on all the player-settings simultaneously. In this case, we demonstrate that, due to the overlap of strategies of different settings, we can train a single, generalist Hanabi agent that plays well on all settings.

It is important to note that the underlying algorithm for both setups is exactly the same. The role of language is crucial here, since its structure and compositionality allows for a great overlap between the representations of the different game-settings. While we do not have qualitative analysis, we assessed the role of language through multiple experiments, and a new ablation study that has been added in the revision. This is explained in the general comment [Role of language and language models]

Its hard to interpret the results in Figure 6 - yes it seems R3D2 is robust to co-players changing but because i don’t have an equivalent evaluation with a team of IQL agents or OBL agents its very hard to evaluate if this is comparatively better.

IQL, OP and OBL are using non flexible architectures (i.e., they assume a fixed state and action space). This means that we simply cannot evaluate them on different player-settings, so there cannot be an equivalent evaluation as for R3D2. You could say they have a score of 0 since, by design, they are restricted to the same-player-setting. This is also the advantage of R3D2: it is the first Hanabi agent that can cope with different player-settings.

We thank you for your comments. We find it uplifting that you are inspired by our use of language modeling to solve the MARL transfer ability issue. Text is a powerful medium of representation, and it allows us to produce agents that can play with other agents, even though they have been trained with self-play.

We appreciate your comments, which we address below:

W1: More up-to-date models like LLAMA 3 or GPT-4o is also encouraged

The objective of our GPT-4 experiments and LLaMA-7B is to evaluate how effectively these models perform on the Hanabi, either out of the box or with minimal adaptation respectively. Our findings demonstrate that SOTA LLMs like GPT-4 are insufficient for playing Hanabi effectively. This aligns with observations by Hu et al. (ICML 2023), who note that LLMs struggle with the game's complexity. However, the main goal of the paper is to demonstrate that by using text-based representation, self-play agents can achieve strong cross-play performance without requiring complex training methods. This is significant because self-play is simpler, more scalable, and computationally efficient compared to methods that require population-based training or hierarchical policies.

We would like to emphasize that LLMs are not the core aspect of our methodology. Our proposed algorithm, R3D2, uses a small 4.1M-parameter TinyBERT model. Despite this, R3D2 is capable of transfer. This shows the soundness of our approach, which is to combine textual observations with a neural network architecture for dynamic action-spaces to enable transferable policies trained using self-play. Finally, we have resumed the training of R3D2, so it matches the number of training epochs of OBL, the strongest baseline. We discuss this in detail in the general comment [About R3D2’s performance], but the gist is that this led to significant improvements in self-play (22.23 → 23.34) and intra-XP scores (19.66 → 23.23).

Is it possible to transfer the learned policy to a different task? How about comparing the proposed approach to existing schemes like curriculum learning or multi-task MARL? Related work should also include such literature.

Yes, it is possible and we have updated the related work section to include multi-task and curriculum based transfer and generalization. We have explored those 2 setups in R3D2,

1.[Zero-shot transfer] R3D2-Single setting (R3D2-S), can handle dynamic action and state spaces which was not possible in the baseline approaches. We show that an R3D2-S agent trained in a 2-player setting can play in a 3,4,5-player setting reasonably well without additional training or architectural modification (Figure 4).

2.[Multi-Task MARL] R3D2-Multi setting, (R3D2-M) makes it possible to train one single model to play all the Hanabi games simultaneously using the same amount of training budget. With the baselines, it is standard to train individual models to play in different settings, which doesn’t resonate well with real-world scenarios. For example, a human player can play 2,3, 4,5 games.

To the best of our knowledge, this is the first study to investigate generalization across game settings in Hanabi while relying solely on self-play for training. For more detailed response please refer to the general comment section for [Is the R3D2 architecture task agnostic?]

Thank you for your thoughtful review and for appreciating the innovative use of language modeling in addressing MARL transferability. We hope our clarifications and additional experiments sufficiently address your concerns and highlight R3D2’s unique contributions. If our responses meet your expectations, we kindly request you to consider raising the score to reflect these strengths. If you have any additional questions, we would be happy to answer them.

We would like to thank you for your review, and your insightful feedback. We are happy that you agree that we conduct extensive experiments to demonstrate the zero-shot capability of R3D2 on 2-player coordination and policy transfer over different settings. Our work is the first that examines cross-play across game-settings in Hanabi, providing new insights into zero-shot coordination in MARL.

You made some very valid concerns, which we address below.

Why is the opinion of 'language has been shown to improve transfer' is valid? Can you provide experimental results to support this claim? For example, how would the performance change if natural language inputs and pre-trained language models were replaced with vector inputs and standard neural networks? Alternatively, could you provide more solid literature to support this idea?

The role of language for transfer has been investigated in previous work (Radford et al., 2019b; Brown et al., 2020), but indeed we previously did not make any experiments to assess the exact impact of text for Hanabi. We have worked hard to improve that aspect, by providing additional ablation experiments that introduce R2D2-text, which show that, while text does offer some help for transfer, handling dynamic action spaces is critical for successful transfer to novel partners (updated results in Section 6.2 and Appendix A.7.2). This is discussed more in details in the general comment section for [Role of language and language model].

Additionally, we performed ablation studies to investigate the role of language representation:

1. Pre-trained weights: Pre-trained weights significantly improve sample efficiency compared to random weights (Figure 12a).
2. Updating the language model: Reducing the frequency of updates drastically limits performance, highlighting the importance of frequent updates (Figure 12b).

What is the relationship among the evaluation metrics SP, Intra-XP, and Inter-XP? Figure 4 shows that R3D2 achieves significantly better Inter-XP compared to the baselines, but the SP metric shows a significant drop, even larger than the improvement in Inter-XP. How should these three metrics be interpreted comprehensively to determine which method performs best?

This is another great comment. Concerning the significant drop of SP, we initially only trained R3D2 on 2000 epochs (compared to 3000 epochs for OBL) due to computational constraints. To satisfy the concerns about performance, we have resumed our experiments for R3D2. We explain this in more detail in the general comment [About R3D2’s performance] but, in summary, R3D2's performance significantly improved in both self-play (SP) (22.23 → 23.34) and intra-XP (19.66 → 23.23).

Regarding the evaluation metrics: all three—SP, Intra-XP, and Inter-XP—are important. However, it is challenging to develop a single algorithm that excels across all metrics, leading to inherent tradeoffs. For example, an agent with superior performance in self-play tends to become overly specialized to a specific setting, making it difficult to generalize to other agents or adapt to different scenarios. This tradeoff between generalization and specialization is a well-known challenge in the machine learning community. If the task in hand requires generalization to novel partners (coordination), Inter-XP is more important to maximize, while if we are interested only in solving the current task, SP is more important. However, we argue that, in the future, we will likely interact with many different types of agents. All these agents should be able to cooperate with each other. To assess this, a high Inter-XP score is required. Our main contribution lies in achieving this generalization across partners and game settings using self-play training, which is typically overspecialized and brittle to changes in partners.

This paper focuses on the Hanabi game. Could you discuss whether the proposed method can inspire improvements in MARL algorithms for other tasks?

Thank you for your comment! While we demonstrate our approach on Hanabi, our core technical contributions - text-based representation for better transfer, architecture for dynamic action/state spaces, and variable-player learning - are domain-agnostic advances that could benefit MARL applications. Please refer to the general comment section for more details [Is the R3D2 architecture task agnostic?]. More previous works like Foerster et al. (ICML 2019), Hu et al. (ICML 2020), Cui et al. (NeurIPS 2022), Hu et al. (ICML 2021) have successfully used Hanabi to develop fundamental MARL concepts that have influenced broader cooperative AI research.

We clarified the importance of Hanabi as a benchmark in the revision of the paper (Section1-Introduction).

We are grateful for your insightful comments and feedback. We appreciate that you agree that the goal of our method, i.e., to generalize to different settings, is a notorious challenge in RL in general. R3D2 is the first agent that is able to achieve this in Hanabi, as no other agent can transfer from one player setting to another.

We have carefully considered the concerns raised and would like to address them as follows:

it seems like the only novelty in R3D2 is the ability to generalize to new rules, specifically changing the number of players. [...] , and in general are presented more as a minor perk of the model. W3: the paper should be more clear about what R3D2 does not achieve, e.g. SOTA in inter-algorithm play, rather than give the impression that R3D2's main contribution is SOTA performance.

We agree that the main novelty of R3D2 is its capability to generalize to new settings in Hanabi. But this is a notorious challenge in MARL, and R3D2 is the first Hanabi agent that can do so, so we would like to emphasize that this is definitely not a minor perk. Even more interesting is that R3D2 achieves generalization across partners and game settings using self-play training. This is surprising, since self-play is typically very brittle to changes of players. But R3D2 achieves similar cross-play performances than much more complex methods. Moreover, the dynamic architecture of R3D2 allows us to have 2 different settings, using the same algorithm:

1.[Zero-shot transfer] - R3D2-Single setting (R3D2-S), can handle dynamic action and state spaces which was not possible in the baseline approaches. We show that an R3D2-S agent trained in a 2-player setting can play in a 3,4,5-player setting reasonably well without additional training or architectural modification (Figure 4).

2.[Multi-Task MARL] R3D2-Multi setting (R3D2-M) makes it possible to train one single model to play all the Hanabi games simultaneously using the same amount of training budget. With the baselines, it is standard to train individual models to play in different settings, which doesn’t resonate well with real-world scenarios. For example, a human player can play 2,3, 4,5 games. Both highlight different aspects of R3D2.

Finally, we have addressed your concern concerning R3D2’s performance. To this end, we resumed training R3D2, which had initially been limited by computational constraints. As outlined in the general response [About R3D2’s performance], this extended training resulted in notable improvements in self-play (22.23 → 23.34) and intra-XP scores (19.66 → 23.23).

The authors claim "R3D2’s intra-algorithmic cross-play score is on par with its self-play score, a first for Hanabi agents learning through self-play". While this may be true for self-play agents, the OBL algorithm also achieves equivalent cross- and self-play scores

Thank you for raising this point! You touch upon one of the main points we wanted to convey in our paper. OBL has been devised precisely because self-play is brittle in cross-play. The fact that self-play can reach similar cross-play performances as OBL, which incorporates specific mechanisms to cope with other players, makes it all the more impressive.

We hope our clarifications and additional results emphasize the unique contributions of R3D2 and its potential impact. If our responses adequately address your concerns, we kindly ask for your consideration in increasing your score. We remain available should you have any more questions.

Thank you for your review, and your feedback. We are grateful that you find our evaluation setup (cross-strategy and cross-setting evaluations) of interest. To the best of our knowledge, we are the first to propose such an extensive evaluation for zero-shot coordination in Hanabi, which allowed us to make a comprehensive evaluation of R3D2’s generalization capabilities.

We acknowledge your concerns, and address them below.

Even though the proposed method is armed with language models, its performance seems not desirable.

We appreciate your concern regarding R3D2’s performance and have addressed it with careful attention. To this end, we resumed training R3D2, which had initially been limited by computational constraints. As outlined in the general response [About R3D2’s performance], this extended training resulted in notable improvements in self-play (22.23 → 23.34) and intra-XP scores (19.66 → 23.23). Now, our agent significantly outperforms both IQL and OP.

Even though the paper pursues a meaningful goal, the final results provide limited insights.

the language model only plays the role of modeling the text embeddings. This usage is somehow not quite novel and effective.

We agree that the role of the language model is limited (even though very important). While we use text due to its abilities for transfer, this is just a means to an end. The main goal of this paper lies in achieving generalization across partners and game settings using self-play training. This is significant because self-play is simpler, more scalable, and computationally efficient compared to methods that require population-based training or hierarchical policies. Seeing this, we argue that using text embeddings has proved quite effective.

Moreover, to provide more insights, we have worked on incorporating additional experiments for 3 and 4-player Hanabi games (Appendix A.6). These experiments show that our results are consistent across game-settings.

Thank you once again for your concerns, which have helped us refine our work further. We hope that the additional experiments, extended training results, and clarifications regarding the role of language models adequately address your questions and demonstrate the broader significance of our contributions. We believe these updates reinforce the effectiveness and scalability of R3D2, particularly in achieving generalization across diverse partners and game settings. If these updates address your concerns, we kindly ask you to consider raising your score to reflect these improvements. Thank you once again for your constructive feedback and for helping us enhance the quality of this paper.

We are thankful for the feedback you have provided for our paper. Thank you for liking the idea of incorporating all game settings during training. Due to our flexible architecture, our agent is the first that can train across all Hanabi game settings (referred to as R3D2-M in the paper), making it the first generalist Hanabi agent.

We thank you for your concerns, as they have allowed us to further improve the paper. We also address them below.

This paper only focuses on the Hanabi game. Whether the proposed algorithm can be scaled to other domains or environments is a critical question.

Thank you for this remark. This is a point that has been raised by multiple reviewers, for which we answer in detail in the general comment [Why we focused on Hanabi?]. We also provided additional comments in the general response on [Is R3D2 architecture task agnostic?].

The claimed first contribution of "framing Hanabi as a text-based game" appears weak. Previous work [1] has already devised text-based observations and actions for the Hanabi game

Hu & Sadigh (2023) indeed use text in the context of the Hanabi game. However, they use it in a completely different manner than we do. The agent’s observation and action remains vectorized, when interacting with the environment and other agents. So text is not used as an alternative representation for a Hanabi-agent. However, the action type is converted into language description to provide better context for GPT-3.5 prompting, which then provides a prior distribution over actions to influence the agent. This allows the agent to play according to some desired behavior.

We have updated the related work section (line 122-124) accordingly for better clarity and to distinguish from previous work.

The explanation of how R3D2 can support environments with varying numbers of players is insufficient.

Changing the number of players impacts both the observation-space, and the action-space. This is incompatible with typical value-network architectures (e.g., feedforward networks, or convolutional networks) which require a fixed input size (for observations) and fixed output size (for actions). By using language models to process the network’s inputs, we can process a variable number of tokens, and thus have texts of arbitrary length (which contain information about an arbitrary number of players). Moreover, by using actions as inputs, we can condition the network on an arbitrary action, allowing us to cope with a variable number of players. Concerning environments other than Hanabi, we have provided a detailed answer in our general comment [Is R3D2 architecture task agnostic?]

open-source code will help other researchers reproduce and build upon this work.

We completely agree. We will release all the models and our codebase upon acceptance.

The authors state that they utilize GPT-4 and LLama-2 to develop action strategies and claim that the results "struggle with optimal planning" in Section 5. However, I could not find any related experimental results shown in the paper, including the appendices.

We apologize for this. We submitted the Appendix as a separate document in the supplementary material. We have now included the Appendix in the main submission itself.

For GPT-4 prompt details, please refer to the Appendix A.1. For LLaMa - LoRA details, please refer to the Appendix A.3.

When introducing the results about fine-tuning LLama-7B, the authors only conduct experiments on different data sizes and LoRA rank settings, without comparing the results to their proposed method. Therefore, how was the conclusion that "the model performs poorly" derived?

The purpose of Section 5 is unclear. If the authors want to justify choosing BERT instead of other language models, they should replace the BERT structure with other LMs in their approach and show the experimental results in an ablation study, rather than listing results that only depend on LMs.

The objective of our LLaMA-7B and GPT-4 experiments is to evaluate how effectively LLMs perform on Hanabi, either out of the box or with minimal adaptation. Our findings demonstrate that LLMs in their current form are insufficient for playing Hanabi effectively. This aligns with observations by Hu et al. (ICML 2023), who note that LLMs struggle with the game's complexity. We agree that we also used the results of these experiments to make an informed initial choice about which type of LM to use, rather than a complete ablative study. Making such an ablative study would have been prohibitive for us in terms of computational and time costs. We would like to emphasize that LMs are not the core aspect of our methodology, and the fact that R3D2 is capable of transfer, despite using a small 4.1M-parameter TinyBERT model shows the soundness of our approach.

can a better presentation be provided regarding the role of language and Theory of Mind for making the imperfect information portion of Hanabi into perfect information portion.

Thank you for your thoughtful and constructive review. We appreciate your acknowledgment of the strengths of our work, including its focus on cooperation with unknown agents.

Your detailed feedback highlights areas where our presentation can be improved, particularly regarding the role of Theory of Mind (ToM) and language in reducing imperfect information to perfect information. We would like to clarify that our work does not aim to address ToM reasoning or solve ToM-related challenges. We explicitly cite Bard et al. (2019), "The Hanabi Challenge: A New Frontier for AI Research," to acknowledge the relevance of ToM in the Hanabi domain. However, we do not claim to address or solve ToM in this paper (we provide an example of ToM in Hanabi at the end of our response).

Indeed, our work focuses on self-play. What is remarkable is that even through self-play, it is possible to learn policies that generalize across game-settings and algorithmic settings. We propose the first agent capable of playing all Hanabi settings simultaneously while also generalizing zero-shot to novel partners and game configurations. A key aspect is indeed language, but not large-scale language models. Our language model is a 4.1M-parameter TinyBERT model whose main purpose is to convert the textual representation to a relevant embedding. But this representation has a large impact on performance. We assess this through an additional ablation experiment (Section 6.2), in which we introduce a novel baseline, called R2D2-text, whose sole difference with R2D2 is the use of textual observations. These experiments show that text improves generalization, but handling dynamic action spaces is critical for successful transfer to novel partners. As to why text is a representation that helps this generalization, it is because language is compositional, and implicitly models symmetries, making our agents robust to strategies that are similar, but vary in e.g., the color of the encoded card, the number of cards one has in front of oneself, or the color another player gives as hint.

Our architecture leads to 2 setups:

1. [Zero-shot Transfer] R3D2-Single Agent (R3D2-S), where we show that an R3D2-S agent trained in a 2-player setting can play in a 3-player setting reasonably well without additional training or architectural modification (Figure 6).
2. [Multi-task MARL] R3D2-Multi Agent, (R3D2-M) which makes it possible to train one single model to play all the Hanabi games simultaneously using the same amount of training budget. With the baselines, it is standard to train individual models to play in different settings, which doesn’t resonate well with real-world scenarios. For example, a human player can play 2,3, 4,5 games.

We hope to have provided clarifications on the goal of the work, language and theory of mind in imperfect information setup. If our clarifications resolve your concerns, we kindly invite you to reconsider your score. For further questions or clarifications, we are happy to address them.

Thank you for your insightful comments on our paper, and for recognizing one of the main strengths of our paper: that R3D2 shows adaptability across varying scenarios without requiring additional retraining, which addresses a significant limitation in existing multi-agent RL models.

Thank you also for raising valid concerns and potential limitations of our work, which we are eager to address here.

An extension or discussion on the applicability limits of textual representation in various domains would provide valuable context.

We acknowledge that the applicability of text representation to some domains might not be as straightforward as Hanabi. We address this more in detail in the general comment [Is R3D2 architecture task agnostic?]. To summarize, we updated Section 7 to highlight this limitation, pointed to interesting avenues for future work, and provided examples where specialized tokenizers could be used to alleviate the limitations of text.

The self-play performance is relatively poor

We acknowledge your concern regarding R3D2’s performance and took it very seriously. To address this, we resumed training R3D2, which had previously undergone fewer training iterations due to computational constraints. As detailed in the general response [About R3D2’s performance], this additional training led to significant improvements in self-play (22.23 → 23.34) and intra-XP scores (19.66 → 23.23).

Could the authors clarify why the focus is primarily on self-play during the training phase?

Thank you for this question. As mentioned in lines 76-82 in the paper’s introduction section, we deliberately focused on self-play to demonstrate a key insight: the choice of representation can fundamentally impact the robustness of learned strategies, even with simple training approaches. While previous work like Hu et al. (ICML 2020) and Cui et al. (NeurIPS 2021) developed sophisticated methods to achieve zero-shot coordination, we show that self-play - typically criticized for learning brittle, specialized conventions - can actually learn robust, transferable strategies when combined with appropriate representations. Our results demonstrate that by using text-based representation, self-play agents can achieve strong cross-play performance without requiring complex training methods. This is significant because self-play is simpler, more scalable, and computationally efficient compared to methods that require population-based training or hierarchical policies. Moreover, our approach is orthogonal to existing advances in zero-shot coordination - methods like Other-Play, trajectory diversity, or Off-belief Learning could potentially be combined with our text-based representation to achieve even stronger generalization performance. This finding opens up new research directions for improving multi-agent coordination by focusing on representation learning alongside algorithmic innovations.

Were alternative architectures considered?

Thank you for pointing us to (Ma et al., 2023). We are using an architecture akin to Ma et al.’s Self-Attention with Action as Inputs approach (Fig3-bottom), while our baselines are based on the Fig3-top (R2D2). Fig3-bottom is the best performing combination observed by Ma et al. (2023). With DRRN, this further confirms our architectural choice. We also added this discussion in Section 4.2 of our paper’s revision.

Furthermore, we performed a number of ablations and experiments on the integration of different language model architectures (Section 5, Appendix A.4), which we detail in the general comment on [Role of language and language model].

To conclude, we are grateful for your thoughtful comments and insightful questions, which helped us further refine our work and address key aspects of R3D2’s contributions and limitations. We hope these updates effectively address your concerns and demonstrate the significance of our approach. If our revisions meet your expectations, we kindly request you to reconsider the score you have assigned. Should you have further questions or require additional clarifications, we are more than happy to provide them.

We would like to thank you for your extensive and in-depth review of our work. We appreciate you find our focus addresses a critical gap in MARL research and that, within Hanabi, we address this gap by achieving effective transfer of strategies across different game settings.

We would like to address your comments as follows:

in what ways does Hanabi reflect the complexities of real-world collaborative tasks, as mentioned in the introduction? The paper does not sufficiently establish why Hanabi is an appropriate testbed for studying adaptability to new settings and collaborators

Thank you for this comment, which has been raised by other reviewers as well. We provide a detailed answer to these questions in the general comments [Why we focused on Hanabi?] and [Is R3D2 architecture task agnostic?]. To summarize, we focus on Hanabi mainly because:

Hanabi has been shown to correlate with improved human-AI coordination (Hu et al. (ICML 2020), Hu et al. (ICML 2023)) and demonstrated that strategies learned in Hanabi can transfer to human-AI collaboration scenarios. We can study Zero-shot adaptation to novel partners and transfer across player settings, where the goal is to have a team of agents that works well together (Cui et al. (NeurIPS 2022)) The Hanabi game incorporates key challenges found in real-world multi-agent scenarios: partial observability, communication constraints, and the need for long-term planning Bard et al. (2020).

While we demonstrate our approach on Hanabi, our core technical contributions - text-based representation for better transfer, architecture for dynamic action/state spaces, and variable-player learning - are domain-agnostic advances that could benefit MARL applications. Previous works like Foerster et al. (ICML 2019), Hu et al. (ICML 2020), Cui et al. (NeurIPS 2022), Hu et al. (ICML 2021) have successfully used Hanabi to develop fundamental MARL concepts that have influenced broader cooperative AI research.

While the authors cite work by Hu and Sadigh as justification for their choice of Hanabi and as a baseline for comparison, this prior research does not fully support the tasks explored here

We cite Hu & Sadigh (2023) primarily to acknowledge another approach that incorporates language models in Hanabi for improving cross-play performance, though in a different context (human-AI coordination). While their work includes human evaluation, they also report self-play and intra-algorithmic cross-play scores that provide relevant benchmarks. Although our frameworks address different aspects of coordination (human-AI vs zero-shot), including their work in our literature review provides important context about language-based approaches in Hanabi. However, we do not use their method as a baseline since their objective (adapting to human-specified strategies) differs fundamentally from ours (achieving zero-shot coordination through representation learning). We updated the related work section to make the distinction more clear.

adaptability may not be fully tested here since the text-based version removes many elements that would typically require generalization, making it a fundamentally easier learning problem.

Text is instrumental in enabling generalization and transfer, which is a central focus of our paper. Notably, we are the first to propose a learning agent capable of simultaneously playing across all Hanabi settings while generalizing zero-shot to novel partners and game configurations. This capability has not been demonstrated by any previous SOTA or baseline methods.

We also performed additional ablation studies, we created an intermediate baseline called R2D2-text (R2D2 backbone, recieving states as text) to understand the role of language. Our results demonstrate that merely converting states to text is insufficient for full generalization - the R3D2 architecture is crucial for adaptation to novel partners and game settings. This shows that while text representation simplifies some aspects of generalization, the combination of appropriate representation and architecture is necessary for robust performance. For reference, in section 6.2 and A.7 ,

The approach also necessitates adapting the replay buffer to handle a variable number of players, which could be a limiting factor.

We fully agree that, to handle a variable number of players, we had to pad sequences, a standard practice in deep learning. The original R2D2 implementation for Hanabi (Hu et al., 2021c) also pads episodes to length 80 since episode lengths naturally vary.

How scalable is the R3D2 approach with respect to increasing the number of players? Given that the current implementation handles a maximum of five players, are there foreseeable limitations if applied to environments with a larger number of agents?

The size of the state space increases slightly to capture the extra hand of the new players, and does not require any architectural changes. In our case, we use a BERT layer to process the textual observation. The maximum context length for BERT is 512 tokens. Our 2-player observation requires 96 tokens. This increases linearly with around 33 additional tokens per extra player, for a maximum of 12 additional players in Hanabi.

In general, our method is thus limited by the maximum context length. This could be increased with larger language models, although it would considerably increase training in terms of walltime. Still, the observation-size also increases for the bitstring encoding. Growing observation-size proportional to the number of agents is a general concern for MARL agents.

lines 422-425 presents an unclear and convoluted explanation of the comparison between methods

We apologize if our explanation seemed convoluted. What we wanted to say was that R3D2 has a better cross-play capacity than OBL since, regardless of which algorithm it is paired with, R3D2 has a higher score than OBL. That is, R3D2 and OP cooperate better together than OBL and OP. This is also true for R3D2 and IQL compared to OBL and IQL. We have rewritten these lines in the updated version of the paper for better understanding and clarity (line 495-500).

Given that R3D2 was trained across all game environments, it is unsurprising that it performs adequately across settings.

We respectfully disagree. Developing an architecture that can handle both dynamic action and state spaces requires careful consideration of representation learning and architectural choices. Our work is the first to demonstrate that a single model can successfully master all Hanabi game settings, a non-trivial technical achievement that opens new possibilities for flexible multi-agent systems. Moreover, We have 2 setups in R3D2,

• [Zero-shot Transfer] R3D2-Single setting (R3D2-S), can handle dynamic action and state spaces which was not possible in the baseline approaches. We show that an R3D2-S agent trained in a 2-player setting can play in a 3,4,5-player setting reasonably well without additional training or architectural modification (Figure 4).
• [Multi-Task MARL] R3D2-Multi setting, (R3D2-M) makes it possible to train one single model to play all the Hanabi games simultaneously using the same amount of training budget. With the baselines, it is standard to train individual models to play in different settings, which doesn’t resonate well with real-world scenarios. For example, a human player can play 2, 3, 4, 5 player games. R3D2-M is the first generalist Hanabi agent, since it can play on all game settings at the same time, but most of our cross-play results focus on R3D2-S.

More detailed comments on the concern on performance are addressed in the general response section [About R3D2’s performance]

The authors make several broad claims in the introduction about multi-agent reinforcement learning (MARL) and the adaptability of agents, but these claims lack supporting evidence or citations We grounded our statements by adding supporting citations in our updated version of the paper.

How does the proposed approach relate to the existing literature on Unsupervised Environment Design

We thank the reviewer for this question. In our work, we proposed the first agent capable of playing all Hanabi settings simultaneously while also generalizing zero-shot to novel partners and game configurations. We have expanded our related work section to better position our approach in relation to the UED literature. While works like Dennis et al. (2020) and Samvelyan et al. (2023) tackle transfer learning by automatically generating training environments of increasing complexity, our approach takes a fundamentally different direction. We reformulate the original environment using language as a unified representation that enables generalization across different game configurations. The key distinction is that UED approaches seek to create diverse training scenarios to help agents develop robust policies, while we maintain the original environment but change how agents perceive and interact with it through text-based representations. Our results suggest that appropriate representation choice can achieve strong generalization without requiring environment modification, though combining both approaches would be a really interesting avenue for future work.

Could the authors clarify if this refers to the player’s hand as revealed by others through clues up to the current time point?

Yes, exactly!

Thank you for your insightful comments and feedback. We are glad that you found our paper well written, and that you found the experiments thorough. Showing that text only is not sufficient to learn to cooperate is an important aspect of our work, which we appreciate you noted down.

You made some very valid points in the review, which we aim to address appropriately below.

work is Hanabi-centric

We address this question in a general response [Why we focused on Hanabi?], but to summarize, we indeed focused on Hanabi because:

1. Hanabi has been shown to correlate with improved human-AI coordination (Hu et al. (ICML 2020), Hu et al. (ICML 2023)) and demonstrated that strategies learned in Hanabi can transfer to human-AI collaboration scenarios.
2. We can study Zero-shot adaptation to novel partners and transfer across player settings, where the goal is to have a team of agents that works well together (Cui et al. (NeurIPS 2022))
3. Finally, the Hanabi game incorporates key challenges found in real-world multi-agent scenarios: partial observability, communication constraints, and the need for long-term planning (Bard et al. (2020)).

R3D2 scores lower on Self-play and intra-XP but higher on inter-XP. The grid in Fig 5, also does not have massive increase in scores against R3D3. Is a jump of 3-5 points a big jump in the game of Hanabi, regarding Table 5?

We understand your concern about R3D2’s performance, and took this comment very seriously. We resumed training of R3D2, which had less training iterations due to computational constraints. We expand upon this in the general response [About R3D2’s performance] but, concretely, this resulted in a significant improvement on selfplay (22.23 -> 23.34) and intra-XP scores (19.66 -> 23.23).

A 3-5 point improvement is indeed significant in Hanabi. The game has a maximum possible score of 25 points, and improvements of even 1-2 points at higher performance levels are considered meaningful in the literature. For context, prior works like Foerster et al. (ICML 2019), Hu et al. (ICML 2020) report improvements in the range of 1-3 points as substantial advances. Additionally, the difficulty of improving scores increases non-linearly as performance approaches the ceiling, making 3-5 point gains even more notable.

“All baselines use R2D2 as a basis” vs “we utilized OP and OBL checkpoints from the original paper which focused exclusively on the 2-player setting”. Were the original checkpoints foundationally R2D2?

Yes, all the baselines are foundationally R2D2.

To conclude, we sincerely thank you for your valuable feedback and insightful comments. We have taken your concerns seriously and have worked diligently to address them through detailed explanations and additional training of R3D2, leading to significant improvements in performance. If you find that our revisions and explanations have sufficiently resolved your concerns, we kindly request your consideration in revisiting the score you have assigned. Should you have any further questions or need additional clarifications, we would be more than happy to address them.

Thank you for your feedback on our submission. We appreciate that you find the experiments extensive. Proposing inter-algorithmic and inter-setting cross-play is a key contribution of our work, which we are glad to have been highlighted.

We have carefully considered the concerns and would like to address them as follows:

text template is not clear enough. What information is included in the text template? Will the text be revealed to other agents? It seems that the template will record all "previous" information which is simply the history of the observations. Then why not use history instead of text recording?

The template contains the current game state information - including tokens, played cards, visible hands, discards, and hints - as shown in Figure 1. Each agent only receives the information it would normally observe in the game, simply formatted as text. We choose text representation for both states and actions because it provides a more consistent state and action space across different player counts and enables transfer, as explained in lines. However, as is the case for R2D2, R3D2 uses RNNs to estimate the history of state-action values. The only difference is that, for R3D2, the state- and action-embeddings that are passed to the RNN result from TinyBERT layers.

We updated the figure caption in the revision to be more descriptive to the readers.

it does not compare to R2D2. Moreover, since R2D2 is not introduced in detail, it is hard to perceive the technical contribution of R3D2.

In this work, R2D2 is basically Independent Q-Learning (IQL) (Tan, 1993; Tampuu et al., 2017). IQL is still frequently used as a baseline for Hanabi, as it serves as the foundation of many state-of-the-art MARL Hanabi algorithms. It shows strong performance with its training partners, having learned highly specialized conventions through self-play. R2D2 are still independent learners, but introduce the following changes to improve the learning process:

• R2D2 uses RNNs to cope with partial observability settings,
• R2D2 collects experiences from multiple environments in parallel, speeding up global walltime,
• R2D2 includes many best practices of DQN, namely double DQN, a dueling architecture, and prioritized experience replay. We hope these explanations, which are also explained on lines 208 - 212 in the paper, clarify the R2D2 agents.

Moreover, to better understand the role of R3D2’s different components, we ran additional ablation experiments (Section 6.2 and A.7).

intuition behind multiplying the state and action embeddings? Where do you integrate the text information? how do you make sure it can accommodate different numbers of agents and task settings?

The choice of elementwise multiplication for combining state and action embeddings is well-motivated by prior work. Our approach builds directly on the Deep Reinforcement Relevance Network (DRRN) architecture (He et. al, ACL 2019), which demonstrated the effectiveness of this operation for text-based games. This multiplicative interaction has been widely adopted in both deep learning and reinforcement learning as an effective way to condition outputs on contextual inputs, as shown in various domains including audio synthesis (Oord et. al, 2016), visual reasoning (Perez et. al, 2017), and sequential decision-making (Kumar et. al, 2019). Text information is integrated through BERT layers that encode both observations and actions into embeddings (shown in Figure 2). These text encoders transform the game state and action descriptions into fixed-size vectors that capture the semantic meaning while being agnostic to the specific number of players.

The architecture accommodates different numbers of agents and settings because:

• The text template naturally extends to include additional players without structural changes
• The action space remains consistent since all actions are encoded as text
• We use zero-padding in the replay buffer to handle varying sequence lengths.

if the text is embedded and used by the Q-functions, does it mean the text is better than observation features? Is there any intuition behind this?

This is an interesting question. Text is not better than observation features per se, as R2D2, which uses a vectorized representation of the environment, also reaches a high self-play score. However, we argue that a textual representation, combined with a general-purpose language model, is better for generalization and transfer learning (Radford et al., 2019b; Brown et al., 2020). To better understand the role of text, we ran more ablation experiments which show that, while a text representation provides some benefits for generalization, handling dynamic action spaces is equally crucial for successful transfer to novel partners. Please refer to [Role of language and language models] for the role of language and language models in learning generalizable policies.

[2. About the focus on Hanabi]

Hanabi has emerged as a popular benchmark for studying collaborative AI systems, as highlighted by Bard et al. (2020) who established it as a new frontier for AI research. The game incorporates key challenges found in real-world multi-agent scenarios: partial observability, communication constraints, theory of mind reasoning, and the need for long-term planning. These properties make it particularly relevant for studying adaptability to new collaborators and settings. Importantly, success in Hanabi has been shown to correlate with improved human-AI coordination (Hu et al. (ICML 2020), Hu et al. (ICML 2023)) demonstrated that strategies learned in Hanabi can transfer to human-AI collaboration scenarios.

While we demonstrate our approach on Hanabi, our core technical contributions - text-based representation for better transfer, architecture for dynamic action/state spaces, and variable-player learning - are domain-agnostic advances that could benefit MARL applications. Previous works like Foerster et al. (ICML 2019), Hu et al. (ICML 2020), Cui et al. (NeurIPS 2022), Hu et al. (ICML 2021) have successfully used Hanabi to develop fundamental MARL concepts that have influenced broader cooperative AI research.

We clarified the importance of Hanabi as a benchmark in the revision of the paper (Section 1).

---

[3. Is the R3D2 architecture task agnostic?]

We convert the textual information to tokens and compute the embedding using language models, where the embeddings capture the semantic meaning while being agnostic to the task. (Radford et al., 2019b; Brown et al., 2020). Then, we multiply state and action embeddings of language model inspired by the existing literature Deep Reinforcement Relevance Network (DRRN, He et al., ACL 2019) architecture, which is a popular baseline till now for most of the text-based games. This operation allows us to capture the relevance between states and actions in a way that generalizes across different game settings since both are encoded in the same semantic space through the language model.

Given we are using embedding and language models in our architecture, it is robust and easily adaptable to other tasks involving texts. However, we acknowledge that the applicability of text representation to some domains might not be as straightforward as Hanabi. For example, environments like continuous control or vision-based tasks might require domain-specific adaptations (we updated Section 7 to highlight this limitation). We acknowledge this is a limitation of our work, however, recent advances in language models are expanding the possibilities. For instance, Llama-2's specialized tokenizer demonstrates remarkable performance on numerical tasks by decomposing numbers into digit sequences (Touvron et. al, 2023), suggesting exciting opportunities for extending our approach to domains with structured numerical representations.

---

[4. Role of language and language model]

To understand the role of language representation we performed the following ablations:

1. What is the role of pre-trained weights? (Appendix A7.1)

• we compared the language model with pre-trained and random weights. Based on the results on five seeds, pre-trained weights significantly enhance sample efficiency. (Figure 12a for reference)

2. What is the role of updating the language model? (Appendix A7.1)

• We aim to understand the role of updating the language model by changing the frequency of updates. A clear trend emerges: reducing the update frequency drastically diminishes the model's potential to reach peak performance. (Figure 12b for reference )

3. Does the performance come from the language representation or the architecture?

• To isolate the contributions of our two key innovations - using language models for state representation and handling dynamic action spaces - we created an intermediate baseline called R2D2-text.
• Our experiments demonstrate that text representation alone provides limited benefits for generalization. We show that R3D2 outperforms R2D2-text, demonstrating that the dynamic network structure is essential for successful transfer to novel partners.. The superior performance of R3D2 compared to R2D2 and R2D2-text in both across game-setting transfer (updated the results of 6.2) and cross-play (Added section A7.2)