Ad-Hoc Human-AI Coordination Challenge

Rebuttal

Thank you for your thoughtful review. We appreciate your commendation of our efforts to maintain fairness by ensuring evaluation partners are not public or solely determined by leaderboard rankings. Please find our response below:

Weaknesses 1. and 2.

Please see the general response.

Weakness 3: Hanabi is a Great Environment but Not Sufficient: While Hanabi is an excellent environment, it alone is insufficient to represent the diversity of scenarios involved in ad hoc teamwork. Ad hoc teamwork spans multiple scenarios, and using only Hanabi as the environment is not enough. The authors should consider integrating additional cooperative environments, such as Overcooked or Google Research Football, to enhance the benchmark.

While we acknowledge that ad-hoc teamwork spans diverse scenarios, many impactful studies focus on a single benchmark to provide insights. We chose Hanabi because it offers a complex, partially observable cooperative setting that's well-suited for studying ad-hoc teamwork.

Regarding Overcooked, it's a fully observable environment, which doesn't align with our focus on partial observability challenges. As for Google Research Football, we currently lack the necessary human data to train human proxy agents, making it impractical for our benchmark at this time.

Weakness 4: Discussion of Benchmark Results: The authors only demonstrate through experiments that the constructed human proxy agents can play Hanabi with different partner agents. As a benchmark paper, the authors should further analyze the results to help the community understand what new insights this Hanabi-based benchmark brings to the problem of ad hoc teamwork. Currently, the discussion only demonstrates that the proposed approach can compute benchmark results. I do not see further insights or novel contributions from the authors regarding this issue.

Our main contribution is to make human-AI coordination evaluation more accessible by approximating evaluations with humans at a lower cost using human proxy agents. This approach brings human-AI coordination studies closer to real-world scenarios without the logistical challenges of involving human participants in every evaluation.

If there are specific analyses or further insights you believe would provide additional value, please let us know.

Question 1: How can the diversity of the generated human proxy agents be ensured?

See the general response.

Question 2: I don’t fully understand how cross-play can verify that the human proxy behaves similarly to humans. A score alone cannot adequately demonstrate behavioral similarity. I hope the authors can provide further explanation.

We agree that achieving similar scores in cross-play is not sufficient on its own to verify that the human proxy behaves humans. To provide a more comprehensive demonstration of behavioural similarity, we have conducted additional analyses beyond score comparisons. We have included the cross-entropy loss for action prediction which measures how well the human proxy can predict the actions of real human players based on game states. We show that human proxies have cross-entropy scores close to BC policies that we know follow human-like strategies, given they are trained on human data only.

Additionally, we have analysed behavioural metrics in the updated version of the paper, following the approach proposed by Canaan et al. (2020). Our results show that these metrics are nearly identical between the human proxy agents and the human gameplay data.

By combining cross-play evaluations with these analyses, we provide evidence that our human proxies not only achieve have strong performance but also exhibit human-like behaviour.

Finally, we would like to clarify that our methodology for creating human proxy agents is not novel but builds upon established techniques from previous research. Specifically, we utilise BC combined with regularised RL, methodologies that have been extensively explored in prior works such as Cornelisse et al. (2024), Jacob et al. (2022), and Bakhtin et al. (2021).

Conclusion

Thank you again for your thorough review. We hope that the reviewer feels we have addressed their questions and welcome any further discussion. We also ask that, if all their concerns are met, the reviewer consider increasing their support for our paper.

Canaan et al. (2020). Generating and Adapting to Diverse Ad-Hoc Cooperation Agents in Hanabi. ArXiv. https://doi.org/10.1109/TG.2022.3169168

Cornelisse et al. (2024). Human-compatible driving partners through data-regularized self-play reinforcement learning. ArXiv. https://arxiv.org/abs/2403.19648

Jacob et al. (2021). Modeling Strong and Human-Like Gameplay with KL-Regularized Search. ArXiv. https://arxiv.org/abs/2112.07544

Bakhtin, et al. (2022). Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning. ArXiv. https://arxiv.org/abs/2210.05492

Rebuttal

Thank you for your thorough and constructive review. We deeply appreciate your recognition of our work's originality in addressing human-AI coordination, the quality of our dataset and engineering efforts, the clarity of our presentation, and the significance of AH2AC2 as a practical testbed for the field. Please find our responses below.

1.

Please see the general response. We clarify and provide additional metrics that are now available in the appendix.

2. Incomplete Introduction to the Challenge: The challenge setup is not thoroughly introduced, leaving the structure and implementation of the leaderboard unclear. Without this information, it is difficult to assess the challenge's usability and rigour. The lack of leaderboard details also raises concerns that the development of the challenge may be incomplete.

Thank you for highlighting the need for more clarity on the challenge setup. In response, we have added additional details about the challenge and its structure to the appendix. Our supplementary material includes the code used for evaluating the baselines, along with the open-sourced weights for these models.

While our original plan was to make the leaderboard public upon the official release of the challenge, we understand your concerns about its availability. To address this, we have decided to release the leaderboard now, accessible at ah2ac2.com. We hope this helps reduce any uncertainties and brings us a step closer to a full public release and access to human proxies.

3. Figure 3 is confusing, especially in the two-player setting, where the performance of all compositions appears similar. This could be due to multiple reasons—such as a highly effective dataset that results in nearly perfect models even using BC. However, this raises questions about the benchmark's ability to differentiate between varying model qualities. Furthermore, it is unclear what the three-player setting represents and how it differs from the two-player configuration.

Thank you for your comments regarding Figure 3. We understand that the performance similarities in the two-player setting may appear confusing. In this setting, Behavioral Cloning (BC) agents achieve decent scores (18.95 and 19.66 in self-play evaluation). However, our human-proxy agents significantly improve these results to 22.56 and 23.02, which is a substantial enhancement. It's important to note that in Hanabi, moving from scores below 20 to above 22 is quite challenging; achieving decent scores is relatively straightforward, but attaining near-perfect scores requires sophisticated coordination strategies and very little mistakes.

Additionally, as shown in Table 2, BC agents perform poorly out of distribution, often resulting in failed games. In contrast, human-proxy agents maintain robust performance. The difference between the human-proxy and BC agents is even more pronounced in the three-player setting, highlighting the advantages of our method.

Regarding the three-player setting, it involves three players participating in the game, unlike the two-player configuration. We included both configurations in our evaluation to provide a comprehensive benchmark. The three-player setup is indeed more difficult and is often overlooked in previous works on Hanabi, whether in zero-shot coordination (ZSC) or ad-hoc settings. By addressing both, we aim to cover a broader spectrum of gameplay scenarios.

We appreciate your attention to these details and your valuable feedback. If any part of our explanation remains unclear, please let us know, and we would be happy to provide further clarification.

4.

Please see the general response. We provide a discussion on the topic and add an additional baseline.

Conclusion

We sincerely thank you for your detailed review and the recognition of our paper's strengths. We hope our responses have adequately addressed your concerns. We believe that with these clarifications and improvements, our work aligns well with the positive aspects you've highlighted in terms of originality, quality, clarity, and significance.

Dear Reviewer c4nE, Could you please read the authors' rebuttal and give them feedback at your earliest convenience? Thanks. AC

Thank you for your response. I will increase my score to 5 to acknowledge your effort. However, I still believe the paper is below the acceptance threshold (especially the novelty) and would benefit from further improvements in the future.

Rebuttal

Thank you for your thorough and insightful review. We are especially grateful for your positive feedback, noting that our paper is well-written and has the potential to make a significant contribution to the multi-agent ad hoc human-AI collaboration research. Your comments are greatly appreciated and motivate us to ensure clarity in addressing your concerns.

(1)

Please see the general response.

(2): As a human-ai benchmark, shouldn’t the qualitative/quantitative evaluation against real humans still be necessary for the public leaderboard? Then, you can compare the correlation between play with humans and play with human proxy. Otherwise, I believe the authors should spend more time justifying why human-proxy=human.

Thank you for raising this important point. While we discuss this issue in our general response, we'd like to provide additional thoughts here. We agree that evaluating agents against real humans would be ideal for a human-AI benchmark. However, conducting large-scale evaluations with human participants is often impractical due to logistical challenges and resource constraints.

To address this, we built human proxies using a methodology that has been extensively studied in prior research. It has been shown that regularised RL leads to policies that are compatible with the social conventions of the human reference group. Additionally, we have also conducted studies to validate that our human proxies exhibit human-like qualities found in the dataset.

Thank you again for your insightful feedback. We hope this explanation clarifies our approach and reassures you of the robustness of our human proxies in the benchmark.

(+1): How does the performance of BC/BC+RL trained on a small released subset compare to that of BC/BC+RL trained on the entire dataset, and what causes this difference? Is it due to the training not converging, or does it suggest a divergence in convention distributions between the two?

Thank you for a great question. We include the performance of BC (Behavioral Cloning) and BC+RL (Behavioral Cloning with Reinforcement Learning) when trained on a small released subset versus the entire dataset in our paper, specifically in Table 2 and Table 4.

In the two-player setting with a 1,000-game limit, BC achieves a mean self-play score of only 2.12. This low score suggests that BC struggles to learn how to navigate diverse game states from such a small dataset since the limited data doesn't cover a wide range of game states.

However, similar is observed even when BC is trained on a large-scale dataset, where the mean score is relatively high. BC agents seem to either achieve strong scores or "bomb-out" by losing all lives, a phenomenon we discuss in Section 5.1. In contrast, BC+RL agents show improved performance even when trained on the smaller subset. They don't exhibit the same tendency to lose all lives, suggesting that the RL component helps them adapt better to new situations. However, we observed that during the RL training loop, the score plateaus. We hypothesise that this plateau occurs because the KL-divergence regularisation term limits significant policy updates, preventing the agent from deviating too far from the initial BC policy.

In summary, BC agents trained with the full dataset generalise better and achieve higher scores. The small subset doesn't provide this richness, so agents trained solely on it can't match the performance of those trained on the entire dataset. We hope this clarifies the differences in performance and their causes. If you have further questions or need additional details, please let us know.

Conclusion

Thank you again for your thorough review. We hope that the reviewer feels we have addressed their questions and welcome any further discussion. We also ask that, if all their concerns are met, the reviewer consider increasing their support for our paper.

Dear Reviewer phEU, Could you please read the authors' rebuttal and give them feedback at your earliest convenience? Thanks. AC

Rebuttal

Thank you for your thorough and constructive review. We particularly appreciate your recognition of our work's value in addressing human-AI coordination evaluation, the potential of human proxies for ad-hoc teamwork, and the importance of our dataset contribution and limited-data challenge for real-world applications. Please find our response to your comments below:

1. The core idea of employing behavioral cloning to create human-like agents in Hanabi builds upon previous work (Carroll et al., 2019), lacking significant innovation.

There appears to be a misunderstanding about the methodology used for training human proxies. We do not use BC to create human proxies – instead, we employ regularised RL, an approach developed very recently for building human-like partners in partially observable environments (Cornelisse et al. 2024, Jacob et al. 2022, Bakhtin et al. 2021).

Also, our paper's primary contribution/innovation isn't in developing new methods for building human-like agents, but rather in delivering the following contributions to the field:

• We are the first to provide strong, validated and reproducible human proxies for standardised human-AI coordination evaluation.
  • This addresses a significant gap in existing work, which relies either on closed-source BC agents trained on private datasets or a very small pool of human players.
• We contribute the first-ever open-source dataset of human Hanabi gameplay.
  • We also use this dataset to formalise the human-AI coordination challenge with limited data.
• We establish the evaluation protocol for ad-hoc human-AI coordination in partially observable environments
  • This protocol ensures fair and reproducible evaluation across different approaches.

A summary of all key contributions can be found in the paper and given that these are acknowledged as strengths of our work in the review, we are surprised by the contribution score and would appreciate clarification.

2. and 3.

We address these issues in our global response.

4. A demo website or interactive platform for the challenge would enhance accessibility and engagement with the community. Is it possible to release all datasets? If not, why? Can you release the pipeline of the dataset preprocess?

Thank you for these constructive suggestions. A demo website is indeed an excellent idea. Currently, all open-sourced human games can be visualised using the JaxMARL package, providing immediate accessibility to researchers and challenge participants. We will look into integrating our agents into the hanab.live platform if there is interest.

Regarding demo access to human proxies and/or their training data, we've made careful design choices to maintain the integrity of AH2AC2. While we aim to release these resources in the future, providing open access now could compromise the fairness of the evaluation procedure. We have developed a server that provides controlled access to human proxies, ensuring standardised and reproducible evaluation for all participants.

On the data preprocessing pipeline, we want to clarify that our repository already includes all code for data loading, augmentation, and transformations – the same pipeline used to train our baseline agents and human proxies. The dataset we release undergoes minimal preprocessing, where we only remove corrupted games (unfinished or abandoned) and games that use Hanabi variants. We don't perform any additional filtering.

Conclusion

Thank you again for your thorough review. We hope that the reviewer feels we have addressed their questions and clarified our work and welcome any further discussion. We also ask that, if all their concerns are met, the reviewer consider increasing their support for our paper.

Canaan et al. (2020). Generating and Adapting to Diverse Ad-Hoc Cooperation Agents in Hanabi. ArXiv. https://doi.org/10.1109/TG.2022.3169168

Cornelisse et al. (2024). Human-compatible driving partners through data-regularized self-play reinforcement learning. ArXiv. https://arxiv.org/abs/2403.19648

Jacob et al. (2021). Modeling Strong and Human-Like Gameplay with KL-Regularized Search. ArXiv. https://arxiv.org/abs/2112.07544

Bakhtin, et al. (2022). Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planning. ArXiv. https://arxiv.org/abs/2210.05492

Dear Reviewer JsHP, Could you please read the authors' rebuttal and give them feedback at your earliest convenience? Thanks. AC

1.Diversity Problem

I disagree with the authors’ statement regarding diversity. If the paper focuses solely on a limited and narrow range of human behaviors, it will severely restrict the evaluation’s effectiveness and applicability. How should evaluations be conducted if future ad hoc algorithms are not optimized for the range of human behaviors considered in this work? Would the authors’ benchmark be inapplicable in such cases? I believe that an effective ad hoc evaluation should prioritize diversity to assess which behaviors, across the widest possible distribution, an agent can adapt to and cooperate with, rather than merely providing a specific and narrow behavior distribution.

2. Human Data

Additionally, please be cautious when using the phrase “the first standardized and cheap human proxies for evaluation.” To my knowledge, the use of human proxies in Overcooked has already become quite widespread. Reconstructing such proxies in Hanabi does not constitute the first instance of building human proxies in human-AI coordination. I am also puzzled by the authors’ claim that using human data to construct proxies is a “cheap” approach. Such a huge human dataset cannot be seen as a cheap solution. For example, the authors mention that the absence of human data in GRF prevents evaluation. If there were a method to construct evaluation partners without relying on real human trajectories, would that not fundamentally resolve this issue?

As I work on ad hoc coordination agents, I am pretty interested in seeing a benchmark that (1) captures the diversity in evaluating partners and (2) has a diagnostic metric to tell the weakness (e.g., which cluster of conventions that the ad hoc agent cannot cooperate well).

There are works that generate diverse evaluating teammates, like Adversity and BRDiv, but I think another way to show the diversity is to measure how diverse the evaluating agents are. If this work focuses more on human compatibility, then maybe it is out of my expertise.

If the work is posed as an Ad Hoc teamwork problem, regarding the diversity problem, I believe that RNN could consistently capture some human trajectories and conventions within the episode; do you have a way to qualitatively show some diversity that the proxy captures human behaviors that belong to H-convention and human behaviors that do not belong to H-convention?

@Reviewer 4ENk, what do you think that the authors can do to show the spectrum of the behaviors?

Without providing relevant baseline implementations and sufficiently diverse evaluation partners, it cannot be claimed that the barrier to entry has been lowered.