Ad-Hoc Human-AI Coordination Challenge

Dear reviewer,

Thank you for the detailed and constructive feedback. We appreciate the opportunity to address your points and clarify aspects of our work. Please find our answers and improvements below.

Addressing Reviewer Questions

Q1. Human Validation

We agree that testing human proxies with real humans would be ideal for a human-AI benchmark, but large-scale evaluations are impractical due to logistical challenges and resource constraints. Therefore, we created these proxies using a method studied extensively in prior research, and our experiments and tests confirm they exhibit human-like qualities and behave as expected. Overall, our validation shows that the proxies capture key aspects of human play, making them suitable and robust partners for the benchmark.

Q2. Three-Player Data

We thank the reviewer for this excellent question! We collected data for two-, three-, four-, and five-player games; our analysis showed that proxies for four- and five-player settings gave less convincing results due to data sparsity. In contrast, the three-player setting, with 46k games, showed strong performance (comparable to the two-player setting), and tests on data subsets revealed diminishing returns with more data. This evidence suggests that the 46k games available for the three-player setting are sufficient to train robust, human-representative proxies, making it a valuable and worthwhile addition to the benchmark.

Q3. LLM Integration

Thank you for your suggestion. To strengthen our benchmark and given the popularity of LLMs, we have implemented an LLM-based agent using o3-mini and are currently benchmarking it with AH2AC2. So far, we're getting scores ranging from 14 to 20 out of 25 (along with some games where all lives are lost), and we'll include these results in the final paper copy, along with additional experiments and results. We believe this is a strong addition to our paper and hope the reviewer will recognise the effort and resources we invested into benchmarking LLMs.

Q4. Benchmark Maintenance

We are committed to the long-term success of the AH2AC2 benchmark. We will maintain and update the evaluation website, submission API, and leaderboard, offer ongoing support, and expand the benchmark, potentially adding four and five-player settings and other Hanabi variants as we collect more data and different techniques emerge.

Addressing Other Comments

Fixed proxies and dynamic behavior: Thank you for raising this point. Although our agents’ parameters are fixed after training, their in-game behavior is dynamic – fixed parameters do not imply fixed behaviour. Due to space constraints, please see our response to reviewers CEGr and FDiM. Action prediction results (Tables 3 & 4) confirm high accuracy/low loss on unseen human data, indicating they capture diverse strategies and adapt in-game.

Challenge utility: While we agree that human-AI play is ideal and we acknowledge this limitation, large-scale human testing is often impractical and hard to reproduce. Our approach offers a practical alternative: we create robust, reproducible proxy agents trained on extensive real human gameplay data using SOTA methods for developing human-like policies.

API-hosted proxies: Thank you for this comment. We believe AH2AC2 improves on previous methods that used completely inaccessible proxies and closed datasets. Our API provides open, but controlled, pre-registered access to human proxies, ensuring fair, reproducible evaluation.

Behavioral metrics: Thank you for your suggestions regarding the breadth of our behavioural evaluation. Our experiments show that BC agents trained on the entire dataset achieve better coordination when paired with our human proxies (Figure 2), proving that our proxies adapt and recover from errors even with fixed BC agents. Low IPP means hints are effective (they carry implicit info that players must interpret), and our action prediction results (Tables 3 & 4) confirm the proxies learn a wide range of implicit conventions.

We couldn’t find standardised definitions for the extra metrics like hint utility or convention compliance in prior Hanabi works, so we'd be grateful if you could point us to any. We believe our current analysis provides substantial evidence of human-like behavior, and we're open to expanding our evaluation with suitable metrics.

Ablation study: We appreciate the time you invested in reading the appendix. We completely agree that it should be present in the main text. We will integrate key findings into the main text in the camera-ready version, using an extra page allowance.

---

We hope these clarifications and planned revisions address your concerns and strengthen our paper. Thank you for your valuable feedback. if you feel our responses have sufficiently resolved your concerns, we kindly ask you to consider updating your score accordingly.

Dear reviewer,

Thank you for your positive feedback and detailed review of our paper. We especially appreciate you taking the time to read the appendices. We hope the following answers clarify the points you raised.

Question 1

We thank the reviewer for raising this point, and we agree with the point that human play is diverse. Regarding the H-Group Conventions mentioned (line 812), we want to clarify what this means in practice and how it relates to strategic variety.

While our dataset mainly features games where H-Group Conventions are used, this term doesn't refer to a single strategy. Instead, it is helpful to think of H-Group Conventions as a collection of different strategies and techniques that players learn and combine. In fact, conventions are sorted into levels. Each level uses a different set of strategies that are used. Mastering even beginner H-Group levels requires learning several of these. Players often mix and adapt these strategies within a single game – depending on the player's strength. Therefore, the dataset itself naturally contains a variety of playstyles. Our action prediction results (Table 3 & Table 4) show that the trained proxies achieve good accuracy and low cross-entropy loss on unseen human data. This suggests that they have successfully learned to represent multiple strategies and patterns present in the human dataset.

We hope this explanation clarifies that the use of H-Group Conventions as a foundation does not overly constrain the strategic variety present in our data, and consequently, in what our proxy models represent. Finally, as a similar concern was raised by multiple reviewers, we will clarify these points in our camera-ready submission.

Question 2

Adaptation is a crucial aspect to consider when developing human proxies, and we thank the reviewer for raising this point. To clarify how our proxies address adaptation, we believe it is helpful to consider two adaptation types:

• Adaptation within a game: This refers to adapting actions based on the current game state, partner's moves, and available information within a single game. Our human proxies exhibit this type of adaptation. While the parameters of the neural network policy are fixed after training, behaviour adapts dynamically, much like a human player reacts to changing circumstances. When the proxy agents see an unexpected action or observation, then from that action-observation history onward, they will account for the fact that the other agent uses a different convention. Thus a fixed policy does not imply fixed behaviour.
• Adaptation between games/learning new strategies over time: This means changing fundamental strategies or learning entirely new conventions. Our proxy models are not designed to do this. We want them to consistently represent the playstyle of the population found in our dataset, providing reproducible and consistent partners for evaluation.

Furthermore, our experiments provide evidence for the proxy's within-game adaptability. As shown in Figure 2, BC agents (which can be quite rigid and brittle) perform better when paired with our human proxies compared to SP. This suggests that our proxies are flexible enough within a game to coordinate effectively even with simpler partners and can adjust to mistakes made by these less sophisticated policies.

We hope this clarifies the specific type of adaptation our human proxy models capture and why they are designed this way for consistent evaluation.

Question 3

We are happy to emphasize why we choose our metrics and how we intend them to be interpreted.

• IPP (Information per Play) is normalised to a scale of 0 to 1. A value of 0.44 means players, on average, know slightly less than one “information” (color or rank) about the cards they play.
• Communicativeness is defined as a percentage of turns where a player gives a hint, when it's possible to give a hint.

The fact that the values for these behavioural metrics are close between different models suggests that they exhibit similar tendencies regarding information usage, hint efficiency, and communication frequency. We aimed for these metrics to provide a clear comparison.

Perhaps we misunderstood your concern regarding normalisation? If you could elaborate on what aspect feels unnormalized or difficult to interpret, we would be happy to provide further clarification.

---

We hope these explanations address your questions. Thank you again for your valuable insights. If our responses have resolved your concerns, we would appreciate it if you would consider increasing the support for our work.

Dear Reviewer,

Thank you for your thorough review and constructive feedback. We have carefully considered your questions and provided our responses below.

Q1. Benchmark Fairness

We thank the reviewer for raising this point and agree that it is theoretically possible for researchers to attempt scraping game data or collecting new human data independently, and believe this is a valid concern.

To clarify, the core goal of AH2AC2 is to measure how well methods perform when trained only on a limited dataset we provide, and using scraped data would not be fair or allowed. Having in mind that scraping is theoretically possible, as correctly noted by the reviewer, we rely on community transparency. We will strongly encourage participants to make their training code publicly available and reproducible. Also, submissions on the leaderboard will indicate whether reproducible code has been provided (and link to the codebase will be added in this case). We will update our submission system to account for this! We hope this helps reduce reviewers' concerns. Finally, while possible, scraping game data is non-trivial, and crucially, there is currently no open-source dataset of human play available.

We should also note that past benchmarks (such as Overcooked AI) relied solely on trust. We believe our approach offers a more rigorous and fair evaluation protocol – we aimed to reduce the chance of overfitting, but we can't exclude the possibility of unethical practices completely.

We hope this addresses your concerns regarding data usage and the fairness of the benchmark.

---

Q2. Human Proxies and Benchmark Goals

We thank the reviewer for raising this important point. We will update the paper to clarify both the specifics of our agents and the goals of AH2AC2. In particular:

1. Number of Agents: We developed four human proxy agents (two for the 2-player setting and two for the 3-player setting), not a single one. We will make this clearer in the paper.
2. Diversity of Playstyles: You are right that real-world human play varies significantly. Our agents were trained on a large dataset of games acquired from the hanab.live community. Players on this platform generally follow H-group conventions, but this does not mean they follow a single strategy. Instead, H-group conventions are a collection of diverse strategies and techniques. Players mix and match these based on their skill and the game context, leading to significant variation in play style within the dataset our agents learned from. We agree this is not clear in our original paper, and we will update the paper to provide more details.

Finally, we would like to clarify our goal for the benchmark. Our approach is pragmatic and mirrors a real-world use case:

• There is a given (inherently meaningful) human population that an algorithm is supposed to cooperate with.
• We assume it is possible to collect a small dataset from this population (as is often the case).
• The task is then to develop a method that can do well with the source population while having access to only this small dataset.

Success on our benchmark indicates an algorithm's ability to effectively learn cooperative strategies from limited, representative human data, which we believe is a key step towards building human-compatible agents/systems.

We hope this explanation clarifies the nature of our human proxy agents and the practical relevance of our benchmark.

---

Q3. 3P OBL Weights

We thank the reviewer for reading our paper in detail and raising this point. We contacted several authors of the original OBL paper, and they informed us that they unfortunately no longer have access to the 3P OBL weights and the weights were not released with their original work. Re-implementing OBL in JAX is a complex task, and it is beyond the scope of our current work.

---

Q4. Multi-task Learning Agent

We definitely agree that lifelong learning and multi-task learning algorithms could be highly competitive in AH2AC2. However, given the substantial complexity involved in fully implementing and tuning such an agent, we consider this outside the immediate scope of our benchmark. That said, we encourage and welcome the broader research community to benchmark such methods within AH2AC2, as these promising techniques may significantly enhance human-AI coordination.

---

Q5. Seeds

This is correct.

---

Regarding https://openreview.net/forum?id=pCj2sLNoJq: We thank the reviewer for highlighting this very relevant ICLR paper. We agree that this recently published method directly addresses one of our proposed future works and represents a valuable addition to our study. We plan to add this approach to our benchmark as soon as possible.

---

We hope we have addressed the reviewer's comments and concerns. We are happy to discuss any of these points further. If our response has resolved your main concerns, we would be grateful if you would consider updating your score.

Dear Reviewer,

Thank you for your feedback. We appreciate the opportunity to address your concerns and clarify the contributions of our work.

Adaptivity of Human-Proxy Agents

You raised an important concern about whether using human proxies, trained using fixed parameters, sufficiently captures the dynamism necessary for Hanabi, where players adjust strategies based on their partners’ behavior. Critically, fixed neural parameters do not imply static behavior; our proxies condition on the history of the game, which includes partner actions. Specifically, when the proxy agents see an unexpected action or observation, from that action-observation history onward, they will account for the fact that the other agent is using a different convention. The proxies are trained using SOTA methods and on a diverse dataset of human gameplay data.

Some concrete evidence that the proxies are capable of adaptivity includes:

• Action Prediction Results (Tables 3 & 4): The proxies accurately predict human actions on unseen human gameplay data, which comes from players of various skill levels, who follow different levels of H-group conventions. Such accuracy requires the ability to infer from a given action-observation history what sort of conventions the human is following.
• Successful Coordination with Brittle Partners (Figure 2): The proxies achieve high performance even when paired with behavioral cloning agents, which are known to be prone to suboptimal play. The proxies’ success even in such scenarios provides strong evidence that they make adaptations during gameplay to ensure effective coordination.

We have further clarified this in the paper and hope this addresses how the proxies reflect dynamic adaptivity similar to human play.

Real-World Relevance of AH2AC2

Success in human-AI collaboration requires the ability to robustly generalize cooperative behaviors from limited human data. As well, the literature requires a standardized testing protocol to rigorously measure algorithmic progress. AH2AC2 directly addresses both of these fronts: we release an experimental suite consisting of carefully validated human-proxy agents, trained on extensive human gameplay data, which serve as robust and reproducible evaluation partners. As well, we open-source a deliberately limited dataset of human games to encourage research specifically on data-efficient coordination algorithms.

Novelty and Suitability for ICML

We appreciate the opportunity to clarify our decision to submit this work to ICML. Benchmark creation is explicitly recognized by ICML as a meaningful and rigorous research contribution. In particular, our benchmark required:

• An Extensive Original Effort: Large-scale human data collection, meticulous data cleaning, validation, and rigorous modeling using SOTA methods.
• Novel Evaluation Platform: Developing reproducible human proxy models, carefully validating them, and establishing a standardized evaluation protocol for human-AI coordination in partially observable, cooperative environments.

To the best of our knowledge, AH2AC2 is the first benchmark to explicitly assess human-AI coordination in complex, partially observable settings using realistic human-proxy agents.

Conclusion

In light of your review, we have modified the paper to make the broader significance of our work more clear. We believe a benchmark such as AH2AC2 is highly necessary for progress in human-AI collaboration, and is to date decidedly lacking in the literature.

Please let us know how we can further improve the paper. Otherwise, if you believe these clarifications have benefited the work, we would appreciate a corresponding update in the score.

Thank you for your continued engagement with the review process.