We thank the reviewer for their comprehensive feedback of our work. We greatly appreciate the reviewer for highlighting some of the strengths of our work, including that our method is well-motivated, effective, and grounded in principled approaches. We are encouraged that our human-agent user study lends credibility to the real-world applicability of the research, and that the paper is well-written and easy to follow. Below, we address the weaknesses and questions raised by the reviewer.

Discussion on Related Work

We greatly appreciate the reviewer’s suggestions with respect to the discussion of related work. We have expanded our discussion, particularly in the Strategy Inference section to highlight E3T, the bayesian adaptation framework of GSCU, the contextual encoder and exploration reward of PACE, as well as the LLM-based belief inference method of ProAgent. We hope that this additional discussion will provide a clearer understanding of the body of work that has informed and contributed to the development of TALENTS. In addition, we hope that this discussion will provide a clearer understanding as to the novelty and contributions of our method.

Furthermore, in addition to discussing them in the related work section, we have prepared E3T and PACE as additional baselines in our experiment section (see next response for details)

Additional Baselines

In relation to the previous comment, we recognize that evaluating our method against the aforementioned methods is important to provide better context and positioning of our novel TALENTS agent. Based on the reviewer’s recommendation, we have decided to implement E3T and PACE as additional baselines. We chose not to evaluate GSCU, as it was included in the PACE paper as a baseline and was significantly outperformed by PACE in the Overcooked setting [1]. The following are our preliminary results from the new baselines:

TALENTS remains the top performer out of all the assessed methods, while E3T is able to outperform all the other baselines. PACE underperforms, achieving comparable scores to the population best response baseline. PACE, similar to other bayesian adaptation methods like GSCU and MeLIBA [2], are effective in the inter-episodic setting, where the agent is able to form beliefs over several episodes of play with the same agent, transitioning from exploratory to exploitative behaviors over time. In our zero-shot coordination setting, in which agents must immediately adapt to unseen partners, PACE is unable to form its belief in the short time frame necessary for high-performance, and therefore achieves a similar score as the naive population best response baseline. This highlights a key differentiator and source of novelty of TALENTS, which is its ability to perform intra-episodic adaptation. This capability is largely made possible by the fixed-share tracking regret minimization method (Algorithm 2), and is a reason why TALENTS achieves SOTA performance as a zero-shot coordinator.

Extension to Competitive Settings

We appreciate the reviewer’s suggestion to explore competitive or adversarial settings. While we agree these are valuable directions, our work is focused on the zero-shot coordination / ad-hoc teaming problem in cooperative environments, where the goal is rapid adaptation to unfamiliar partners. Introducing competition would shift the scope to a different class of problems involving adversarial intent and asymmetric payoffs, requiring distinct modeling and evaluation. That said, we agree that extending TALENTS to mixed-motive or adversarial domains is a promising avenue for future research, and we now note this in the conclusion. We also agree with the reviewer’s point on environment complexity. This was a key motivation for utilizing a modified version of Overcooked, as the original setting [4] only had limited opportunities for behavioral diversity. While recent extensions [3, 1] introduced more difficulty, our modified environment introduces considerable complexity through the inclusion of larger maps, longer episodes (T=240s rather than the typical T=40s [9,4] or T=60s [3,8]), temporal constraints, and multi-step tasks, enabling a more intricate evaluation of agent robustness in highly complex teamwork tasks. While using image-based observations is feasible, we chose the grid-based representation of Overcooked to enable a smaller model footprint, requiring fewer parameters and less data. Moreover, the structured representation focuses the learning signal, making it easier to analyze and ablate the key factors that contribute to our method’s success. That said, we agree this raises an interesting direction for future work and would be more akin to real-world applications.

User Study Survey Scores

This is a great question, we thank the reviewer for raising this discrepancy. The survey responses in the trust and fluency category are typically indicative of the human’s perception of their agent partner’s capability and proficiency [6,7], whereas the coordination and satisfaction metrics tend to be more indicative of the human’s perception of the joint teamwork [5]. We hypothesize that the lack of statistically significant ratings in these regards may be due to the distribution shift between agent-agent games and human-agent games. Agents tend to move differently in the Overcooked environment compared to human players, and may perform actions perceived as unpredictable or erratic to their human teammates. Since our cooperator agents were only paired with other agents during training, there was no mechanism to incentivize them to behave in a “human-interpretable” way. This shift is equally present among the three methods assessed in the user study. To further assess this hypothesis, we would need to ask more post-game questions (particularly open-ended questions) to participants, which was beyond the scope of the initial human subject study.

[1] Ma, L., Wang, Y., Zhong, F., Zhu, S. C., & Wang, Y. (2024). Fast peer adaptation with context-aware exploration. arXiv preprint arXiv:2402.02468.

[2] Zintgraf, L., Devlin, S., Ciosek, K., Whiteson, S., & Hofmann, K. (2021). Deep interactive bayesian reinforcement learning via meta-learning. arXiv preprint arXiv:2101.03864.

[3] Liang, Y., Chen, D., Gupta, A., Du, S. S., & Jaques, N. (2024). Learning to cooperate with humans using generative agents. Advances in Neural Information Processing Systems, 37, 60061-60087.

[4] Carroll, M., Shah, R., Ho, M. K., Griffiths, T., Seshia, S., Abbeel, P., & Dragan, A. (2019). On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32.

[5] Richard M Ryan and Edward L Deci. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American psychologist, 55(1):68, 2000

[6] Guy Hoffman. Evaluating fluency in human–robot collaboration. IEEE Transactions on Human-Machine Systems, 49(3):209–218, 2019

[7] Richard M Ryan and Edward L Deci. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. American psychologist, 55(1):68, 2000

[8] Strouse, D. J., McKee, K., Botvinick, M., Hughes, E., & Everett, R. (2021). Collaborating with humans without human data. Advances in neural information processing systems, 34, 14502-14515.

[9] Zhao, R., Song, J., Yuan, Y., Hu, H., Gao, Y., Wu, Y., ... & Yang, W. (2023, June). Maximum entropy population-based training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 5, pp. 6145-6153).

We would like to thank the reviewer for their thoughtful comments and suggestions. We are encouraged to hear that our method, TALENTS, has been perceived as novel, original, and well motivated. Additionally, we are excited to hear that our empirical results, particularly TALENTS' performance across Overcooked layouts and the human-agent evaluation, were seen as strong and indicative of real-world applicability. We also appreciate the recognition of the technical soundness of our approach, especially the integration of VAE-based behavior modeling with fixed-share regret minimization. In the following, we will address the weaknesses and questions of the reviewer:

Limited Evaluation on Overcooked

Thank you for requesting further information about the evaluation domain. Overcooked is a challenging domain for AI agents [1,2,3], as well as humans, providing a rich testbed for agent-agent and human-agent teaming capabilities due to various factors, such as time pressure, multiple cooking stations, multi-step recipes, and a reward structure that encourages fast deliveries. While we agree that evaluating TALENTS in additional domains such as Hanabi (which introduces partial observability) or cooperative pursuit-evasion (which adds adversarial dynamics) would be valuable, incorporating such experiments is beyond the scope of this submission. We consider these exciting directions for future work and plan to explore them in follow-up studies.

Cluster Sensitivity

We appreciate the request for further insights into the number of clusters as well as the sensitivity of TALENTS with respect to the number of chosen clusters. To clarify, we utilize silhouette analysis to cluster our latent embedding space with the goal of finding a suitable representation of discrete strategy types. There is precedence of this method as an effective way of partitioning a latent embedding space [7,8]. To the reviewer’s point, prior work in opponent modeling and latent encoding methods have explored modifications to the VAE objective in order to learn a clustered embedding space in a more end-to-end fashion [10,9,6], but these objectives often require privileged information such as the identity of the agent from which the trajectory originated from in order to function [10,9]. Unlike these methods, TALENTS does not make such assumptions. To evaluate the efficacy of silhouette analysis for selecting an optimal number of clusters, we perform a sensitivity analysis over a number of cluster partitions, ranging from the naive one-cluster case to 16 clusters. We find that the silhouette analysis optimal number of clusters (11) achieves the highest score, empirically demonstrating the strength of our clustering approach.

The issue of potentially misclassifying a partner’s behavior and selecting an inappropriate cluster is a valid concern. To this end, we assess TALENTS’s ability to correctly identify player behaviors. We have conducted additional experiments in which TALENTS is paired with all static experts, correctly identifying the correct expert in 91.3% of evaluations.

Generalization and Unseed Human Data

Thank you for requesting further insights on TALENTS’s ability to handle novel human behavior. It is indeed correct that one assumption of TALENTS is that the VAE’s latent space sufficiently covers novel human behavior. The latent space is trained via a dataset of 11 reward-shaped policies. Evaluation is done using a population of 10 diverse unseen policies for all experiments to ensure meaningful generalization of our agent. There is no guarantee that these evaluation policies are necessarily represented by any particular cluster in the latent space, but the rationale of using a regret minimization framework for partner inference is the guarantee that out-of-distribution policies will not cause catastrophic performance degradation due to the framework's theoretical bounds. Furthermore, we analyzed the VAE’s ability, which is only trained on our population agents, to encode the data from our human experiments and found that human data covers a similar distribution in our VAE in terms of cluster representation as the behavior preference agents used to train the model. Calculating the Jensen-Shannon divergence of the agent and human data distributions yields 0.1516, indicating a high degree of similarity between the two. This value suggests that the agent behavior closely approximates human behavior, with only minor differences.

Fixed-Share Regret Minimization

We appreciate the request for a more in-depth evaluation of non-stationary partners and agree that this is an important area of analysis. As shown in Figure A.6, TALENTS is capable of adapting to novel partners within just a few seconds. Additionally, we show that TALENTS successfully identifies the correct expert in 91.3% of evaluation episodes. Given that the VAE effectively captures the range of human behaviors observed in our human-agent experiments, and that our human subject study reports high ratings for fluency, trust, and overall performance relative to other baselines (see Figure 4), we conclude that TALENTS performs robustly in settings with non-stationary partners (such as humans).

Additional Baselines

We appreciate the reviewer for pointing out the need for additional baselines and are excited to report that we have conducted additional experiments with two more baselines, namely E3T [4] and PACE [5]. We have added these baselines in addition to our prior evaluation of TALENTS against several leading zero-shot coordination baselines (GAMMA, MEP, FCP, BP). E3T [4] utilizes a modified self-play objective alongside a partner modeling module to perform ZSC, and PACE [5] adapts its behavior via a learned context encoder. Preliminary results are as follows:

With these results, TALENTS demonstrates that it can, consistently, outperform other baselines, providing a significant improvement over existing literature.

[1] Liang, Y., Chen, D., Gupta, A., Du, S. S., & Jaques, N. (2024). Learning to cooperate with humans using generative agents. Advances in Neural Information Processing Systems, 37, 60061-60087.

[2] Zhao, R., Song, J., Yuan, Y., Hu, H., Gao, Y., Wu, Y., ... & Yang, W. (2023, June). Maximum entropy population-based training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 5, pp. 6145-6153).

[3] Carroll, M., Shah, R., Ho, M. K., Griffiths, T., Seshia, S., Abbeel, P., & Dragan, A. (2019). On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32.

[4] Yan, X., Guo, J., Lou, X., Wang, J., Zhang, H., & Du, Y. (2023). An efficient end-to-end training approach for zero-shot human-AI coordination. Advances in neural information processing systems, 36, 2636-2658.

[5] Ma, L., Wang, Y., Zhong, F., Zhu, S. C., & Wang, Y. (2024). Fast peer adaptation with context-aware exploration. arXiv preprint arXiv:2402.02468.

[6] Zintgraf, L., Devlin, S., Ciosek, K., Whiteson, S., & Hofmann, K. (2021). Deep interactive bayesian reinforcement learning via meta-learning. arXiv preprint arXiv:2101.03864.

[7] Zhao, M., Simmons, R., & Admoni, H. (2022, October). Coordination with humans via strategy matching. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 9116-9123). IEEE.

[8] Pandya, R., Zhao, M., Liu, C., Simmons, R., & Admoni, H. (2024, May). Multi-agent strategy explanations for human-robot collaboration. In 2024 IEEE International Conference on Robotics and Automation (ICRA) (pp. 17351-17357). IEEE.

[9] Papoudakis, G., & Albrecht, S. V. (2020). Variational autoencoders for opponent modeling in multi-agent systems. arXiv preprint arXiv:2001.10829.

[10] Aditya Grover, Maruan Al-Shedivat, Jayesh K Gupta, Yura Burda, and Harrison Edwards. Learning policy representations in multiagent systems. International Conference on Machine learning, 2018a.

We thank the reviewer for their thorough and insightful review. We appreciate that the reviewer found the proposed TALENTS method to be well-motivated, sophisticated, and effective, as well as being well-supported by empirical results. Below, we will address the critiques and weaknesses:

Additional Baselines (MeLIBA, non-VAE methods):

We thank the reviewer for their suggestion for additional baselines and evaluating the impact of VAE and non-VAE based methods. We agree that this is an important to evaluation. To this end, we currently evaluate against GAMMA [1] (a SOTA method which uses a VAE for partner generation), as well as BP [2], FCP [3], and MEP [4] (SOTA population-trained methods which do not use a VAE at all). The requested MeLIBA baseline falls into the same category as the GAMMA agents, using a VAE as part of its methodology [5]. However, to further extend our experiments, we have implemented E3T [6] as an additional baseline (a SOTA self-play method which does not use a VAE but has a partner modeling module for strategy inference), as well as PACE [7] (a SOTA population-trained method which uses a context encoder to explore and adapt to partners). Overall results, including the new baselines, are as follows, showing strong performance of TALENTS as compared to other baselines on the open and hallway layouts.

With the inclusion of new baselines, TALENTS still maintains the highest performance out of all assessed methods, while E3T is able to outperform all the other baselines. PACE underperforms, achieving comparable scores to the population best response baseline. PACE, similar to other bayesian adaptation methods like GSCU [8] and MeLIBA [4], are effective in the inter-episodic setting, where the agent is able to form beliefs over several episodes of play with the same agent, transitioning from exploratory to exploitative behaviors over time. In our zero-shot coordination setting, in which agents must immediately adapt to unseen partners, PACE is unable to form its belief in the short time frame necessary for high-performance, and therefore achieves a similar score as the naive population best response baseline. This highlights a key differentiator and source of novelty of TALENTS, which is its ability to perform intra-episodic adaptation. This capability is in part made possible by the fixed share tracking regret minimization method (Algorithm 2), and is a reason why TALENTS achieves SOTA performance as a zero-shot coordinator.

Switching Between Experts

Thank you for the question about how TALENTS switches between various experts when observing the player and how experts represent specialization. Fundamentally, our VAE (section 4.1) is used for strategy learning by encoding a set of trajectories from our population agents into a latent space. The optimization target for the VAE is, intuitively, to take a sequence of prior states and actions in order to predict the next actions (e.g. “chop meat”) over some horizon. Given this objective, the latent space can be interpreted as a representation of agent behavior. Clustering this latent space results in a set of experts that exhibit certain behavior patterns, e.g., facilitating a strong preference for cooking rice. We can track the high-level action distribution of each expert to evaluate how specialized each best response behavior is. The mean Jensen-Shannon divergence in the hallway layout across all the action distributions with respect to each other is 0.483 +/- 0.125, indicating moderate behavioral differences. A near-zero divergence would indicate the cooperator has not learned significantly different responses to different partner strategies, while an entirely disjoint near-one divergence would suggest that the cooperator has overfit to each partner type, losing some ability to generalize to out-of-distribution partners.

Exactly quantifying how often TALENTS switches its internal estimate of which expert it is playing with depends on various factors, e.g., how skilled the partner is or how often they change their action patterns. To assess TALENTS’s ability to correctly identify player behaviors, we have conducted additional experiments in which TALENTS is paired with static expert agents generated from its own VAE, which allowed us to know the ground truth clusters the partners are supposed to map to, allowing us to evaluate the player behavior categorization method in isolation. Through this experiment, TALENTS converged to identifying the correct expert in 91.3% of episodes. In this setting, it typically took the fixed-share mechanism approximately 30 timesteps to converge to an expert it maintained confidence about for the rest of the episode, seldom changing cluster beliefs partway through an episode. Intuitively, this is not true for experiments with switching policies or dynamic human partners, as we discuss next.

Figure A.6 in the appendix (see supplemental material) provides a figure demonstrating a policy switch, dynamically changing which partner TALENTS is paired with during a game. The experiment shows a partner switch one minute into the game (at time-step 600). As shown in the figure, TALENTS changes the estimate of which partner it is playing with within ~200 timesteps (about 20 seconds), underlining its capability to adjust to changing partners.

Impact of Offline Data

We appreciate the request for analyzing the impact of offline data and agree that this analysis provides valuable insights. To clarify, we generate a dataset from our pretrained behavior policies in order to train the VAE (see Section 4.1). With this data, we ensure that the VAE training is comprehensively covering the possible space of behaviors that potential partners could have. As a preliminary experiment to examine the effect of data quantity, we compare agents trained on a large dataset (550 joint trajectories, 2.64 million total timesteps) with those trained on a smaller dataset (200 joint trajectories, 960,000 total timesteps). We find the large dataset agent to significantly outperform the small dataset agent. Intuitively, the VAE is able to learn a richer latent space with the larger dataset and reduces the risk of overfitting to predominant behaviors.

RNN Policy:

Thank you for suggesting an additional baseline RNN policy. We agree that such results would be valuable and are providing them below. TALENTS-Ablated refers specifically to a TALENTS agent trained with clustered generative VAE agents (Algorithm 1) but does not employ the fixed-share adaptation mechanism (Algorithm 2), while TALENTS-Full refers to our full method. As can be seen in the following results, the addition of the fixed-share adaptation mechanism (Algorithm 2) provides a significant improvement:

Missing caption and incomplete sentence

Thank you for identifying these issues, we have amended the draft to fix them.

[1] Liang, Y., Chen, D., Gupta, A., Du, S. S., & Jaques, N. (2024). Learning to cooperate with humans using generative agents. Advances in Neural Information Processing Systems, 37, 60061-60087.

[2] Wang, X., Zhang, S., Zhang, W., Dong, W., Chen, J., Wen, Y., & Zhang, W. (2024). Zsc-eval: An evaluation toolkit and benchmark for multi-agent zero-shot coordination. Advances in Neural Information Processing Systems, 37, 47344-47377.

[3] Strouse, D. J., McKee, K., Botvinick, M., Hughes, E., & Everett, R. (2021). Collaborating with humans without human data. Advances in neural information processing systems, 34, 14502-14515.

[4] Zhao, R., Song, J., Yuan, Y., Hu, H., Gao, Y., Wu, Y., ... & Yang, W. (2023, June). Maximum entropy population-based training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 5, pp. 6145-6153).

[5] Zintgraf, L., Devlin, S., Ciosek, K., Whiteson, S., & Hofmann, K. (2021). Deep interactive bayesian reinforcement learning via meta-learning. arXiv preprint arXiv:2101.03864.

[6] Yan, X., Guo, J., Lou, X., Wang, J., Zhang, H., & Du, Y. (2023). An efficient end-to-end training approach for zero-shot human-AI coordination. Advances in neural information processing systems, 36, 2636-2658.

[7] Ma, L., Wang, Y., Zhong, F., Zhu, S. C., & Wang, Y. (2024). Fast peer adaptation with context-aware exploration. arXiv preprint arXiv:2402.02468.

[8] Fu, Haobo, et al. "Greedy when sure and conservative when uncertain about the opponents." International Conference on Machine Learning. PMLR, 2022.

We thank the reviewer for the thoughtful and detailed review. We appreciate the informed feedback and suggested areas for improvement. We appreciate the positive remarks regarding our clear methodology discussion and the strength of our human subject study which lends credibility to the efficacy of the TALENTS method. Furthermore, we are encouraged by the reviewer’s assessment that our specification of compute requirements for training aid the reproducibility of our method. Below, we will address the stated weaknesses and questions.

Incremental novelty:

Our work is inspired by the idea of latent policy representations and online partner adaptation. However, there are two primary novelties of TALENTS which significantly differentiate the method from existing state-of-the-art methods:

1. Learning a clustered latent embedding of agent behaviors from offline game trajectories to both condition a cooperator agent and serve as a generative model to produce strategy-specific partner agents during training. GAMMA [1] introduced the method of using a variational autoencoder to create a latent representation of agents from offline trajectories. However, we extend GAMMA by adding a recurrent element to the decoder architecture, enabling the latent to encode a longer-term behavioral representation of agents. Furthermore, we propose silhouette analysis as a method of partitioning the learned latent into individual clusters, enabling our agent to more efficiently learn the best response behavior to rarer agent strategies (in GAMMA, these strategies are sampled much less frequently compared to the predominant strategy). Lastly, the TALENTS cooperator is explicitly conditioned on these clusters during training, which is a novel contribution with respect to generative agent-trained cooperators.

2. TALENTS enables intra-episodic adaptation to zero-shot coordination problems through a tracking regret minimization framework. Prior work in the agent modeling and online adaptation setting (MeLIBA, GSCU, etc) define the adaptation objective as identifying and optimizing strategies over multiple episodes. However, in the zero-shot coordination setting, successful agents must be able to adapt to unseen partner agents within one episode. TALENTS proposes fixed-share tracking regret minimization as an effective method for accomplishing this, and demonstrates in an ablation (Fig 3) the necessity of optimizing tracking regret as opposed to static regret. This novel contribution results in a SOTA adaptive zero-shot coordination agent.

Limited Baselines:

We appreciate the reviewer for pointing out the need for additional baselines and are excited to report that we have conducted additional experiments with two more baselines, namely E3T and PACE. We have added these baselines in addition to our prior evaluation of TALENTS against several leading zero-shot coordination baselines (GAMMA, MEP, FCP, BP). E3T [2] utilizes a modified self-play objective alongside a partner modeling module to perform ZSC while PACE [3] adapts its behavior based on a context encoder. Preliminary results are as follows:

TALENTS remains the top performer out of all the assessed methods, while E3T is able to outperform all the other baselines. PACE underperforms, achieving comparable scores to the population best response baseline. This can be attributed to the fact that PACE, alongside similar methods like MeLIBA [4], are effective inter-episodic adaptation frameworks. However, in this zero-shot coordination problem setting, PACE is unable to form its belief in the short time frame necessary for high-performance, and therefore achieves a similar score as the naive population best response baseline. This highlights a key differentiator and source of novelty of TALENTS, which is its ability to perform intra-episodic adaptation.

We also appreciate the suggestion to consider “Bayesian meta-task reuse”. Unfortunately, it is unclear which specific method the reviewer is referring to. Thus, we would be grateful if the reviewer could kindly point us to a relevant reference or paper, so that we can incorporate that work.

Cluster Selection and Sensitivity:

We appreciate the request for further insights into the number of clusters as well as the sensitivity of TALENTS with respect to the number of chosen clusters. To clarify, we utilize silhouette analysis to cluster our latent embedding space with the goal of finding a suitable representation of discrete strategy types. There is precedence of this method as an effective way of partitioning a latent embedding space [5,6]. To the reviewer’s point, prior work in opponent modeling and latent encoding methods have explored modifications to the VAE objective in order to learn a clustered embedding space in a more end-to-end fashion [9,4], but these objectives often require privileged information in order to function, such as the identity of the agent from which the trajectory originated from [9]. Unlike these methods, TALENTS does not make such assumptions. To evaluate the efficacy of silhouette analysis for selecting an optimal number of clusters, we perform a sensitivity analysis over a range of clusters, ranging from the naive one-cluster case to 16 clusters. We find that the silhouette analysis optimal number of clusters (11) achieves the highest score, empirically demonstrating the strength of our clustering approach.

Cross-domain generality:

Thank you for requesting additional information with respect to the applicability of TALENTS to additional environments, such as Hanabi or pursuit-evasion tasks. TALENTS, as proposed in this work, is targeting pure collaborative tasks between multiple players in a fully observable environment. Overcooked is a challenging and complex environment for human-agent, as well as agent-agent teamwork, requiring effective collaboration between all players [1,7,8]. As such, Hanabi, which utilizes a partial information setting, and pursuit-evasion games, which utilize an adversarial component, are out of scope for the proposed method, but provide an interesting avenue for TALENTS in potential future work.

Computational Cost and Real-Time Capabilities

We acknowledge the need for further clarification of TALENTS’s ability to run in real-time and perform inference with minimal latency to collaborate effectively. As such, we measured the inference time of TALENTS. We tracked the average total time taken from when the policy receives an observation from the environment to when it sends its action back to the environment (expert evaluation and cluster inference occur between those two states). We compare to a policy without any adaptation component as a baseline.

Human-agent experiments in the Overcooked environment run at 10 frames per second, so even with a 23ms latency and the extreme case of expert computation every frame, there is still a considerable margin. The inference time roughly scales linearly with respect to the cluster number. If the number of experts was considerably larger, the cluster inference steps could be parallelized across GPUs, leading to decreased latency. However, since this latency was not an issue in our experiments, we saw no need to make this optimization.

[1] Liang, Y., Chen, D., Gupta, A., Du, S. S., & Jaques, N. (2024). Learning to cooperate with humans using generative agents. Advances in Neural Information Processing Systems, 37, 60061-60087.

[2] Yan, X., Guo, J., Lou, X., Wang, J., Zhang, H., & Du, Y. (2023). An efficient end-to-end training approach for zero-shot human-AI coordination. Advances in neural information processing systems, 36, 2636-2658.

[3] Ma, L., Wang, Y., Zhong, F., Zhu, S. C., & Wang, Y. (2024). Fast peer adaptation with context-aware exploration. arXiv preprint arXiv:2402.02468.

[4] Zintgraf, L., Devlin, S., Ciosek, K., Whiteson, S., & Hofmann, K. (2021). Deep interactive bayesian reinforcement learning via meta-learning. arXiv preprint arXiv:2101.03864.

[5] Zhao, M., Simmons, R., & Admoni, H. (2022, October). Coordination with humans via strategy matching. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (pp. 9116-9123). IEEE.

[6] Pandya, R., Zhao, M., Liu, C., Simmons, R., & Admoni, H. (2024, May). Multi-agent strategy explanations for human-robot collaboration. In 2024 IEEE International Conference on Robotics and Automation (ICRA) (pp. 17351-17357). IEEE.

[7] Zhao, R., Song, J., Yuan, Y., Hu, H., Gao, Y., Wu, Y., ... & Yang, W. (2023, June). Maximum entropy population-based training for zero-shot human-ai coordination. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 37, No. 5, pp. 6145-6153).

[8] Carroll, M., Shah, R., Ho, M. K., Griffiths, T., Seshia, S., Abbeel, P., & Dragan, A. (2019). On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32.

[9] Papoudakis, G., & Albrecht, S. V. (2020). Variational autoencoders for opponent modeling in multi-agent systems. arXiv preprint arXiv:2001.10829.