Learning to Cooperate with Humans using Generative Agents

Thank you for your detailed review and positive feedback on our comprehensive real-human study. We're glad you find it has high ecological validity.

There are a few experiment results (Fig 5a and 6b) where GAMMA + HA (FFT) is not good on Multi-strategy Counter.

We acknowledge that the FFT method originally performed poorly due to limited human data. Our further experiments found that increasing the regularization can mitigate the problem of lack of human data, enabling GAMMA HA FFT to provide good performance in Multi-Strategy Counter as well. See global response and Fig (e) in the rebuttal PDF for details.

One potential future direction is to infer the human latent variable […]

Thank you for pointing out this! We agree that we can also make $z$ part of $\pi$. Note sampling from $z_i$ and making $z_i$ part of $\pi$ are orthogonal ideas and can be combined to train the cooperator.

We implement this idea by allowing the Cooperator to condition on $z$ inferred by the VAE. We present the results in Fig. (c) of the rebuttal PDF and find that this improves both learning efficiency and performance on Multi-strategy Counter.

Are trajectories in the human dataset all from the same human-human team […]

For layouts proposed by [1], we use their open-source human from randomly paired human-human teams. For the new Multi-strategy Counter, we collect data similarly, sampling players randomly and explaining the game rules before collecting their gameplay data. These strategies are not always optimal; in Figure (f) of the rebuttal PDF, 49.5% of the players are novice overcooked players. People played according to their own strategies without prompts. We will add this information to the paper.

Details of Fig 1.

Please refer to our common response.

Could PPO guarantee to learn an optimal cooperator policy?

In general, gradient-based RL is not guaranteed to converge in multi-agent settings [3]. Although we can not ``guarantee’’ PPO to learn optimally, we give it enough training time (1.5e8 steps) to converge.

There seems to be no theoretical guarantee [...]

We will soften the claims in lines 198-203. We acknowledge assuming the human data for training is representative of the target population during the human study. We will add the following limitation to the paper:

"We assume the human dataset for VAE training and human-adaptive sampling matches the distribution of human players during evaluation. This may not hold if the dataset quality is low. Addressing distribution shift from training to testing is important and will be our future work."

How is the held-out performed on human data and self-play agents respectively? [...]

The 70%-30% split is only used to determine the validation dataset for VAE training. The held-out agents will never be used for training VAEs or Cooperators.

[...] examining the quality and features of the used data set, [...]

We provide a detailed introduction to human data here and will add this to the paper. For the human dataset in the original Overcooked paper [1], their open-sourced dataset contains ``16 joint human-human trajectories for Cramped Room environment, 17 for Asymmetric Advantages, …, and 15 for Counter Circuit.’’ with length of $T\approx 1200$. In Multi-strategy Counter, we collect 38 trajectories with length $T\approx 400$ which is closer to the actual episode length during training.

For the human study, we have made additional plots that show the participants’ distribution of experience levels for playing Overcooked. In Figure (f) in the rebuttal PDF, 49.5% of players agreed that they had no experience of playing Overcooked. We also recommend looking at the demos on our website, where we show videos of humans using different strategies.

What does this sentence mean in lines 286-288? [...]

While the proxy metrics are relatively cheaper to run than real human evaluations, both our work and [1] find that they favor the PPO-BC method, and thus can be misleading for final coordination performance with real humans. Again, we emphasize that the most important experiment is the human evaluation (Figure 6) and GAMMA is statistically significantly better.

How novel are the strategies in human evaluation […]?

For all human players, we only give them basic instructions about the game, so they are free to use unique strategies. Especially for Counter Circuit, we recruit an entirely new population of participants using a different platform and potentially different recruiting criteria of [1]. Therefore, this test-time population could be quite different from the training dataset [1] released several years ago.

How accurate is the BC model? [...]

The accuracy is around 55-60%. Under self-play, our BC model reaches 40-80 scores (2-4 soups). We use the same dataset [1] to train the BC model as prior work.

Compare GAMMA with type-based reasoning work [2].

We thank the reviewer for pointing us to [2], which has a similar setting to our PPO + BC + GAMMA method. During training, [2] clusters human trajectories into different strategies and learns the best response for each, while GAMMA uses a VAE over the entire human dataset to train a universal Cooperator. When facing a novel human, [2] infers the human's strategy and applies the best response to it, whereas GAMMA uses the universal Cooperator. Thus, while [2] is limited to partner strategies in the human dataset, PPO + BC + GAMMA can potentially generate novel partner policies through interpolation. Moreover, unlike [2], which only uses human data, GAMMA can combine synthetic and human data with controllable sampling like GAMMA HA. We will cite this work and include the discussion in the paper.

[1] M. Carroll, et al. On the utility of learning about humans for human-ai coordination.

[2] Zhao, et al. Coordination with humans via strategy matching.

[3] Mazumdar, et al. "On finding local nash equilibria (and only local nash equilibria) in zero-sum games."

Thank you for your review! For typos, we will correct them in the next version. We address your concerns below.

The baselines are outdated.

Thanks for the suggestion, we have run MEP (see Fig 2 in the rebuttal PDF) and show that GAMMA can still improve its performance.

We agree all the methods (MEP, E3T, SBRLE, and PECAN) that you mentioned are important and newer works in the human-AI cooperation field. We would like to point out that our GAMMA can not only 1) improve the diversity of the population with generative agents, which those prior works did; but also 2) combine another line of work of human modeling from human data under our human-adaptive sampling techniques.

To test 1), we need to show that GAMMA can improve diversity by generative agents trained on any synthetic population. Since FCP is the most popular and CoMeDi is one of the latest works, we believe these are two representative populations to test GAMMA. We will include the additional results with MEP in the revised version of the paper.

GAMMA's improvements seem marginal in Fig. 4,5

• First of all, for the most important human experiment in Fig 6, it is statistically significant that GAMMA is better than baselines (p < 0.05). See more details in the common response.

• Regarding Cramped Room and Forced Coordination in Figure 4, we agree GAMMA does not provide significant improvement. The main reason is that the environments are so simple that the optimal policies can be easily found for certain algorithms. In more detail, in Cramped Room, the agents simply need to fill the pot whenever it is available. In Forced Coordination, the left agent only needs to place the onions and dishes on the central table, while the right agent only needs to collect the onions for the pot and use the dish to complete a soup order. Note that for more complex environments like Coordination Ring or Counter Circuit, GAMMA gives significant gains.

• Regarding Figure 5, we choose these experiments to follow existing works [1] [4]. We agree that GAMMA’s performance is only comparable to the strongest baseline but not significantly better. On the other hand, we note there is a problem with this experiment. As shown in [5], the performance evaluated by the human proxy model (a BC model) is not predictive of performance with real humans since BC models are trained on limited data, are not reactive or adaptive in the same way as humans and therefore have considerably different behavior than real human agents on evaluation. While the proxy metrics are relatively cheaper to run than real human evaluations, both our work and [5] find that they can be misleading in terms of final coordination performance with real humans. Again, we emphasize that the most important experiment is the human evaluation (Figure 6) and GAMMA is statistically significantly better.

What do the points in Figure 1 stand for?

We provide a detailed response in the common rebuttal. It is the latent vector $z$. For all population $P$, each point of $P$ is computed by 1) first sampling an agent in $P$; 2) using the agent to generate an episode with self-play, and 3) encoding the episode into $z$ using VAE. These high-dimensional latent vectors are compressed to 2D using t-SNE. We promise to add this explanation to the paper in the future.

Compared to PECAN [2], what is the advantage of GAMMA by interpolating at the level of hidden vectors?

Thank you for pointing us to the PECAN paper [2], we will include a citation of this work and the following discussion of the advantages of using GAMMA in the revised version of the paper:

''PECAN [18] proposes an ensemble method to increase the partner population diversity. In PECAN, partner agents are categorized into three groups based on their skill levels. With GAMMA, the skill levels are automatically learned and encoded in the latent variable $z$. And by sampling different $z$, generative partner agents with different skill levels are thus sampled.

GAMMA offers several advantages over this method. First, GAMMA provides a controllable way to sample only from a target subpopulation (e.g., humans) by adaptive sampling over the latent space. Second, it is known that with interpolation on the latent variable $z$ [3], generative models can produce novel samples beyond the weighted sum of low-level pixels/actions. ''

Typos: "data-hungry seuential decision making algorithms...", on page 1.

Thank you for your time and attention to detail. We promise to fix that in the future version.

[1] C. Yu, J. Gao, W. Liu, B. Xu, H. Tang, J. Yang, Y. Wang, and Y. Wu. Learning zero-shot cooperation with humans, assuming humans are biased.

[2] Lou, Xingzhou, et al. "Pecan: Leveraging policy ensemble for context-aware zero-shot human-ai coordination."

[3] Radford, Alec, Luke Metz, and Soumith Chintala. "Unsupervised representation learning with deep convolutional generative adversarial networks."

[4] D. Strouse, K. McKee, M. Botvinick, E. Hughes, and R. Everett. Collaborating with humans without human data.

[5] M. Carroll, R. Shah, M. K. Ho, T. L. Griffiths, S. A. Seshia, P. Abbeel, and A. D. Dragan. On the utility of learning about humans for human-ai coordination.

We thank you for your detailed review. We address your questions below.

Results lack statistical significance

• First of all, for the most important human experiment, GAMMA is statistically significantly better (p < 0.05). See the common response in detail.
• For Cramped Room and Forced Coordination in Fig 4, we agree GAMMA does not provide significant improvement. The main reason is that the environments are so simple that the optimal policies can be easily found. Specifically, in Cramped Room, the agents simply need to fill the pot whenever it is available. In Forced Coordination, the left agent only needs to place the onions and dishes on the central table, while the right agent only needs to collect the onions for the pot and use the dish to complete an order. Note that for more complex Coordination Ring or Counter Circuit, GAMMA gives significant gains.
• Regarding Fig 5, we choose these experiments following [2] [4]. We agree that GAMMA’s performance is only comparable to the strongest baseline. Note there is a problem with this evaluation. As shown in [1], the performance evaluated by the BC human proxy model is not predictive of human study. The BC model trained on limited data, is not reactive or adaptive in the same way as humans and therefore has considerably different behavior. While the proxy metrics are relatively cheaper to run than real human evaluations, both our work and [1] find that they can be misleading for performance against real humans. Again, we emphasize that the most important experiment is the human evaluation (Fig 6) and GAMMA is statistically significantly better.

Illustrate the necessity of sampling from $z_i$

We first explain the rationale of sampling from $z_i$. Our primary idea is to build a population and train a cooperator against it. We choose VAE to represent this distribution and here $z_i$ serves as the latent vector to generate partner policies.

We agree that we can also make $z_i$ part of $\pi$. Note that sampling from $z_i$ and making $z_i$ part of $\pi$ are orthogonal ideas and can be combined together. We have conducted two additional experiments.

First, we make the Cooperator condition on the $z$ inferred by the VAE. In Fig (c) in the rebuttal PDF, we found this approach improves both learning efficiency and performance on Multi-strategy Counter.

Second, as another baseline, we use the $z$-conditioned decoder of the VAE as the policy. The results are presented in Fig (d) in the rebuttal PDF. We find that the performance of the $z$-conditioned decoder is not good compared to the Cooperators trained by sampling partners from the VAE.

The length of games is short

We appreciate your insight as an experienced Overcooked player. We chose this time limit to be consistent with prior works; 60s for FCP and 40s for CoMeDi. In our evaluation, the human-AI team can usually finish 2-5 soups (60-100 scores) in 60s.

Can increasing the sampling temperature also boost the performance?

Making the partner's actions more random could simulate a less skilled agent. Our VAE training dataset already includes random agents, enabling it to interpolate between expert and random agents.

The VAE offers a significant advantage over merely adding noise to players' actions, as it can combine different skillful strategies. For example, if partners follow strategies A, B, or C, adding noise to A produces an inexpert A. In contrast, the VAE can create a partner proficient in all strategies, like ACBAAC. We hypothesize the VAE better covers possible skilled partner strategies (Fig 1), and our experimental results support this hypothesis.

Why mention ``training on human data is highly effective’’ particularly in line 95?

We agree that it is intuitive that adding human data should improve performance. Prior work [2] shows that FCP > PPO-BC in four layouts including our Counter Circuit, and PPO-BC > FCP for Asymmetric Advantage. In our experiments, we found PPO-BC also effective for both Counter Circuit and Multi-strategy Counter. We hope our findings can encourage the community to engage in further discussion and empirical studies on this problem.

Why use held-out self-play agents in Fig 5b? Line 95 states that human data is not so effective in human evaluation. Then, why is fine-tuning on human data is good?

We use held-out self-play agents to follow existing works [2, 4]. The reason to use self-play agents is that a human-proxy model is not a good evaluation metric, as shown in prior work [1]. We emphasize again that the most important experiment is the human evaluation (Fig 6) and GAMMA is statistically significantly better.

Our finding shows that human data is quite effective, while prior work [2] mentioned in line 95 focuses on training without human data. We hope our findings can encourage more attention to this problem.

How are the number of policies (8) and the number of random seeds (5) decided in human evaluation?

We follow the same number of five random seeds as prior works [2, 3] to ensure the statistical significance of our results We did two separate human evaluation rounds of the two layouts we used. For each round, only one layout would be tested. We first shuffled the order of eight algorithms randomly, and then for each algorithm, we sampled one of five random seeds to play against humans.

H3 is not a hypothesis.

Thanks. We will revise H3 to, “Generalizing to novel humans: we hypothesize that GAMMA will generalize more effectively to novel humans than prior methods, showing improved performance in both game score and human ratings.”

[1] M. Carroll, et al. On the utility of learning about humans for human-ai coordination.

[2] D. Strouse, et al. Collaborating with humans without human data.

[3] R. Zhao, et al. Maximum entropy population-based training for zero-shot human-ai coordination.

[4] C. Yu, et al. Learning zero-shot cooperation with humans, assuming humans are biased.

We thank the reviewer for your detailed and insightful feedback. We are grateful that you took the time to check out our demo. We hope the following responses address your concerns.

Does the proposed method GAMMA fairly consistently outperform baselines statistically significantly?

GAMMA does outperform baselines, which we clarify in the global response as well. The human evaluation results (Fig 6 and 7) should be considered the gold standard evaluation, and here GAMMA shows a statistically significant improvement over all baselines on both layouts. We provide a table of human evaluation results (Table 1) in our rebuttal PDF which makes this more clear. Other simulated metrics (Fig 4 and 5) are not always predictive of human evaluation performance.

What is the error bar in Fig 6? Which of the results are statistically significant?

Because the error bars correspond to the Standard Error of the Mean (SE), if two error bars are non-overlapping, the results are considered statistically significantly different at the p < 0.05 level. For more information on this, please see: G. Cumming, F. Fidler, and D. L. Vaux. Error bars in experimental biology. The Journal of cell 377 biology, 177(1):7–11, 2007. Table 1 in the rebuttal PDF bolds the GAMMA models that are significantly better than the baseline model trained with the same data.

How complex are the strategies that the Cooperator can learn? Can they deal with adaptive humans?

In our evaluation with real humans, participants are free to change the strategy they play during the episode. In fact, since many of our participants are novice players who have little experience with the game Overcooked or with video games in general, often their performance continues to improve as they play. We show in Figure (e) in the rebuttal PDF that performance improves consistently with the number of trials, providing evidence that humans are changing their gameplay throughout the evaluation. Since our method provides the best performance when playing with real humans, we believe it is best at adapting to their changing game play.

How is the human training data collected? If I understand correctly, the human data is collected either in solo play or interaction with a simple existing policy. It seems unlikely then, that the GAMMA-based methods could deal with humans learning, adapting, and changing their strategies.

We would like to clarify that the data is collected by randomly assigning two human players to play together, so it consists of human-human trajectories where players can adapt and change their strategies. Further, as we show above, in our evaluation human players do learn to improve their performance over time, showing evidence of learning and adapting throughout the evaluation.

The gains are reduced for more complex settings.

We believe GAMMA is a general approach that can still be applied to more complex settings. Figure 6 shows that GAMMA HA DFT still provides statistically significantly higher performance than competitive baselines in the more complex layout. In the original experiment, FFT is less effective on Multi-strategy Counter due to the relative lack of human data. In Figure (b) in the rebuttal PDF, We solve this issue by using a larger KL Divergence penalty coefficient and now both FFT and DFT achieve the best performance on Multi-strategy Counter.

Can the Cooperator instead be meta-learned and adapt online?

Currently, our Cooperator adapts according to the history of the other agents. Thank you for the suggestion. We agree that we can also make $z_i$ part of $\pi$. Here we note, sampling from $z_i$ and making $z_i$ part of $\pi$ are orthogonal ideas and can be combined to train the cooperator. We have conducted the following experiment.

We implement this idea by allowing the Cooperator to condition on the z inferred by the VAE. We run this experiment and present the results in Fig. (c) of the rebuttal PDF. Our findings indicate that this approach improves both learning efficiency and performance on the Multi-strategy Counter layout.

Limitations. The human data in this work may not be representative. The current experiments are only in two-player settings.

Thank you for pointing out these limitations. We will add a limitations section to the revised version of the paper with the following text:

''In this work, our human studies recruit participants from Prolific, which may not be representative of broader populations. Additionally, our human dataset is limited, which could reduce the diversity of strategies and force participants to adapt to strategies that the Cooperators are already familiar with.

We focus on the two-player setting in this study following prior work [1] [2] because it is a first step toward enabling an AI assistant that could help a human with a particular task. Scaling up to more agents would exponentially increase the dataset size with our current techniques. Therefore, better sampling techniques are needed to address this issue.''

[1] D. Strouse, K. McKee, M. Botvinick, E. Hughes, and R. Everett. Collaborating with humans without human data.

[2] C. Yu, J. Gao, W. Liu, B. Xu, H. Tang, J. Yang, Y. Wang, and Y. Wu. Learning zero-shot cooperation with humans, assuming humans are biased.

Thank you for the rebuttal! Some of my concerns have now been addressed, but the most major one still remains: With regard to standard errors, a gap of one standard error between the edges of the standard error bars corresponds to p=0.05. A gap of one standard error between the means does not correspond to p=0.05. See Fig 5 of the Cumming et al. (2007) paper for an example of this. It is still unclear which results are statistically significant. Consider applying a traditional hypothesis test to get the exact p-values rather than estimating them based on the gap between SEs.

Copying this over from my individual response below:

With regard to standard errors, a gap of one standard error between the edges of the standard error bars corresponds to p=0.05. A gap of one standard error between the means does not correspond to p=0.05. See Fig 5 of the Cumming et al. (2007) paper for an example of this. It is still unclear which results are statistically significant. Consider applying a traditional hypothesis test to get the exact p-values rather than estimating them based on the gap between SEs.