Thank you for your time and effort in reviewing our paper, and for recognizing its strengths in clarity, empirical validation, and the intriguing insights it provides.

New experiments

Based on your suggestions, we’ve added the following results to broaden the experimental scope to more complex scenarios, and improve the analytical strength of our work:

• CEC in Partially Observable Environments. Testing CEC in a partially-observed version of the Dual Destination problem yielded similar results: CEC achieved high cross-play performance on novel environments (0.74), outperforming population-based methods (0.61) and naive self-play (0.03).
• Combining Task and Partner Diversity. Following reviewer mZRY and 11py’s suggestions, we tested combining CEC with E3T. While it performed poorly on the original Overcooked layouts (CEC=130.51, CEC-E3T=28.21), it outperformed CEC-Finetune on held-out grids (CEC-FT=41.73, CEC-E3T=58.13). We hypothesize the E3T partner noise will require larger networks and more training time when combined with CEC’s diverse environments.
• Analyzing the effect of RNNs in CEC. Following reviewer SmWr’s suggestions, we provide new experiments ablating the use of an RNN for CEC, where the RNN is used to provide a simple meta-learning algorithm (Wang et al, 2018; Rabinowitz et al, 2018). In Overcooked and the Dual Destination problem, agents without recurrence failed to converge or adapt effectively. As shown in the learning curves for the Dual Destination problem, CEC with LSTMs successfully converged, while the non-recurrent version could not even achieve positive rewards.

Overcooked benchmark

Following many prior papers published at top conferences like ICML (Li et al (2023); Mahlau et al. (2024)), NeurIPS (Carroll et al. (2020), Strouse et al. (2022), Yan et al. (2023), Sarkar et al. (2023), Myers et al. (2024), Liang et al. (2024)) and ICLR (Yu et al. (2023), Gessler et al. (2025)), we focus on the Overcooked benchmark as a challenging human-AI cooperation task. A core strength of Overcooked is that it enables real-time evaluation with actual human players, and shows that even for the simplest layouts coordinating with heterogenous and unpredictable human players requires a level of robustness that even state-of-the-art AI algorithms typically do not achieve. Since your review did not mention our human-study results (Figures 9-11), we would like to draw your attention to the fact that CEC achieves performance equivalent to state-of-the-art techniques in real human evaluations without using population-based training, a surprising finding that contradicts much of the human-AI coordination literature.

Novelty

While multi-task RL is well-established (Tobin et al., 2017), our work is, to our knowledge, one of the first to show that environment diversity can outperform partner diversity for human-AI coordination, challenging the dominance of Population-Based Training (PBT) in ZSC (Vinyals et al., 2019; Carroll et al., 2020; Strouse et al., 2022; Zhao et al., 2022; Sarkar et al., 2023; Liang et al., 2024). To support this claim, we reference Reviewer 11py’s comment, “I really like the idea of training the learning agent across variations of environments, which has not been investigated much, if at all, in the literature. I think the authors could also go as far as claiming cross-environment cross-play evaluation as a novel evaluation setup, which would serve as another contribution of the paper.”

Methodological details

Our task sampling strategy is described in Section 4 and Appendix A.1. Figure 12 details how we determine which items to sample to create solvable environments, and how this leads to procedural generation of over $1.16 * 10^{17}$ diverse, solvable coordination challenges by sampling wall structures and randomizing features like goals, plates, pots, and onions.

Our paper provides several insights into how and why training in multiple environments enhances coordination with many partners:

• Real Human Experiments: Quantitative and Qualitative assessments (Section 6, Figures 9-11, 16-21) indicate agents learn to "adapt to partners in service of completing the task."
• Heat maps (Figures 16-18) visualize the frequency of states covered by naive IPPO agents and CEC. CEC agents are able to adapt to novel partner strategies better by covering a greater distribution of states in self-play.
• Empirical game-theoretic analysis (Section 5) demonstrates CEC agents forming more robust equilibria with novel partners.
• Learning curves (Figure 7) illustrate how CEC agents improve differently on held-out tasks based on optimal strategy diversity.

Thank you again for your feedback, which will help us improve the final version of our paper.

Thank you for your interest in our work and recognizing the novelty of our training and evaluation approaches. We are glad you found our writing to be clear and our experiment section to be thorough. Your idea for framing cross-environment cross-partner evaluations as a novel contribution is one we will take on board in our revision of the paper, and we feel this will make our paper even stronger. We would now like to address some of their questions below:

Combining Task and Partner Diversity

Following your suggestion, we tested the impact of combining partner diversity with environment diversity using E3T under the CEC paradigm. We set the partner policy randomness to 0.5, consistent with human experiments and E3T’s original design. Results in simulation showed CEC-E3T performed worse than other models on Overcooked’s five original layouts (CEC=130.51, CEC-E3T=28.21) but outperformed CEC-Finetune on 100 held-out grids (CEC-FT=41.73, CEC-E3T=58.13). The attached learning curves reveal that noisy partners introduced additional training noise, which, combined with dynamic environments, likely requires larger networks and longer training times for convergence compared to vanilla CEC. While the vanilla E3T struggled in simulation, it excelled in human trials, outperforming all other models in terms of reward.

Fixed Task Oracle

In Figure 3 of the paper, we report the score in Dual Destination normalized by the theoretical maximum possible score which would be achieved if both agents spawned on the correct goals in the first timestep. Figure 3 plots these normalized cross-play rewards for CEC, FCP, and IPPO. To obtain the oracle score that you suggested on the Fixed Task, we calculate that an oracle agent that perfectly responds to either optimal strategy would require 3 steps of receiving a -1 step cost to move to the target location before receiving a (3 positive - 1 step cost) reward for 97 steps, equating to (2*97 - 3) = 191 / 200 = 0.955 normalized reward. CEC scored 0.931 normalized reward with a standard error of 0.013, indicating it underperforms the oracle’s cross play performance by about 2.5%.

CEC in Partially Observable Environments

We tested CEC with partial observability, by modifying the Dual Destination problem. We trained CEC, FCP, and IPPO using the same architectures and 300 million steps of training. The challenge of breaking handshakes when learning multi-agent policies is even more pronounced in the partially observable case, as agents may form arbitrary conventions to handle high uncertainty about the state of the world. From our results below, we find the same conclusions in the partially observable setting as we did in the fully observable results described in the paper: CEC has high cross-play performance in ZSC with other agents on novel environments (0.74), outperforming population based methods (0.61) and naive self-play (0.03). We believe this finding strengthens our paper, and thank the reviewer for providing this suggestion.

New experiments with no RNN

Following reviewer SmWr’s suggestions, we provide new experiments ablating the use of an RNN for CEC, where the RNN is used to provide a simple meta-learning algorithm (Wang et al, 2018; Rabinowitz et al, 2018). We find that in Overcooked and the Dual Destination problem, agents without recurrence failed to converge or adapt effectively. As shown in the learning curves for the Dual Destination problem, CEC with LSTMs successfully converged, while the non-recurrent version could not achieve positive rewards.

E3T performance difference in cross-play performance in simulation vs with humans

During training, E3T uses a partner policy defined as π_p = (1-ε) * learned_policy + ε * uniform_policy, maintaining entropy to induce greater strategy coverage without requiring diverse co-players like population-based methods. Yan et al. (2023) found ε = 0.5 most effective for collaborating with humans, which we used for our baseline. For AI zero-shot coordination, ε = 0.3 performed best, while ε = 0.0 excelled in low-exploration layouts like Forced Coordination.

The code will be open sourced, from the environment code to training scripts to the human evaluation interface, which supports running experiments on arbitrary Jax-based reinforcement learning environments.

We would like to thank the reviewer for their thoughtful and constructive feedback on our paper, and address your comments below:

ZSC vs Ad-hoc Teamplay

We sincerely thank you for clarifying the distinction between these two evaluation settings. From our understanding, our use of the empirical game theoretic analysis to compare how agents trained with different algorithms cooperate with each other, as well as our human study, both fall within the ad-hoc teamplay setting. However, we acknowledge that some of our experiments (particularly Figures 3, 5, and 7) address the ZSC setting, where we evaluate performance against the same algorithm with different random seeds. We will follow your suggestions to clarify our references and experimental analysis to distinguish between the two, and make a more nuanced characterization of the conclusions we can draw. We will also make sure to reference "Other-Play.”

CEC in Partially Observable Environments

Following your suggestions, we tested CEC with partial observability, by modifying the Dual Destination problem. We trained CEC, FCP, and IPPO using the same architectures and 300 million steps of training. The challenge of breaking handshakes when learning multi-agent policies is even more pronounced in the partially observable case, as agents may form arbitrary conventions to handle high uncertainty about the state of the world. From our results below, we find the same conclusions in the partially observable setting as we did in the fully observable results described in the paper: CEC has high cross-play performance in ZSC with other agents on novel environments (0.74), outperforming population based methods (0.61) and naive self-play (0.03). We believe this finding strengthens our paper, and thank the reviewer for providing this suggestion.

New experiments with no RNN

We include recurrent networks with CEC to enable a basic meta-learning algorithm (Wang et al, 2018; Rabinowitz et al, 2018). Following your suggestion, we tested whether it is possible for CEC to retain reasonable performance without using recurrent policies. In Overcooked and the Dual Destination problem, agents without recurrence failed to converge or adapt effectively. As shown in the learning curves for the Dual Destination problem, CEC with LSTMs successfully converged, while the non-recurrent version couldn't even achieve positive rewards.

Combining Task and Partner Diversity

Following reviewer mZRY and 11py’s suggestion, we tested combining partner diversity algorithms with environment diversity using E3T under the CEC paradigm. Results showed CEC-E3T performed worse than other models on Overcooked’s five original layouts (CEC=130.51, CEC-E3T=28.21) but outperformed CEC-Finetune on 100 held-out grids (CEC-FT=41.73, CEC-E3T=58.13). The linked learning curves reveal that noisy partners combined with dynamic environments introduced additional noise, suggesting larger networks and longer training times for convergence may be needed compared to vanilla CEC.

Learned Conventions in CEC

You raised an interesting point about learned conventions from CEC. As noted in our qualitative analysis, conventions like "move out of the way" emerged from CEC agents trained across diverse environments. Unlike fixed strategies (e.g., "red agent cooks onions while blue delivers"), this adaptability reflects a general form of learned convention that enhances transferability and generalization to novel scenarios.

We appreciate your attention to detail in pointing out the typos and suggestions for improving our Markov Game tuple definition. We will address these in our revision.

We thank you for recognizing the strengths in our work, particularly the demonstration of cross-environment coordination's benefits for ad-hoc coordination, the principled human-AI experiments showing strong transfer to human partners, and the creative use of empirical game-theoretic analysis to demonstrate cross-algorithm performance. We thank the reviewer again for their valuable feedback, which will help us improve the clarity and rigor of our paper.

Ref: Rabinowitz, N., Perbet, F., Song, F., Zhang, C., Eslami, S. A., & Botvinick, M. (2018, July). Machine theory of mind. In International conference on machine learning (pp. 4218-4227). PMLR. Wang JX, Kurth-Nelson Z, Kumaran D, Tirumala D, Soyer H, Leibo JZ, Hassabis D, Botvinick M. Prefrontal cortex as a meta-reinforcement learning system. Nat Neurosci. 2018 Jun;21(6):860-868. doi: 10.1038/s41593-018-0147-8. Epub 2018 May 14. PMID: 29760527.

We thank the reviewer for their detailed feedback and recognition of our work’s novelty and clarity. Below, we address key concerns and how we plan to integrate this feedback:

Experiments Beyond Overcooked

To help address your suggestions, we have conducted 3 new experiments showing that our method generalizes beyond Overcooked. First, we would like to highlight the “Dual Destination” environment (Section 4, Figures 2-3 of the submission), where rewards are only given when both occupy separate green squares. For the rebuttal, we have conducted new experiments in a partially observable version of this environment, and found that CEC once again achieved high cross-play performance in novel environments (0.74), outperforming population-based methods (0.61) and naive self-play (0.03).

Combining Task and Partner Diversity

Following your suggestion, we tested combining partner diversity algorithms with environment diversity using E3T under the CEC paradigm. Results showed CEC-E3T performed worse than other models on Overcooked’s five original layouts (CEC=130.51, CEC-E3T=28.21) but outperformed CEC-Finetune on 100 held-out grids (CEC-FT=41.73, CEC-E3T=58.13). The linked learning curves reveal that noisy partners combined with dynamic environments introduced additional noise, suggesting larger networks and longer training times for convergence may be needed compared to vanilla CEC.

New experiments without RNNs

Per reviewer SmWr, we ablated RNNs in CEC (used for meta-learning per Wang et al, 2018; Rabinowitz et al, 2018). In Overcooked and Dual Destination, non-recurrent agents failed to converge or adapt. Learning curves show CEC with LSTMs converged successfully, while non-recurrent versions could not achieve positive rewards.

DR generates unsolvable environments

We also appreciate the opportunity to clarify the distinction between CEC and standard domain randomization (DR). While CEC uses environment randomization, it employs a procedural generator to ensure all tasks are solvable, unlike naive randomization, which we found in our experiments often produces unsolvable layouts and poor learning signals. This mirrors the conclusions of the Overcooked Generalisation Challenge (Ruhdorfer et al., 2024), which shows that random/UED approaches often fail to generate solvable layouts or train agents capable of completing tasks. This highlights the need for structured procedural generation like ours.

Computational Feasibility of CEC

We acknowledged that CEC’s 3 billion training timesteps exceed the computational budget of prior work like CoMeDi for a single layout. However, Population-based training (PBT) scales poorly when the goal is to cooperate on many levels: training eight agents (5M steps each) plus an ego cooperator (10M steps) results in a 50-million-step budget per layout. Scaling to 100 layouts would require 5 billion steps but still only generalize to fixed levels. CEC generalizes better to thousands of unseen layouts with lower total compute costs (Figures 3,6,9), making it more efficient for broad adaptability. We also note that for our experiments, we ensure that all algorithms are able to use the same compute budget of 3 billion steps, and we find that CEC is able to better use this compute to improve performance.

We agree that formalizing procedural generators for naturalistic settings is a challenging direction for future work. One idea could involve combining program induction with procedural generation, as in WorldCoder (Tang et al., 2024), where LLMs generate OpenAI Gym-like environments with unknown dynamics. For CEC, inferred dynamics could be encoded into simulators like Unity before generating diverse scenes for RL training—a form of real-to-sim-to-real transfer (Torne et al., 2024).

CEC and related work

Thank you for highlighting MAESTRO and MADRID, which we will include in our related works section. MAESTRO complements our work by focusing on zero-sum settings, and showing that in that case, jointly considering environment and partner diversity is optimal, while focusing on either leads to suboptimal outcomes. While we used uniform sampling over grids irrespective of the agent’s abilities during training, we will add to our discussion about how future work can explore autocurricula for partners and tasks to improve sample efficiency through strategies such as regret minimization ala MADRID and MAESTRO.

Analyzing CEC’s Norms

The reviewer astutely questions how CEC learns human-like norms without explicit instructions. As noted in Section 6, we believe these norms reflect principles for cooperation in dynamic environments, such as “stay out of each other’s way.” These norms may be perceived as more human-like because humans also follow them.

Figures 13 and 14 show cross-play scores across all algorithms, forming the payoff matrix used to create Figure 8. The diagonal reflects results from Figure 6.