CMIMP: Effortlessly Achieving Diverse Population Training for Zero-Shot Coordination

• I find the paper to be generally hard to follow especially the math equations. Many variables are not defined, e.g., $N$ at line 172, $v$ in Theorem 1. Some variables are used confusingly, e.g. $u_j$ and $u_i$ in Eq.3, $N$ is Table 2. From the text, $i$ and $j$ have totally different meaning (agent index vs number of transition). $i$ and $j$ are being compared in Eq. 5 despite having different meaning.
• Table 1 is not fact-based nor scientific but a subjective judgement from the authors. I suggest the authors to either remove or justify the information with facts and citation.
• Eq. 5 needs a lot more motivation and introduction. It is not clear what is the reason we should use this specific form to approximate the mutual information
• Regarding the approximation in Eq. 5, it is not clear to me at all why taking the mean of action probability or q-values gives the approximation of the mutual information.
• The instantiation in Eq.8 gives me some understanding of the method, instead of initial the derivation, which should not be the case. Still, I find Eq. 8 to be cryptic. Partly because $i$ and $j$ are being compared despite their difference. But my rough understanding is that the objective drives different agent to have different q-values given the same observation. Being different in this case comes from the fact that the objective tries to reduce the q-values of all other agents, which is also my guess for the notation $i \neq j$, which apparently seems to well in Hanabi.
• I do not find the proposed architecture related to Meta-RL.
• The proposed training objective is supposed to "guarantee" diversity, but Eq. 8 optimizes for different Q-values which does not guarantee different behaviors.
• The paper does not ablate the inclusion of the proposed training objective. With the given architecture, I imagine that using TrajeDiv or MEP diversity constraints should also work well but is not shown in the paper.
• The paper will benefit from also running the same experiment as in Table 5 but with $\alpha=0$ to show effectiveness of the proposed method across training setups.

• The paper lacks theoretical justification for the update method used for optimizing mutual information. In the main theoretical result, Theorem 1, it shows that one specific update rule specified in equation (6) will increase $I_j$ for some component $j$, but it does not show that the method will converge to the optimal mutual information when the update stops.

• The authors suggest similarities between population-based training and meta-RL, yet the connection is vague. While meta-RL focuses on fast adaptation to new tasks, CMIMP uses a multi-headed architecture that is more aligned with shared parameter frameworks than true meta-RL. A more explicit clarification on how CMIMP draws on meta-RL principles, or how it might benefit from these principles, would strengthen the narrative.
• The experimental section lacks crucial details, including the evaluation protocol, the number of random seeds, and the number of players in Hanabi (especially in Table 3). Additionally, it is unclear which algorithms were used for the 1ZSC-XP score. These omissions make it difficult to assess the results.
• The experiments are only conducted on a single environment. The authors should consider evaluating CMIMP on a broader set of environments, such as Overcooked.
• The paper does not discuss CMIMP’s sample efficiency. Evaluating how CMIMP compares to other methods in terms of sample efficiency (not just final convergence performance) would be valuable for understanding its practical viability.
• Although, E3T [1], a modern method in ZSC is mentioned in the related work, it is not included in the experiments. A comparison with E3T might be relevant to show the effectiveness of CMIMP.
• The speed and resource consumption results in Figure 3 are surprising. CMIMP is expected to be slower and require more memory with an increase in population size, because of the additional mutual information maximization objective, which requires all observations to be passed through all agent heads.
• While using mutual information as a diversity metric appears novel in ZSC, it is commonly applied in diversity maximization in MARL (e.g., [2]). Citing these related works would better position CMIMP within the existing literature.

1] Yan, X., Guo, J., Lou, X., Wang, J., Zhang, H. and Du, Y., 2024. An efficient end-to-end training approach for zero-shot human-AI coordination. Advances in Neural Information Processing Systems, 36.

2] Li, C., Wang, T., Wu, C., Zhao, Q., Yang, J. and Zhang, C., 2021. Celebrating diversity in shared multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34, pp.3991-4002.

Methodology (Section 3)

The presentation of the problem setup in Section 3.1 requires significant revision for clarity and rigor. The meta-agent can be formally presented as a tuple containing all mentioned modules, with definitions and ranges for functions and variables. Variables $h$ and $c$ in equation 1 are used without prior definition. Some of the preliminary terms like the $Q$-function are well-known concepts but it's better to provide a definition using your own notation as they are frequently used (in Sections 3.2 and 3.3) in presenting the core methodology of the framework.

The paper requires a more thorough explanation of how the meta-agent's hierarchical architecture is reducing the number of trainable parameters. There is only a high-level statement (lines 171-173) in this regard which is not presented well to justify the central claims of the paper.

Experiments (Section 4)

All the results presented in the paper are only based on the Hanabi environment. The current experiment setup is good, but the gain of performance in a single environment can always be misleading.

The authors mention that the CMIMP framework is compatible with both value-based methods and policy gradient-based methods and mention two possible forms for the function $F$ in Section 3.2. The choice of value-based methods for the Hanabi environment is backed by prior work, but there are no experiments in the main section or appendix for policy-gradient-based methods. The authors should either add experiments for policy-gradient methods such as PPO on a different environment or narrow the framework's scope to focus exclusively on value-based methods.