Attention-Guided Contrastive Role Representations for Multi-agent Reinforcement Learning

Q1: There is no reasonable explanation or formula for the promotion of credit assignment by attention, and the paper only demonstrates the effectiveness of the method through experiments.

A1: Thank you for your insightful comments. Throughout the paper, we have provided explanations for the promotion of more expressive credit assignment in many places. For example, the first paragraph of Sec. 2.2 (top, page 5) presents the line our research motivation for introducing the attention mechanism as "Popular CTDE algorithms, such as QMIX, realize behavior coordination across agents via a mixing network that estimates joint action-values as a monotonic combination of per-agent values, and the mixing network weights are conditioned on the system’s global state. Naturally, it is interesting to incorporate the learned role information into the mixing network to facilitate skillful coordination across roles. The simplest approach is to concatenate the global state and role representations for generating mixing network weights, while it fails to exploit the internal structure to effectively extract correlations in the role space. Fortunately, the attention mechanism aligns perfectly with our intention by prompting the global state to attend to learned role patterns, thus providing more expressive credit assignment in value decomposition." On the other hand, the visualization in Sec. 3.3 clearly illustrates that the global state has successfully attended to the learned role patterns, which in turn provides evidence for our attention-guided promotion of credit assignment.

Q2: Does GRU encoding $\bf{s}$ play a more significant role in the effect?

A2: We have included an additional ablation baseline "ACORM_w/o_MHA (Vanilla)" that removes both the attention module and the additional GRU-MLP structure. In this way, the input to the mixing network is only the current global state \bf{s}$$^t, which is kept the same as the original QMIX algorithm. In Sec. 3.1 of the updated paper and the updated Appendix F, it can be observed that ACORM_w/o_MHA (Vanilla) obtains very similar performance compared to ACORM_w/o_MHA. It can be concluded that the effectiveness of our attention module comes from prompting the global state to attend to role representations other than the GRU encoding of the global state trajectory.

Q3: How do you know the state encoding after this attention, input to the mix network can have an impact on credit assignment?

A3: As stated in A1, the idea of our attention-guided role coordination comes from QMIX's value decomposition. QMIX realizes behavior coordination across agents (i.e., credit assignment) via the mixing network whose weights are conditioned on the system’s global state. Intuitively, if we could condition the mixing network weights on extra role representations, we might realize more efficient role coordination for better credit assignment. Fortunately, the attention mechanism aligns perfectly with our intention by prompting the global state to attend to learned role patterns, thus providing more expressive credit assignment in value decomposition. Moreover, ablation studies (Sec. 3.1 and Appendix F) and visualization of role coordination (Sec. 3.3) have also verified that our attention module indeed has a positive impact on credit assignment.

Dear reviewer,

We were wondering if our response and revision have resolved your concerns.

In our responses, we have provided the source code and significantly extended the experimental evaluation. We conduct ablation studies with the same network size, and add another ablation baseline to identify the respective effects of the attention module and the state trajectory encoding. We have also provided detailed explanations and insights on the attention-guided role coordination for more expressive credit assginment.

If our response has addressed your concerns, we would be grateful if you could re-evaluate our work.

If you have any additional questions or suggestions, we would be happy to have further discussions.

Best regards,

The Authors

Dear Reviewer jtxX,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair

Q1: Experimental evaluation on more multi-agent testbeds.

A1: In the original version, we evaluated our method on six representative SMAC maps. Following your suggestion, we have conducted additional experiments on more SMAC maps and on Google research football environments, and provided the extended results in the updated Appendices B and C. The results on massively extended experiments are generally consistent with our previous conclusion that ACORM obtains the best performance in most scenarios and outperforms baselines by a larger margin on harder tasks.

Q2: About reproducibility and some experimental details (e.g., $T_{cl}$).

A2: Thank you for your careful reading. We have provided the source code of ACORM in the updated supplementary material to ensure reproducibility. In the original appendices, $T_{cl}$ is the time interval for updating contrastive loss as shown in Algorithm 1 and Tables 2 & 4. To make these details clearer, we have modified these notations to be consistent across all appendices.

Q3: More sufficient ablation experiments, since ACORM_w/o_MHA still has an additional MLP and GRU for the global state.

A3: Thank you for your helpful advice. We have included an additional ablation baseline "ACORM_w/o_MHA (Vanilla)" that removes both the attention module and the additional GRU-MLP structure. In this way, the input to the mixing network is only the current global state \bf{s}$$^t, which is kept the same as the original QMIX algorithm. In Sec. 3.1 of the updated paper and the updated Appendix F, it can be observed that ACORM_w/o_MHA (Vanilla) obtains very similar performance compared to ACORM_w/o_MHA. It can be concluded that the effectiveness of our attention module comes from prompting the global state to attend to role representations other than the representation learning of the state trajectory via GRU.

Q4: The number of baselines is relatively tiny. Why not use CDS as a baseline, since you mentioned it in the paper?

A4: As mentioned in our original paper, there are a large number of related methods that involve agent heterogeneity and role emergence. For a moderate demonstration, we chose the basic QMIX and the four most relevant methods as our five baselines. We skipped some methods that generally perform worse than the selected baselines. For example, ROMA and RODE are both classical role-based methods, and RODE generally outperforms ROMA in the literature. Hence, we compare ACORM to RODE only. Further, we have included CDS as an additional baseline in the updated paper to make our evaluation more persuasive. Extended experimental results are consistent with our previous conclusion, and our method outperforms CDS in most scenarios.

Q5: More related works that infer the role of an agent based on its trajectory.

A5: We agree with you that inferring the role of an agent based on its trajectory has been investigated in many previous works, such as those role-based methods in our original paper. We have indeed included [2] as a Role Emergence method in the Related Work section (top, page 9) of the original paper, and we will also include [1] in our related work in the updated paper to make our literature review more complete and comprehensive. Compared to existing role-based methods, our method introduces two innovative components, i.e., the contrastive learning formulation for learning discriminative role representations and the attention mechanism to coordinate learned role representations for more expressive credit assignment, and our method obtains SOTA performance on most experiments compared to them.

[1] Cao, Jiahang et al. LINDA: multi-agent local information decomposition for awareness of teammates, Science China Information Science 2021.

[2] Yang, Jiachen et al. Hierarchical Cooperative Multi-Agent Reinforcement Learning with Skill Discovery, AAMAS 2020.

Dear reviewer,

We were wondering if our response and revision have resolved your concerns. We are grateful for your recognition of the suitability and organization of our framework, the detailed case studies, and the soundness of the ELBO derivation.

In our responses, we have provided the source code and significantly extended experimental evaluation, including experiments on more SMAC maps and the Google research football testbed, an additional ablation study, and another baseline method. We have included more related work on inferring an agent's role based on its trajectory.

If our response has addressed your concerns, we would be grateful if you could re-evaluate our work.

If you have any additional questions or suggestions, we would be happy to have further discussions.

Best regards,

The Authors

Dear Reviewer 1yPm,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair

Q4: ACORM's correct execution under the CTDE paradigm.

A4: As stated in A2 and A3, we use the same mixing network architecture as QMIX and just modify the input to the mixing network. During centralized training, we can access the global state $\bf{s}$ and role representations $z_i$ of each agent $i$. After prompting the global state to attend to role representations, we concatenate the attention output and the original global state as the input to the mixing network for value decomposition (where the hypernetworks produce the weights and biases for mixing network layers). During execution, each agent can make decisions according to the individual Q-network only, without access to the global state or others' role representations, i.e., decentralized execution. As we have emphasized in our original paper, ACORM strictly follows the CTDE paradigm and is fully compatible with CTDE methods.

Q5: In Fig. 4(b), agent 0 is even further from other points in its cluster compared to agent 5.

A5: We cluster agent embeddings in an unsupervised way, and there is no ground truth to indicate the oracle cluster assignment. The effectiveness of our method relies on the mechanism of whether agents with similar behavior patterns tend to be grouped into the same cluster, and whether the contrastive learning objective can push points within the same cluster closer to each other and promote different clusters to be notably separated. Fortunately, our method meets this condition in most cases, as shown in Figs. 4-5. Several outliers, e.g., Agent 0 is even further from other points in its cluster compared to Agent 5 in Fig. 4(b), will not have a fundamental impact on the effectiveness of our method.

Q6. In Figs. 4(b) and 4(c), it seems that just the agent embedding alone might form good coordination teams.

A6: Thank you for your careful reading. Your observation totally matches the experimental analysis in our original paper (middle, page 7) as: "Initially ($t=1,12$), all agent embeddings tend to be crowded together with limited discrimination, and the K-means algorithm moderately separates them into several clusters. Via contrastive learning, the acquired role representations within the same cluster are pushed closer to each other, and those in different clusters are notably separated. At a later stage ($t=40$), agent embeddings are scattered widely throughout the space with a good clustering effect. This phenomenon indicates that the system has learned effective role assignment with heterogeneous behavior patterns. Then, the role encoder transforms these agent embeddings into more discriminative role representations." The discriminative patterns of agent embeddings in later stages could benefit from the early training of contrastive role representations. While the agent embedding alone might form good coordination teams at later stages, our contrastive learning module can help yield better coordination teams with more discriminative patterns. Ablation studies in Sec. 3.1 and Appendix F have also verified this point.

Q1: Reproducibility and more experimental evaluation.

A1: We have provided our source code in the updated supplementary material to ensure reproducibility. In the original version, we evaluated our method on six representative SMAC maps. Following your advice, we have conducted additional experiments on more SMAC maps and on Google research football environments, and provided the extended results in the updated Appendices B and C. The results of massively extended experiments are generally consistent with our previous conclusion that ACORM obtains the best performance in most scenarios and outperforms baselines by a larger margin on harder tasks.

Q2: More related work of introducing the attention mechanism in the mixing network.

A2: We agree with you that ACORM is not the first work to introduce the attention mechanism in the mixing network. As stated in Related Work of the original paper (bottom, page 9), attention has been widely applied to MARL domains for various utilities, including Qatten. However, our method differs largely from Qatten in how to use attention in the mixing network. Qatten proposes a novel value decomposition formulation that transforms individual Q-values to $Q_{tot}$ with a multi-head attention mechanism. That is, Qatten introduces a new mixing network structure and uses the attention mechanism within the mixing network. In contrast, we introduce the attention mechanism to incorporate role representations into the mixing network's input for better value decomposition. That is, we maintain the same mixing network architecture as QMIX and use the attention mechanism at the input of the mixing network. Further, our method mainly contains two innovative modules, i.e., the contrastive learning formulation for learning discriminative role representations and the attention mechanism to coordinate learned role representations for more expressive credit assignment. The two components work as a whole to efficiently incorporate the role information into MARL.

Q3: The compatibility with the IGM principle of ACORM.

A3: As claimed in A2, ACORM uses the same mixing network architecture as QMIX. Hence, ACORM's compatibility with the IGM principle is the same as that of QMIX. Formally, with QMIX's mixing network architecture, ACORM can enforce the monotonicity through the constraint on the relationship between $Q_{tot}$ and each $Q_a$ as $\frac{\partial Q_{tot}}{\partial Q_a}\ge 0, \forall a\in A$, and can naturally satisfy the IGM principle. Modifying the input to the mixing network (other than the mixing network structure itself) will not influence the monotonicity and the compatibility with the IGM principle.

Dear reviewer,

We were wondering if our response and revision have resolved your concerns. In our responses, we have provided the source code and significantly extended experimental evaluation, including experiments on more SMAC maps and the Google research football testbed, an additional ablation study, and another baseline method. We have clarified ACORM's full compatibility with the IGM principle and the CTDE paradigm.

We are really looking forward to discussing this with you so that we could continually improve our work. Your feedback is critical to us.

If our response has addressed your concerns, we would be very grateful if you could re-evaluate our work.

Best regards,

The Authors

Dear Reviewer,

Apologies for reaching out once more, as today might be the final day of the discussion phase.

We were wondering what could we do to further address your concerns for a higher rating. Our first guess on the reason that prevents you from a higher rating is that you might share the concerns on the reproducibility and experimental evaluation (also raised by two Reviewers 53j5 and 1yPm). We would like to raise your attention that Reviewer 1yPm replied to us very recently, approved the significantly extended experimental results, and increased the rating. We thus hope our responses to them could also solve your potential concerns.

At the same time, we will be extremely happy if you still have any additional concerns to share with us. Your opinion is critical to us. We will try our best to address your concerns.

Best regards,

The Authors

Dear Reviewer WWCs,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair

Q6: How is the setting cluster_num=3 applied on the map 2c_vs_64zg? Besides, why didn't the performance decline since it forces the strategies of each agent to diverge?

A6: The number of clusters is less or equal to the number of agents. We have specifically pointed out that the cluster number is 2 on the 2c_vs_64zg map in the updated Appendices B and D. When each agent represents a distinct role cluster that forces the strategies of each agent to diverge, the performance may decline compared to those with effective team composition (e.g., K=5 compared to K=2,3,4 on the map 5m_vs_6m). However, it could still obtain improved performance compared to most baselines in Fig. 2, since it can explore cooperatively heterogeneous behaviors via our discriminative role representation learning.

Q7: Relationship between MHA and CL.

A7: We use CL to promote discriminative role representation learning, and utilize an MHA mechanism to coordinate learned role representations for more expressive credit assignment. The two innovative components work as a whole to efficiently incorporate the role information into MARL to promote behavior heterogeneity, knowledge transfer, and skillful coordination across agents.

Q8: Clarify the difference between clustering directly on agent embeddings and on role representations, and further elucidate the necessity of obtaining discriminative representations through contrastive learning.

A8: As we clarified in the 'Negative Pairs Generation' part of the original paper (bottom, page 4), we perform clustering according to agent embeddings, not role representations. First, we obtain agent embedding by encoding the agent's trajectory as $e_i^t=f_{\phi}(o_i^t, a_i^{t-1}, e_i^{t-1})$, and the agent embedding is transformed into role representation by a role encoder as $z_i^t=f_{\theta}(z_i^t|e_i^t)$ where the role encoder $\theta$ is optimized by a contrastive learning objective. As shown in the original paper, we have elucidated the necessity of obtaining discriminative representations through contrastive learning in Sec. 2.1. Further, experimental performance and ablation studies in Sec. 3.1 have also verified the effectiveness of our constrastive learning module, and visualization in Sec. 3.2 has clearly shown that the role encoder is capable of transforming agent embeddings into more discriminative representations.

Q9: There are some typos and the writing of the paper needs some improvement.

A9: Thank you for your careful advice. We have fixed several typos and improved the writing as much as we can.

Q10: Additional MAPPO experiment on 3s5z_vs_3s6z.

A10: In the original version, we evaluated MAPPO-based ACORM on three representative SMAC maps. Due to the very limited time for rebuttal revision, the performance of MAPPO-based ACORM on more SMAC maps is not updated. Debugging MAPPO-style algorithms is generally more time-consuming, since it involves many hyperparameters and empirical tricks. We hope that the extended experimental evaluation could demonstrate adequate persuasiveness of our method, and we are rushing to complete the evaluation results and trying to update them before the rebuttal deadline.

Q1: Lack of source code and experiments.

A1: We have provided the source code of our paper in the updated supplementary material to ensure reproducibility. In the original version, we evaluated our method on six representative SMAC maps. Following your suggestion, we have conducted additional experiments on more maps within SMAC and on Google research football environments, and provided the extended results in the updated Appendices B and C. The results of massively extended experiments are generally consistent with our previous conclusion that ACORM obtains the best performance in most scenarios and outperforms baselines by a larger margin on harder tasks.

Q2: It is suggested to have more agents and experiments with larger values of K.

A2: Following your suggestion, we have conducted additional experiments with larger numbers of agents, including the maps of 30m and bane_vs_bane. Results can be found in the updated Appendix B.

Q3: It is recommended to conduct ablation studies with the same network size, as our model is added with a global state GRU and an MHA structure.

A3: Following your advice, we have kept the network size of ACORM the same as all ablations for a fair comparison. Further, we have added another ablation baseline to identify the respective effects of the attention module and the state trajectory encoding via GRU. The updated experimental results are shown in Sec. 3.1 of the revised paper and in the revised Appendix F. It can be observed that the ablation performance is generally consistent with that of our original paper.

Q4: Analysis on the visualization of agent embeddings and role representations via t-SNE.

A4: We agree with you that after t-SNE reduction, the distance between clusters does not match the real absolute distance. However, what we care about is the relative distance between points or clusters. The t-SNE is a widely used dimensionality reduction technique for embedding high-dimensional data for visualization in a low-dimensional space, in such a way that similar objects are modeled by nearby points and dissimilar objects are modeled by distant points with high probability. Hence, the distance between points or clusters via t-SNE reduction can reflect their relative distances in the original space.

Q5: Why dead Marines 5 and 8 are clustered into different classes in Fig. 4(c).

A5: We cluster agent embeddings in an unsupervised way, and there is no ground truth to indicate the Oracle cluster assignment. The effectiveness of our method relies on the mechanism that agents with similar behavior patterns tend to be grouped into the same cluster. Fortunately, our method meets this condition in most cases, as shown in Figs. 4-5. Several outliers, e.g., dead Marines 5 and 8 are clustered into different groups, will not have a fundamental impact on the effectiveness of our method. As we can see in Fig. 4(c), Marines {2,3,4,6,7} are in the offense team, Marauders {0,1} and Marine 5 form one dead group, Marine 8 alone represents another dead group, and Medivac 9 alone forms the rescue group.

Dear reviewer,

We were wondering if our response and revision have resolved your concerns. In our responses, we focus on ensuring the reproducibility of our work and significantly extending empirical evaluation, including experiments on more SMAC maps and the Google research football testbed, an additional ablation study, and another baseline method.

If our response has addressed your concerns, we would be grateful if you could re-evaluate our work.

If you have any additional questions or suggestions, we would be happy to have further discussions.

Best regards,

The Authors

Dear reviewer,

We are rushing to complete the evaluation results all these days. Promisingly, we have updated more experimental results on MAPPO-based ACORM in Appendix D (pages 19-20 in the updated main PDF), with a total of six representative maps: 2s3z, 3s5z, 5m_vs_6m, MMM2, corridor, and 3s5z_vs_3s6z. Akin to the QMIX-based version, ACORM improves the performance by the largest margin on super-hard maps that demand a significantly higher degree of behavior diversity and coordination. Extended experimental results consistently demonstrate ACORM’s superior learning efficiency in complex multi-agent domains.

If our response has addressed your further concerns, we would be grateful if you could re-evaluate our work again. We will be extremely happy if you still have any additional concerns to share with us. We will try our best to address your concerns.

Best regards,

The Authors

Dear Reviewer,

Apologies for reaching out once more, as today might be the final day of the discussion phase.

We were wondering what could we do to further address your concerns for a higher rating. Our first guess on the reason that prevents you from a higher rating is that you might share the concerns on the reproducibility and experimental evaluation (also raised by two Reviewers WWCs and 1yPm). We would like to raise your attention that Reviewer 1yPm replied to us very recently, approved the significantly extended experimental results, and increased the rating. We thus hope our responses to them could also solve your potential concerns.

At the same time, we will be extremely happy if you still have any additional concerns to share with us. Your opinion is critical to us. We will try our best to address your concerns.

Best regards,

The Authors

Dear Reviewer 53j5,

As we progress through the review process for ICLR 2024, I would like to remind you of the importance of the rebuttal phase. The authors have submitted their rebuttals, and it is now imperative for you to engage in this critical aspect of the review process.

Please ensure that you read the authors' responses carefully and provide a thoughtful and constructive follow-up. Your feedback is not only essential for the decision-making process but also invaluable for the authors.

Thank you,

ICLR 2024 Area Chair

Dear reviewers,

We were wondering if our responses and revision have resolved your concerns. In our responses, we have significantly extended experiment results and provided the source code based on your suggestions. Please let us know if we have addressed your concerns. We are more than delighted to have further discussions and improve our manuscript.

Best regards,

The Authors