R3DM: Enabling Role Discovery and Diversity Through Dynamics Models in Multi-agent Reinforcement Learning

We sincerely thank the reviewer for their detailed and thoughtful feedback, which has greatly contributed to improving the scientific rigour of this paper. We detail our responses to the questions below and add the required ablation studies.

1. Intrinsic Reward:

We clarify the intuition behind our intrinsic rewards. The rewards encourage agents to diversify their future behaviors, conditioned explicitly on their assigned roles. Formally, it represents the difference between the entropy of future trajectories and the conditional entropy of these future trajectories given specific roles. Therefore, the intrinsic reward inherently promotes diversity by increasing the overall trajectory entropy, ensuring diverse future behaviors while simultaneously maintaining the coherence and alignment of behaviors with respect to roles.

2. Clarification on the MI Objective :

The MI objective in R3DM is designed to ensure that an agent’s current role influences its future behavior while being grounded in its prior observations. This objective quantifies the dependency between the concatenated observation-action trajectory (which includes both the agent’s observation-action history and its future trajectory) and its role. Intuitively, maximizing this MI objective ensures that the roles derived from observation-action histories can effectively predict future actions and observations. This, in turn, enables better specialization within the team, which facilitates more sample-efficient multi-agent learning, as demonstrated in our experiments.

3. Ablations:

We conduct 3 ablations on the design choices. The plots are in the link. https://drive.google.com/file/d/1b4IkSXhNhXFedi2-d8K8tTEJjCVz4hey/view?usp=sharing

a) Impact of the reward imagination: We analyze the impact of the number of imagination steps in the role-conditioned future trajectory, which is used to predict intrinsic rewards. In R3DM, we use a single imagination step to compute intrinsic rewards. We observe no statistically significant improvements when increasing the number of imagination steps to 2. However, with a higher number of imagination steps (e.g., 5 and 10), we observe a degradation in performance (more pronounced for 10 steps). We believe this degradation is due to compounding errors in the predictions by the model, which is conditioned solely on an agent’s past observations and actions. These errors lead to intrinsic rewards with higher variance and bias, ultimately degrading learning.

b) Impact of the number of roles

We vary the number of role clusters used in R3DM progressively from 2 to 8 for the 3s5z_vs_3s6z environment. We don’t observe a significant difference in final performances when varying the number of role clusters. We observe that the learning with 3 role clusters is more sample efficient compared to other role clusters.

c) Impact of Contrastive Learning (CL)

We compare the influence of intrinsic rewards and CL on the performance of R3DM. The variant of our R3DM without intrinsic rewards is equivalent to ACORM, as our method builds upon it. Next, we evaluate a variant of our algorithm that includes intrinsic rewards but excludes CL, which is used to enable more distinct role clusters. We observe that this degrades performance CL. Interestingly, this variant still outperforms ACORM, underscoring the significant impact of our proposed intrinsic rewards. These rewards enhance the entropy of future trajectories based on roles while reducing their conditional entropy with respect to intermediate role representations.

4. Balancing task rewards with exploration diversity

The reviewer raises an insightful point regarding scenarios where it would be optimal for multiple agents (e.g., drones) to collaboratively target the same objective due to task difficulty. Our proposed approach accounts for such task-specific complexities by integrating intrinsic rewards with task-specific rewards. The balance between these two reward types is controlled by the hyperparameter \alpha, which can be tuned based on the requirements of the specific task. This design allows our method to flexibly adjust the trade-off between promoting diversity and fostering cooperation, depending on the inherent difficulty and coordination demands of the task.

5. Using a single dynamics model:

The reviewer rightly points out that employing two separate dynamics models (one role-conditioned and one role-agnostic) increases training computational overhead. Importantly, this overhead is confined to the training phase, and dynamics models are not needed during the testing or inference phase. We have explored the possibility of using a single dynamics model with multiple forward passes using different sampled role embeddings to compute intrinsic rewards. However, empirically, we observed this approach to be computationally slower due to the necessity of multiple forward passes with different role embeddings when computing the intrinsic reward.

We thank the reviewer for their positive feedback on the paper and for raising interesting and relevant questions. We aim to highlight this clearly in the camera-ready version of the paper.

1. Impact and Details on Contrastive Learning

Contrastive learning plays a crucial role in R3DM by deriving intermediate role embeddings, essential for enabling role-specific intrinsic rewards that drive diverse future behaviors. Contrastive learning is conducted throughout the training process. Specifically, following the training protocol outlined in ACORM (Hu et al., 2024), contrastive learning updates occur after every 100 episodes of data collection. We refer the reviewer to the ablation study under the rebuttal for reviewer STTJ for more details.

2. Compute Cost Increase

The proposed R3DM algorithm does not incur additional computational overhead during the testing or deployment phase, as only the learned policy is executed. The overhead introduced by R3DM arises solely during training, specifically due to the learning of an additional dynamics model required for computing intrinsic rewards. This extra step significantly enhances coordination through role-specific behavior. This extra step results in the increases moderately increases the training time of R3DM by 50-75% compared to the baselines. With more efficient implementations of the dynamics model on frameworks such as Jax, we believe that the training run-time can be optimized further.

3. Diversity of Strategy

Our approach inherently encourages strategic diversity as a mechanism to optimize coordination objectives, which in turn improves sample complexity and efficiency. Qualitative analyses provided in the paper illustrate that R3DM successfully facilitates the emergence of diverse and sophisticated strategies. For instance, agents have learned distinct tactics such as deliberately distracting subsets of enemy agents to weaken their overall strength, thereby enabling other teammates to effectively neutralize remaining threats.

4. Have the authors compared their approach against using a shared base policy with small single layers for individual agents?

Yes. In our experiments, the baseline QMIX algorithm inherently implements a shared local critic structure that agents use to formulate policies for choosing the desired actions. This implementation persists in R3DM, where we use a shared local critic and the shared role embedding network across multiple agents. We provide the anonymized code here https://anonymous.4open.science/r/R3DM-F1A0/README.md.

We sincerely thank the reviewer for their time in providing detailed feedback on the paper to better highlight the key contributions and novelties of the paper. We detail our responses to the questions below.

1. Clarification of the contributions

Our primary contribution lies in introducing a novel information-theoretic objective specifically tailored for role-based multi-agent reinforcement learning (MARL). We theoretically demonstrate that this objective can be naturally decomposed into two complementary sub-objectives. One that derives intermediate role embeddings from observation-action histories, which can be effectively optimized using contrastive learning, and another that refines these embeddings by aligning them more closely with future behaviors through intrinsic rewards. This decomposition ensures that roles derived from past interactions are not only compact representations of historical data but also predictive of future actions, thereby enabling role-specific coordination and specialization.

While our framework incorporates contrastive learning and clustering techniques borrowed from ACORM, our approach subsumes this to optimize the broader optimized objective. Hence, they are integral to derive these intermediate role embeddings to optimize our proposed objective. For the sake of completeness and transparency, we have included them in the methodology section of our paper. To avoid any potential confusion, we will make this distinction clearer in the final camera ready version of the paper by explicitly identifying these elements as preliminary or existing work.

2. Source Code

We have uploaded and anonymized source code with the necessary data used for the plots in the paper and rebuttal (ablation studies for reviewer STTJ) to the best of our ability here: https://anonymous.4open.science/r/R3DM-F1A0/README.md

3. Clarification with respect to Entropy-based Methods

[1] demonstrates that training a population of policies through adversarial diversity improves robustness by ensuring cross-policy compatibility. However, their approach requires maintaining multiple policies per agent to achieve competitive performance across permutations of partner policies, which increases complexity and computational overhead. We believe that this approach solves a related problem in the field of adhoc-teaming [2]. In contrast, our single-policy learning paradigm focuses exclusively on rapid task-reward convergence by leveraging role-driven trajectory diversity rather than optimizing for robustness across diverse partner policies. Entropy-based exploration methods such as CDS (which we benchmark against) are limited to static role assignments based on the identities of agents, which result in fixed behavior distributions that fail to adapt to evolving team dynamics. R3DM addresses this limitation by employing a mutual information objective (Theorem 4.1) to induce role emergence dynamically from observation-action histories. This allows agents to learn complementary roles based on the similarity or divergence of their trajectory histories. Additionally, R3DM’s intrinsic rewards are designed to capture diversity, specifically in future expected behaviors conditioned on the agent’s role, ensuring that roles evolve naturally and adaptively during interactions with the environment.

4. Test Returns

The "test returns" mentioned in line 353 refer to cumulative rewards accumulated during testing episodes. We will clarify and correct this terminology in the final version of the paper to avoid ambiguity.

5. Changing of role-clusters

Roles dynamically change throughout the episode because they depend on each individual agent’s evolving observation-action history. Thus, as agents interact with the environment and their experiences diverge over time, their corresponding roles naturally adjust, causing fluctuations in role cluster assignments, even for agents that become inactive or "dead.

6. Error Bars

The error bars depicted in the learning curve figures represent the standard deviation computed across the 5 different random seeds used in our experiments. We will explicitly state this detail in the final manuscript to clarify the interpretation.

[1] Cui, Brandon, et al. "Adversarial diversity in hanabi." The Eleventh International Conference on Learning Representations. 2023.

[2] Mirsky, Reuth, et al. "A survey of ad hoc teamwork research." European conference on multi-agent systems. Cham: Springer International Publishing, 2022.