HyperMARL: Adaptive Hypernetworks for Multi-Agent RL

We thank reviewer QQ2N for their thorough and thoughtful review. We appreciate the reviewer's positive feedback on our novel insight into gradient decoupling, our comprehensive experimental evaluation, and our paper's clarity. We address your points as follows:

1. SePS[1] - Related work (W1, Q2)

We thank the reviewer for this point. SePS [1] is a parameter sharing method that uses an autoencoder during pre-training to cluster agents based on their transitions and shares parameters between these agent groups. This differs from our work, as we 1) do not use a pre-training phase to collect transitions and cluster agents, 2) do not maintain $K$ per-agent networks, but rather generate these networks on the fly based on agent embeddings, and 3) we use agent-conditioned hypernetworks that explicitly decouple observation- and agent-conditioned gradients (Section 4.3). This decoupling helps mitigate gradient interference between agents. Although we briefly mentioned SePS in our related work, we will improve our discussion of this work.

New Baseline and Environment: Furthermore, since this work was mentioned as closely related by two reviewers, we conducted additional experiments comparing HyperMARL to SePS. We use their original source code and hyperparameters and evaluate on all their Blind Particle Spread settings (v1-4), which include up to 30 agents.

IQM final total reward (with 95% confidence intervals) across 5 seeds on the BPS environments from [1], Bold indicates the highest IQM score; * indicates scores whose 95% confidence intervals overlap with the highest score.

Takeaway: The results highlight HyperMARL's competitive performance. In all four scenarios, its 95% confidence intervals overlap with SePS and NoPS, indicating it is statistically on par with these baselines. Notably, SePS relies on a pre-training clustering phase, whereas HyperMARL learns end-to-end without pre-training.

2. Agent-Specific Parameters Integrated into the Agent’s Networks (W2, Q5)

A hypernetwork generates the agent-specific parameters $\theta^i$ that form all the weights (and biases) of policy $\pi_{\theta^i}$ (can also be seen in Fig. 1).

For RNNs, HyperMARL maintains the same architecture as for feedforward methods, generating only the actor and critic feedforward weights, not the GRU weights (we also mention this in G.2, but we will highlight this in the main text). The RNN hidden state is then fed into the networks as input, as is common in recurrent architectures in RL.

3. Could specialisation lead to redundant diversification policies? Could specialisation lead to a decrease in adaptability? (Q1)

Thank you for these insightful questions. Our opinion is that enforced specialisation, for example enforcing a diversity loss/objective, could lead to redundant diversification when not necessary to solve a task, since the mutual information objective could simply enforce different agent policies to be different, irrespective of the task. This could also hamper adaptability to tasks that do not require specialisation. This is the exact reason we do not enforce specialisation, but rather enable it, by learning end-to-end when specialised or homogeneous behaviours are required. To validate this, we experimentally test our approach in a wide array of settings, requiring homogeneous, heterogeneous, or mixed behaviours.

4. FuPS+ID in the Synchronisation Environment (Q4)

FuPS+ID performs well in the Synchronisation environment with a few agents, but as the number of agents increases, this environment becomes more challenging and FuPS+ID performance drops. This is because the probability of getting the maximum reward (all agents selecting the same action) during exploration is $\frac{n!}{n^n}$, where $n$ is the number of agents e.g. $n=8$, $Pr(X = max_reward)=0.002$. This means that as $n$ grows large, it is harder for agents to synchronise. Agents get rewards for partial matches as described in section E.1 in the Appendix, but this is still a challenging environment. We will make this clearer in the updated manuscript.

5. FuPS in the Gradient Conflict Section (3.2) (Q5)

The primary goal of Section 3.2 was to investigate why FuPS+ID struggles to specialise, despite being a universal approximator ($\pi_{\theta}(a^i|o^i,id^i)$ should be able to approximate $\pi_{{\theta}_i}$). As you note, our analysis in this section omits the plain FuPS baseline because it has a more fundamental limitation: a shared policy without agent IDs cannot learn distinct agent behaviours. We prove this formally for the 2-player Specialisation Game in Appendix E.3. The proof and empirical results shown in Table 3 show that agent IDs can be beneficial.

However, our central claim is that the method of providing this ID is critical. The FuPS+ID (No State) ablation, which outperforms standard FuPS+ID by ignoring observations, demonstrates that naively coupling the observation and ID inputs introduces destructive gradient interference. Therefore, the conclusion about the "Importance of Observation and ID Decoupling" still holds -- while IDs are necessary for specialisation, they must be decoupled from observation processing to be most effective. We will clarify this in the paper.

6. Can the inputs to the hypernetworks and policy network be swapped (i.e. use observation as input to the hypernetworks, …) (Q5)?

This is an interesting idea, but we believe this will likely not work. Swapping the inputs would re‑entangle ID and observations. The agent‑conditioned Jacobian in Eq. (4) would become observation‑dependent (not deterministic with respect to mini-batch samples), removing the variance/decoupling benefit that HyperMARL relies on. Furthermore, this would result in other challenges as an entirely new policy would be generated every timestep. The agents would also effectively be blind since the policy is not conditioned on observations, and therefore would likely not perform well in most environments (even if the weights were generated conditioned on the observations).

7. Minor Wording (Q3)

• "PG-FuPS+ID" - Thanks for pointing this out. Apologies, "PG-FuPS+ID" is the same as "FuPS+ID"; PG stands for policy gradient. We will fix this minor wording error in the updated manuscript.
• Lines 108-109 - Thanks for the clarifying question. Our claim regarding "scalability" refers not to FuPS's performance degradation, but to the widening performance gap between the FuPS and NoPS architectures as the number of agents increases, i.e. for larger values of $n$, the gap between FuPS and NoPS increases (NoPS performing better in Specialisation, FuPS in Synchronisation). We will clarify the wording accordingly.

Conclusion

In summary, based on your feedback we added:

• A new baseline SePS and environment, Blind Particle Spread. Extending our evaluation to 22 scenarios, up to 30 agents and six baselines (NoPS, FuPS, DiCo, HAPPO, Kaleidoscope and now SePS). HyperMARL is competitive with this baseline.
• Clarified the integration of agent-specific parameters into RL networks, the interplay between specialisation and redundant diversification, and other clarifications.
• Clarified the FuPS results in the Synchronisation Environment and the Gradient Conflict Section.

During the rebuttal, we have also added:

• A sensitivity analysis using recurrent MAPPO in the challenging 20-agent SMAC-v2 style environment.
• Discussed the computational trade-off of using our method and referred to our existing sections on the memory, speed and parameter efficiency of our method (sections F3, F4).
• A learned agent embeddings analysis showing the embeddings move closer together when the agents should behave similarly, and embeddings remain far apart when agents need to specialise.
• A comparison with dual-channel networks, showing our approach is more effective.

We would like to thank the reviewer again for their thorough review and questions. We hope these clarifications and additional experiments have fully addressed the points raised, and if you have any follow-up questions or comments, please let us know, and we will be happy to discuss further.

References:

1] Christianos, F., Papoudakis, G., Rahman, M.A. and Albrecht, S.V., 2021, July. Scaling multi-agent reinforcement learning with selective parameter sharing. In International Conference on Machine Learning (pp. 1989-1998). PMLR.

2] Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359–366, 1989.

We thank reviewer z98m for their review and for encouraging comments regarding our novel insight with respect to gradient decoupling in MARL, our strong motivation for using agent-conditioned hypernetworks and our clarity. We address your comments as follows:

1. Using ID or Embeddings … may limit representation ability (W1)

We would like to clarify that the agent IDs or embeddings inputs are not the main components of the hypernetworks that influence their representation ability, but rather the functional form of the hypernetwork. Linear hypernetworks can only capture linear relationships between agents’ embeddings and their corresponding weights. To enhance expressiveness, we introduced MLP hypernetworks for most of our experiments, allowing our method to adapt to more complex tasks. Furthermore, using learned embeddings means that the conditioning can automatically adapt to the task. We will make this property clearer in the updated paper.

2. Using SMAC and SMACv2 vs SMAX and Could HyperMARL work well in... SMACv2 (W2, Q2)

We evaluate on SMAX, an accelerated approximate reimplementation of SMAC, which includes v1 and v2 style scenarios. We benchmark on both v1 (e.g. 2s3z,3s5z) and v2 style maps from SMAX (10 and 20 units). Although some dynamics are different in SMAX, it features a more sophisticated heuristic AI opponent, and some maps are harder in this version [1]. We believe these settings provide a strong and appropriate testbed for our method.

Furthermore, we included SMACv2-style scenarios in SMAX that also have this “meaningful partial observability” property of SMACv2-style maps. Our approach performed well in this case (Fig. 8c and d) showing that the hypernetworks do not limit the agent's network.

3. Comparison with role-based approaches, such as ROMA, in the Experiments (W3)

Role-based methods, like ROMA and RODE [3, 4], assign agents to subtasks/roles but rely on mutual information (MI) objectives for diversity. Our primary reason for not including these methods in the experiments is that they have been shown to have limited empirical performance (Table 1 in [2]), which we discussed in Table 4 in the Appendix. Furthermore, we have six baselines (NoPS, FuPS, DiCo, HAPPO, Kaleidoscope and now SePS), which we feel serve as a good testbed. Nonetheless, we will expand on our discussion of role-based methods in the related work, in the updated manuscript.

4. Comparison with Dual-channel Network Architecture

New Ablation: We thank the reviewer for their thoughtful suggestion. Following the reviewer's advice, we implemented and evaluated a dual-channel network architecture. This baseline processes the agent's observation and ID in separate channels before concatenating their outputs. We tested two variants: one using one-hot encoded agent IDs and another using learned agent embeddings. We evaluated this using IPPO on the Dispersion task from our paper. Our results below show that HyperMARL outperforms dual-channel networks.

Results: IQM and 95% CI of mean test episode return across 5 seeds, Bold indicates the highest IQM score.

Takeaway: HyperMARL outperforms dual-channel architectures.

5. Compatibility with value-factorisation/decomposition methods (e.g. QMIX, QPLEX) (Q3)

Yes, HyperMARL is compatible with these methods as it simply generates policy or critic networks and could do so in any setting or method. In Table 5 in the App, we showed that HyperMARL also works well in off-policy methods such as MATD3. We believe these results would also hold for value-decomposition methods. However, due to limited time resources and time during the rebuttal, we are unable to run additional experiments using HyperMARL for value-decomposition methods. We will discuss this as a possible future work direction in the updated manuscript.

Conclusion

In summary, based on your feedback we added:

• A comparison with dual-channel networks, showing our approach is more effective.
• Discussed the compatibility of our method with value-decomposition methods and discussed the representations that can be learned.

During the rebuttal, we have also added:

• A new baseline SePS and environment, Blind Particle Spread. Extending our evaluation to 22 scenarios, up to 30 agents and six baselines (NoPS, FuPS, DiCo, HAPPO, Kaleidoscope and now SePS). HyperMARL is competitive with this baseline.
• A sensivitiy analysis using recurrent MAPPO in the challenging 20-agent SMAC-v2 style environment.
• A learned agent embeddings analysis showing the embeddings move closer together when the agents should behave similarly, and embeddings remain far apart when agents need to specialise.

We would like to thank the reviewer again for their thoughtful review and hope these clarifications and additional experiments address your concerns. If there are no further issues, we respectfully ask reviewer z98m to provide an updated assessment of our work.

References:

1] Rutherford, A., Ellis, B., Gallici, M., Cook, J., Lupu, A., Ingvarsson Juto, G., Willi, T., Hammond, R., Khan, A., Schroeder de Witt, C. and Souly, A., 2024. Jaxmarl: Multi-agent rl environments and algorithms in jax. Advances in Neural Information Processing Systems, 37, pp.50925-50951.

2] Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. In 34th Conference on Neural Information Processing Systems, 2020.

3] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: multi-agent reinforcement learning with emergent roles. In Proceedings of the 37th International Conference on Machine Learning, 407 pages 9876–9886, 2020.

4] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523, 2020.

We thank reviewer uZHN for their thoughtful and constructive review. We found your points valuable for helping improve the quality of our manuscript. We also thank the reviewer for acknowledging our diverse experimental evaluation, novel insight concerning gradient decoupling in MARL, and for recognising our clarity and presentation. We address your comments as follows:

1. Scalability with Target Network Size (W1)

We thank the reviewer for this point. The parameter overhead of the hypernetwork does indeed scale with the size of the target network. While we argue this is often an acceptable trade-off for the smaller actor-critic networks commonly used in MARL and the near-constant scaling with respect to the number of agents might be a viable trade-off for some use cases (Section F.3), this could be problematic for much larger target networks such as Transformers. However, as noted in our limitations and your review, this is a known characteristic of hypernetworks, and there is a clear and established path for scaling to such architectures using parameter-efficient techniques like chunking or low-rank weight approximations.

2. Complexity of Hypernetwork Training and Sensitivity to Hyperparameters (W2)

New Sensitivity Analysis: We thank the reviewer for raising this important point. Based on your recommendation, we run a sensitivity analysis on the hypernetworks parameters, in the 20-agent SMAX scenario with MAPPO, which also includes a recurrent GRU, a challenging setting for optimisation.

Findings:

1. Similarly to FuPS+ID, and in general on-policy RL methods like PPO, learning rate is an important parameter for performance.
2. Agent embedding size can be an important parameter for the performance of HyperMARL, with a smaller agent embedding size outperforming a larger one in this case. This could be the homogenous nature of some SMAX tasks, where with smaller embeddings, it could be easier to learn similar agent embeddings and hence behaviours.
3. Other parameters, such as the width, have limited impact beyond a modest size.

IQM and 95% CI of mean win rate across 5 seeds. Bold indicates the highest IQM score; * indicates scores whose 95% confidence intervals overlap with the highest score.

Takeaway: Agent embedding size can be an important hyperparameter in our method, one that could correspond to the task's diversity requirements.

We thank the reviewer for recommending this analysis, and we will include a discussion of this in the updated manuscript.

3. Analysis of Learned Agent Embeddings (Q1)

We thank the reviewer for this suggestion. We conducted an analysis of the learned embeddings as per the reviewer's recommendation. We chose the 4-agent Navigation environment since we could test the embeddings in the same settings with two different objectives: one where all agents have the same goal position (from Fig. 7a) and the one where agents have different goal locations (Fig. 7b).

New Agent Embedding Analysis: We compute the cosine distance between agent embeddings. At initialisation, due to the orthogonal initialisation, agent embeddings have a distance of 1 from each other.

Findings: When the task requires agents to learn the same behaviour (go to the same goal), the embeddings move closer together (-0.118, p-value using one-sample t-test = 0.005, i.e. we reject that embeddings have the same distance as at init). When agents have different goals, the embeddings remain near orthogonal (+0.010, p-value using one-sample t-test = 0.331, i.e. we cannot reject that embeddings maintain their initial distance). This confirms that the hypernets have learned useful and adaptive representations.

Results: The table below shows mean cosine distances (± standard deviation) across 5 random seeds:

We believe this provides further motivation for our approach and will include this and our t-SNE plots in the updated manuscript. Thanks again for the recommendation.

4. Disentangling the Effects of Gradient Decoupling vs. Parameter Specialisation (Q2)

We believe both gradient decoupling and parameter specialisation are important, but gradient decoupling is a more critical factor. Our ablation studies in Fig. 9 provide strong evidence of the importance of the gradient decoupling. In the "w/o GD" (without gradient decoupling) variant, agents still have specialised parameters, but we re-couple the gradients by also passing the ID to the policy, and this hurts performance. This demonstrates that simply providing specialised parameters is insufficient without the architectural decoupling that HyperMARL provides. We will make this clearer in the updated manuscript.

5. HyperMARL in Mixed/Competitive Settings (Q3)

We thank the reviewer for the suggestion and agree that this would be an interesting angle to our work, especially with regard to the conflicting objectives. While parameter sharing has traditionally been investigated in cooperative tasks, it would be interesting to study this in other game settings. Unfortunately, due to limited resources and time, we are not able to robustly measure HyperMARL performance in these settings. We believe this might be an interesting direction, and will include it in potential avenues of future work.

6. Comparison with Role-Based Approaches (Q4)

Role-based methods, like ROMA and RODE [1, 2], assign agents to subtasks/roles but rely on mutual information (MI) objectives for diversity, and have had limited practical success [3]. We will include a more detailed comparison in the updated paper.

7. Assumption of Fixed Agent Population (L1)

It is correct that we only consider the setting where the team in training and evaluation is fixed and doesn't change, i.e. no generalisation to an unknown number of agents. As noted by the reviewer, the linear hypernetwork variant is tied to the number of agents, but not the MLP variant.

We actually consider this setting an ideal one for our MLP hypernetworks, because, unlike traditional FuPS+ID, which cannot handle a different number of agents due to the dimension one-hot encoded agent IDs, MLP hypernets with learned agent embeddings (of a fixed size) enable adding new agents to the team with some fine-tuning of the agent ID embeddings. Rather than explicitly a limitation, since all baselines are either incompatible or have the same property, we see this as an interesting area for future work. We will mention this in the updated manuscript.

8. Trade-off between Computational Cost (at Inference) and Adaptability (L2)

We thank the reviewer for raising this important point. HyperMARL's adaptivity does come at a higher inference cost than FuPS -- as we showed in F.4, and this trade-off would need to be balanced depending on the scenario. We will explicitly mention this in the limitations section in the updated manuscript.

Conclusion

In summary, based on your feedback we added:

• A sensitivity analysis using recurrent MAPPO in the challenging 20-agent SMAC-v2 style environment.
• A learned agent embeddings analysis showing the embeddings move closer together when the agents should behave similarly, and embeddings remain far apart when agents need to specialise.
• Discussed the importance of gradient decoupling vs parameter specialisation.

During the rebuttal, we have also added:

• A new baseline SePS and environment, Blind Particle Spread. Extending our evaluation to 22 scenarios, up to 30 agents and six baselines (NoPS, FuPS, DiCo, HAPPO, Kaleidoscope and now SePS). HyperMARL is competitive with this baseline.
• A comparison with dual-channel networks, showing our approach is more effective.

We would like to thank the reviewer again for their thoughtful review and for the opportunity to improve our manuscript. We hope these clarifications and additional experiments address your concerns. If there are no further issues, we respectfully ask reviewer uZHN to provide an updated assessment of our work.

References:

1] Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: multi-agent reinforcement learning with emergent roles. In Proceedings of the 37th International Conference on Machine Learning, 407 pages 9876–9886, 2020.

2] Tonghan Wang, Tarun Gupta, Anuj Mahajan, Bei Peng, Shimon Whiteson, and Chongjie Zhang. Rode: Learning roles to decompose multi-agent tasks. arXiv preprint arXiv:2010.01523, 2020.

3] Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Shared experience actor-critic for multi-agent reinforcement learning. In 34th Conference on Neural Information Processing Systems, 2020.

We thank reviewer 1sWF for their detailed and constructive review. We appreciate the reviewer's positive feedback on our novel insight into gradient decoupling, the diversity of our experimental evaluation, and our paper's clarity. We address your points as follows:

1. Computational Comparison (W1,Q1)

We thank the reviewer for raising this important point. Our paper provides a detailed analysis of these computational trade-offs in Appendix F.3 and F.4, and we summarise the key findings here:

• Parameter Scaling (F.3): There is a direct trade-off between scalability and versatility. While our MLP-based hypernetworks have more initial parameters than FuPS, they scale nearly constantly with the number of agents. In contrast, NoPS sees its parameter count grow linearly with respect to the number of agents. This makes our approach far more parameter-efficient for large-scale MARL, especially when the task could require specialisation. FuPS+ID parameters scale better than our hypernetworks, but these approaches fail to solve tasks that require specialisation, as we showed in Sections 3.2 and 5. We also mention that techniques like chunked hypernetwork or low-rank approximations could be used in future work to further improve the parameter efficiency of our method. In Figure 15, we also see that if we scale FuPS+ID to match the number of trainable parameters in our MLP hypernetworks, our approach still outperforms these methods, showing the benefit is architectural and not just from a higher parameter count.
• Runtime & Memory (F.4): HyperMARL offers a practical middle ground. It is more computationally intensive than the simple FuPS architecture, but it is more efficient than NoPS in terms of speed and memory. Furthermore, NoPS would also incur data transfer and synchronisation costs not fully captured in our single-GPU benchmarks.

2. Comparison with HetGPPO - Multi-Robot Heterogeneous Methods with Communication (W2, Q3)

We thank the reviewer for this suggestion. Our work focuses on behavioural heterogeneity, where physically identical agents learn diverse or homogenous policies. This is distinct from methods like HetGPPO, which are designed for structural heterogeneity (e.g., agents with different sensors or actuators) and often incorporate explicit communication.

While HetGPPO is an important baseline in its own domain, we benchmarked against the most relevant state-of-the-art methods for our specific problem (NoPS, FuPS, DiCo, HAPPO, Kaleidoscope, and SePS). To clarify this distinction, we will update our related work section to include a discussion of HetGPPO.

3. Sensitivity to Hyperparameters (Q2)

New Sensitivity Analysis: We thank the reviewer for raising this important point. Based on your recommendation, we run a sensitivity analysis on the hypernetworks parameters, in the 20-agent SMAX scenario with MAPPO, which also includes a GRU, a challenging setting for optimisation.

Findings:

1. Similarly to FuPS+ID, and in general on-policy RL methods like PPO, learning rate is an important parameter for performance.
2. Agent embedding size can be an important parameter for the performance of HyperMARL, with a smaller agent embedding size outperforming a larger one in this case. This could be the homogenous nature of some SMAX tasks, where with smaller embeddings, it could be easier to learn similar agent embeddings and hence behaviours.
3. Other parameters, such as the width, have limited impact beyond a modest size.

IQM and 95% CI of mean win rate across 5 seeds. Bold indicates the highest IQM score; * indicates scores whose 95% confidence intervals overlap with the highest score.

Takeaway: Agent embedding size can be an important hyperparameter in our method, one that could correspond to the task's diversity requirements.

We thank the reviewer for recommending this analysis, and we will include a discussion of this in the updated manuscript.

Conclusion

In summary, based on your feedback we added:

• A sensitivity analysis using recurrent MAPPO in the challenging 20-agent SMAC-v2 style environment.
• Discussed the computational trade-off of using our method and referred to our existing sections on the memory, speed and parameter efficiency of our method (sections F3, F4).

During the rebuttal, we have also added:

• A new baseline SePS and environment, Blind Particle Spread. Extending our evaluation to 22 scenarios, up to 30 agents and six baselines (NoPS, FuPS, DiCo, HAPPO, Kaleidoscope and now SePS). HyperMARL is competitive with this baseline.
• A learned agent embeddings analysis showing the embeddings move closer together when the agents should behave similarly, and embeddings remain far apart when agents need to specialise.
• A comparison with dual-channel networks, showing our approach is more effective.

We would like to thank the reviewer again for their thoughtful review and hope these clarifications and additional experiments have fully addressed the points raised. If you have any follow-up questions or comments, please let us know, and we will be happy to discuss further.

We thank reviewer bHoX for their review and for encouraging comments regarding the paper’s motivation, important problem setting, novel decoupling of observation- and agent-conditioned gradients and clear writing. We address your comments as follows:

1. Evaluations & Benchmarks (W1,Q1)

Experiment Design

We thank the reviewer for this point. Our primary goal was to ensure the most rigorous comparison for each scenario by using baselines in the environments where they were originally tuned and shown to be most effective (e.g., DiCo in Navigation; HAPPO and Kaleidoscope in MaMuJoCo; FuPS in SMAX). By using original code and hyperparameters, we ensure fairness, reproducibility, and adherence to best evaluation practices [4]. We acknowledge that this can appear fragmented; to address this, we will add a paragraph to our existing discussion of this methodological choice (line 209) in Section 5 to clarify this further.

Additional Baselines

We thank the reviewer for proposing further baseline methods [1-3] to add to our five existing baselines.

New Baseline and Environment: Based on the reviewer's recommendation, we have added an additional baseline -- SePS [1], a parameter sharing method that uses an autoencoder during pre-training to cluster agents based on their transitions and shares parameters between these agent groups. We use their original source code and hyperparameters and evaluate on all their Blind Particle Spread settings (v1-4), which include up to 30 agents.

IQM final total reward (with 95% confidence intervals) across 5 seeds on the BPS environments from [1], Bold indicates the highest IQM score; * indicates scores whose 95% confidence intervals overlap with the highest score.

Takeaway: The results highlight HyperMARL's competitive performance. In all four scenarios, its 95% confidence intervals overlap with SePS and NoPS, indicating it is statistically on par with these baselines. Notably, SePS relies on a pre-training clustering phase, whereas HyperMARL learns end-to-end without pre-training.

We thank the reviewer for the suggestion and for the opportunity to further expand our experimental evaluation, now up to 22 scenarios, up to 30 agents and six baselines (NoPS, FuPS, DiCo, HAPPO, Kaleidoscope and now SePS). We will include this in the updated manuscript.

Other baselines you mentioned:

• As mentioned in Table 4 in the App., SNP-PS [2] does not have any publicly available source code. Furthermore, we already have Kaleidoscope as a baseline, which outperformed SNP-PS.
• GradPS [3] was made available online after 5 June (ICML camera-ready deadline) after the NeurIPS deadline (15 May), with no previously available preprint. Furthermore, there is no publicly available source code. We will include this work in the related work discussion.

2. Computational Costs & Trade-offs (Q3)

We thank the reviewer for raising this point. We analysed and discussed these points in the "F.3 Scalability and Parameter Efficiency" and “F.4 Speed and Memory Usage” sections in the Appendix, and will ensure key trade-offs are also mentioned in the main text.

• Parameter Scaling (F.3): There is a direct trade-off between scalability and versatility. While our MLP-based hypernetworks have more initial parameters than FuPS, they scale nearly constantly with the number of agents. In contrast, NoPS sees its parameter count grow linearly with respect to the number of agents. This makes our approach far more parameter-efficient for large-scale MARL, especially when the task could require specialisation. FuPS+ID parameters scale better than our hypernetworks, but these approaches fail to solve tasks that require specialisation, as we showed in Sections 3.2 and 5. We also mention that techniques like chunked hypernetworks or low-rank approximations could be used in future work to further improve the parameter efficiency of our method. In Figure 15, we also see that if we scale FuPS+ID to match the number of trainable parameters in our MLP hypernetworks, our approach still outperforms these methods, showing the benefit is architectural and not just from a higher parameter count.
• Runtime & Memory (F.4): HyperMARL offers a practical middle ground. It is more computationally intensive than the simple FuPS architecture, but it is more efficient than NoPS in terms of speed and memory. Furthermore, NoPS would also incur data transfer and synchronisation costs not fully captured in our single-GPU benchmarks.

3. Additional Related Work (W2, Q2)

Thank you for suggesting additional related work. As mentioned, GradPS [3] was published after the submission deadline, but thanks for the recommendation, and we will include a discussion of this in the updated manuscript. For the other papers mentioned, our focus was reinforcement learning and so we focused on hypernetworks in this setting (not in other fields like federated learning). Based on your recommendation, we will also position our work within the broader hypernetwork literature in the updated manuscript.

4. HyperMARL vs FuPS in Homogeneous Tasks like SMAX (Q4)

These experiments were presented to validate Hypermarl’s adaptability claim, i.e. that it can learn specialised or homogeneous behaviour when required. Matching FuPS performance in homogeneous tasks shows that HyperMARL can also adapt to these settings, i.e. does not enforce specialisation in a detrimental manner. The challenge is that in most settings, you do not already know if the task requires homogeneous, specialised or mixed behaviour a priori, therefore, it is useful to have a method that can adapt to varying problem classes and a versatile architecture that is effective across these settings. To validate these claims was the core focus of the experimental section's Q1 and Q2 in Section 5. In summary, this confirmed that our method does not degrade performance in homogeneous regimes while improving performance in heterogeneous ones (compared to FuPS).

Conclusion

In summary, based on your feedback we added:

• A new baseline SePS [1] and environment, Blind Particle Spread. Extending our evaluation to 22 scenarios, up to 30 agents and six baselines (NoPS, FuPS, DiCo, HAPPO, Kaleidoscope and now SePS). Our approach is competitive with this baselines, without requiring a pre-training phase.
• Discussed the computational trade-off of using our method and referred to our existing sections on the memory, speed and parameter efficiency of our method (sections F3, F4).

During the rebuttal, we have also added:

• Sensitivity Analysis showing which parameters of our method influence performance the most.
• An analysis of the learned agent embeddings showing the embeddings move closer together when the agents should behave similarly, and embeddings remain far apart when agents need to specialise.
• A comparison with dual-channel networks, showing our approach is more effective.

We hope these clarifications and additional experiments address your concerns, and we thank the reviewer for the opportunity to improve our manuscript. If there are no further issues, we respectfully ask reviewer bHoX to provide an updated assessment of our work.

References:

1] Christianos, F., Papoudakis, G., Rahman, M.A. and Albrecht, S.V., 2021, July. Scaling multi-agent reinforcement learning with selective parameter sharing. In International Conference on Machine Learning (pp. 1989-1998). PMLR.

2] Kim, W. and Sung, Y., 2023. Parameter sharing with network pruning for scalable multi-agent deep reinforcement learning. arXiv preprint arXiv:2303.00912.

3] Qin, H., Liu, Z., Lin, C., Ma, C., Mei, S., Shen, S. and Wang, C., GradPS: Resolving Futile Neurons in Parameter Sharing Network for Multi-Agent Reinforcement Learning. In Forty-second International Conference on Machine Learning.

4] Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in reinforcement learning. Journal of Machine Learning Research, 25(318):1–63, 2024.

We thank reviewer bHoX for their engagement and important clarifying question.

1. Clarifying Breakdown of Scenarios

We consider a scenario, a grouping of an environmental setting and a number of agents. This is common in the literature as the number of agents changes the dynamics of a problem, e.g. SMAC 3m vs 5m vs 8m, v2 5 vs 10 vs 20 agents.

Our 22 scenarios are comprised of:

• MAMuJoCo: 5 scenarios, including Swimmer-v2-10x2 as Kaleidoscope provided tuned hyperparameters for it.
• Dispersion: 1 Scenario.
• Navigation: 8 scenarios across different agent counts and goal configurations (shared, unique, mixed).
• SMAX: 4 Scenarios.
• BPS: 4 Scenarios (BPS (1) -BPS (4), some have 15 or 30 agents, but the type distribution is different, e.g. BPS (3) has 5 types/groups each with 6-6-6-6-6 (6 agents per goal), while BPS (4) has5 groups with goal distribution 2–2–2–15–9).

This total does not include the eight diagnostic Specialisation/Synchronisation games, as they were used for targeted analysis of specific behaviours rather than for broad, comparative benchmarking.

2. Rationale for a Principled, Targeted Evaluation

2.1. Fairness and Reproducibility

Our experimental design was deliberately structured to ensure fairness and scientific validity, following the best practices outlined in recent comprehensive guides on empirical RL [1]. This work explicitly warns against the pitfalls of comparing against "untuned baselines" and advises that for the most rigorous comparisons, one should evaluate baselines in environments where they are already known to perform well -- using their released code and hyperparameters.

2.2 Targeted comparisons serve the paper’s scientific claims

Accordingly, our goal is not to create a leaderboard; it is to test whether explicit gradient decoupling allows a single architecture to adapt across homogeneous, heterogeneous, and mixed-behaviour tasks, without altered objectives, manual preset diversity levels, or sequential updates.

This focused goal necessitates a targeted evaluation. We compare HyperMARL against the strongest, most relevant baselines for each specific capability: FuPS for homogeneity in SMAX, and specialisation-focused methods like HAPPO and DiCo in their respective domains of strength (MAMuJoCo and Navigation). This targeted approach allows for a clearer, more scientifically robust understanding of a decoupled hypernetwork's properties, which is the primary goal of our study.

Example: Challenges of Porting Baselines

To illustrate the challenge and potential unfairness of porting baselines, consider Kaleidoscope. It employs learnable masks, an ensemble of five critics and an explicit diversity loss, and numerous specialised hyperparameters (e.g., for critic ensembling: ‘critic_deque_len‘, ‘critic_div_coef‘, ‘reset_interval‘; for mask and threshold parameters: ‘n_masks‘, ‘threshold_init_scale‘, ‘threshold_init_bias‘, ‘weighted_masks‘, ‘sparsity_layer_weights‘, etc.).

Porting this intricate design to our on-policy PPO backbone would constitute a significant research deviation, not a direct benchmark of the established method. It would also be unfair to Kaleidoscope, as components designed for the off-policy case might not be effective on-policy. Therefore, we followed best practices by comparing HyperMARL within Kaleidoscope's original off-policy MATD3 framework, ensuring a true apples-to-apples comparison on a benchmark it was tuned for.

2.3. Conventions in the Field

Conventions in the field vary. For example, SePS, DiCo, HAPPO use on-policy algorithms and compare against FuPS and NoPS, other methods like GradPS (released after the NeurIPS deadline) use off-policy algorithms and focus on SMAC and predator-prey environments, while comparing against FuPS, NoPS, SePS and Kaleidoscope.

Summary

In summary, our methodology prioritises the scientific validity and fairness of each comparison. We believe this targeted and thorough approach (22 scenarios, six baselines), which is supported by best practices in empirical RL research [1], provides the strongest and clearest evidence regarding the specific claims made in our paper.

We would like to thank the reviewer again for their engagement. We hope these clarifications address the point you mentioned, and that the additional baseline added during rebuttal, together with the pointers to the computation/memory sections in the Appendix and our discussion of memory–speed–performance trade-offs, were useful. If you have any follow-up questions or comments, please let us know, and we will be happy to discuss further.

Reference:

1] Andrew Patterson, Samuel Neumann, Martha White, and Adam White. Empirical design in reinforcement learning. Journal of Machine Learning Research, 25(318):1–63, 2024.

I thank the authors for the discussion. I summarize my view below:

• As noted in my original review, the idea of decoupling observation- and agent-conditioned gradients through hypernetworks is conceptually novel and well-motivated.
• However, my concerns regarding performance remain unresolved after the discussion. It is difficult to fully assess the effectiveness of the proposed method because the selection of baselines varies across benchmarks. While it is understandable that including every baseline in every benchmark may not always be feasible, the absence of a single benchmark that provides a comprehensive comparison with all or most of the baselines mentioned in the paper limits the ability to evaluate the method. This is an important consideration and is critical for readers to understand when and where the method is most applicable.
• My concerns regarding the cost are mostly addressed. Through Appendix F, I see that while HyperMARL does introduce extra complexity compared to the baselines, this added complexity is acceptable as long as it is clearly reported (as the authors have done!) and balanced against improved performance (which unfortunately lack sufficient support).

I have no further questions at this stage. Thanks.