Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learning

We thank Reviewer PfFs for appreciating the high quality, experiment design, writing and significance of findings of our paper. A detailed response is provided below.

Q1: Too many math included. Some developments, or an inequality between the three objectives, could improve the understanding.

We will increase the clarity of these concepts by using power grid as an example:

Cooperation describes optimal behavior under nominal conditions. For instance, on a typical day, the power grid functions in a coordinated and efficient manner, supplying energy smoothly across all nodes. This corresponds to high expected reward under ideal settings.

Robustness refers to the ability to maintain performance during a disturbance. Suppose an earthquake causes sudden shifts in energy demand or damages part of the grid. A robust power grid can tolerate such perturbations, maintaining basic functionality despite degraded performance. This corresponds to a drop in reward, but without total failure.

Resilience captures the system’s ability to recover after the disturbance. Once the earthquake ends and demand returns to normal, the grid must recover from a perturbed and potentially degraded state. A resilient system gradually restores optimal performance, resulting in increasing reward as it recovers.

Regarding the equations, we acknowledge that too much math can hinder readability. However, since resilience has not been formally defined in prior literature, we believe the mathematical clarity is necessary, but will additionally add the example above to increase readability, such that readers can understand the concepts without relying solely on equations.

Q2: Ambiguities between these two definitions of robustness and resilience. The difference between resilience and robustness (lines 111-119) could be made earlier, group all the related works (l 103-107) in Section 5?.

Thanks for your helpful suggestion. To reduce ambiguity, we will state the difference between resilience and robustness at the beginning of Section 2, and group all related works (Line 103-107) in Section 5 as suggested.

Q3: Resilience and generalisation seem to be highly related, but the latter is not mentioned. Aren't resilience and generalisation the same?

Thank you for raising this important point. We agree that resilience and generalization are closely related concepts, but we deliberately distinguish them to highlight their different focuses and implications for algorithm design.

In our work, resilience refers specifically to the capability of an agent to recover performance after encountering unexpected uncertainties. We formalize this as performance under a new state distribution $s^u$, which represents the state of the system after an external perturbation has occurred. These post-uncertainty states are often far from nominal initial states and are not under the control of the agent.

In contrast, generalization in RL typically involves evaluating how well a policy performs when trained and tested on different initial states or environment dynamics. The variations are usually known or deliberately randomized (e.g., random starts or human-initialized states as discussed in [1]). The key distinction lies in the nature of the initial state distribution:

In generalization, the agent starts from a known distribution of initial states, potentially with limited or no exposure to unseen variations.

In resilience, the agent starts from an unknown, perturbed state $s^u$ induced by adversarial or stochastic uncertainties, often in parts of the state space rarely encountered during training.

Thus, we view resilience as a special, practically motivated subset of generalization that focuses on the real-world challenges of recovery and generalization under unknown perturbation budgets. Moreover, because the perturbed state $s^u$ is the result of certain perturbations, it opens up new algorithmic possibilities that go beyond classical generalization, such as designing policies that can actively adapt or recover given a disturbed state. We will revise the manuscript to include this discussion and clarify how resilience relates to, but differs from, standard notions of generalization.

[1] Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents, JAIR 2018.

Q4: It is counterintuitive to define resilience as agents evolving in an environment with a new initial state distribution. It makes sense formally, but it does not help the intuition behind.

To reconcile the difference between formal definition and intuition, we revise resilience as starting at time t, after encountering uncertainties. Thus, the task starts at an arbitrary time t with an inferior state for recovery. The revised equation is: $J^{\text{resilience}}(\pi) = \mathbb{E}\_{u \sim \mathcal{U}} [ \mathbb{E}\_{s^u \sim \rho^u} \mathbb{E}\_{\pi} [ \sum\_{t}^\infty \gamma^t r\_t | s\_t = s^u ] ]$.

Q5: It is always difficult to choose hyperparameter and their values. What are the default choices and why you selected them?

Currently, there do not exist a universal hyperparameter for MARL algorithms. In our paper, we select parameters that are commonly used in existing codebases, including the codebase of MAPPO [1] and HAPPO [2]. These codebases includes a set carefully designed hyperparameters that works well for MPE, SMAC and MAMujoco, which varies in different tasks. Among all hyperparameters in these codebase, we conduct a pilot study to select hyperparameters that works well in most of our real-world tasks, and select other hyperparameters as alternate solutions.

[1] The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games. NeurIPS 2022.

[2] Heterogeneous-agent reinforcement learning. JMLR 2024.

Q6: How would you define/compute the threshold in uncertainty when robustness becomes resilience?

In fact, such threshold can be hard to define mathematically, but more of an application perspective. Imagine a power grid in urban areas, in common days, there may be small perturbations to the power grid, such as sensor errors or one user request more power than usual. In such cases with small perturbations, the power grid should be robust, such that it can serve all users despite these small variations.

In rare events such as earthquake, the power grid may collapse physically and receives zero request if facing blackout, or high request if facing a short-circuit. In such cases, after engineers have fixed the power grid, it is crucial for the system to continue functioning well after recovery.

Q7: insights on how MADDPG and MAPPO/HAPPO behave differently facing different types of uncertainty. Maybe discrete/stochastic policy is the answer?

Please allow us to explain this in detail. If using discrete or stochastic policy is the answer, MAPPO and HAPPO should be robust to action uncertainties by using stochastic policies. However, our finding shows MADDPG is more robust to action uncertainties than MAPPO/HAPPO. To explain this, we empirically find the exploratory noise of MAPPO and HAPPO converge to a small value when training is over, bringing limited explorations in action space, and more deterministic policies. In contrast, in MADDPG, the exploratory noise in action space remains constant. Since the policy is constantly exploring in action space, it becomes more robust against action uncertainties.

Q8: Alternative approaches than hyperparameter tuning, like train agents in robust and resilient configurations to investigate whether agents can learn them directly? This could also improve the claim that being robust for one type of uncertainty does not imply being robust to all, and to improve all claims anyway.

We fully admit that hyperparameter tuning is not the only solution to robust and resilient MARL. The problem lies in robust MARL itself.

First, training MARL against uncertainties is commonly referred as robust MARL, which is a complex process that requires high expertise. Existing robust MARL algorithms are specific to their own type of perturbation and their backbone MARL algorithms. For example, EIR-MAPPO [11] works only on MAPPO backbone with action uncertainty applied to one agent, M3DDPG [24] works for MADDPG backbone only with action uncertainty applied to all agents. Similarly, in observation uncertainty, noise can be applied to each agent [8] or all agents [9]. Moreoverr, as reported by EIR-MAPPO [11], since their method is designed against action uncertainties, they are not robust to obervation uncertainties. To the best of our knowledge, there do not exist a single work on robust MARL that (1) can be applied to any backbone, (2) works for perturbations on one, several or all agents and (3) works for perturbations over observation, environment and actions.

The method of training MARL against uncertainties is most similar to the work of [1], which use curriculum learning to train robust MARL against diverse uncertainties in action, observation and rewards. However, the authors of this paper found training MARL against diverse uncertainties is hard, can often harm convergence, and can be robust against at most two uncertainties. We additionally conducted a prelinminary study ourselves, and found the same difficulty on convergence.

[1] Robustness to Multi-Modal Environment Uncertainty in MARL using Curriculum Learning. NeurIPS 2023.

Second, we have compared hyperparameter tuning on ERNIE (Fig 9), a general-purposed robust MARL that turns uncertainty into regularization terms, which we find to have good convergence properties empirically. While ERNIE is itself a robust MARL method, performing hyperparameter tuning further enhance its performance. Thus, we deem hyperparameter tuning and robust MARL methods are orthorgonal ways to achieve robust and resilient MARL.

We thank Reviewer GvL9 for appreciating the novelty, large-scale empirical study, and significance of our findings. A detailed response is provided below.

Q1: the robustness and resilience considered in this paper are both defined over a pre-defined set of uncertainties, which might limit the scope of possible application scenarios. For example, the uncertainties might switch from one type to another and might yield non-stationary property.

Thanks for the observation. We answer this from three perspectives. First, we control the variable of noise to limit the amount of randomness faced in our analysis. The variable control is significant for our statistical tests since only one variable is changed, which ease our effort to study the impact of each noise/algorithm/tasks/hyperparameters.

Second, using mixed type of uncertainties may introduce large variations in experiments. For example, if we select some type of uncertainties to appear after another uncertainty at some time, we must control the time, type and magnitude of uncertainty we applied, which makes the evaluation computationally intractable and introduce large variations during evaluations.

Third, existing robust MARL literature use bounded noises to evaluate their performance. We select the current setting to connect with current literatures.

We fully agree that mixing these uncertainties is important for comprehensive evaluation. We will add this as a function in our codebase to support real-world evaluation of these non-stationary noises.

Q2: It’s a bit unclear if higher LR of critic is caused by the insufficient number of mini-epochs the critic is optimized with. Other factors could also affect this conclusion, e.g., the batch size.

Our code is based on the widely used codebase of MAPPO [1], where actors and critics are optimized with the same mini-epochs. The batch size also inherits the default of MAPPO, which is the same for actors and critics. Thus, higher LR of critic indeed helps, and suggests a potential optimization direction for MARL.

[1] The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games. NeurIPS 2022.

Q3: It’s a bit surprising to see PopArt leads to inconsistent effectiveness when the reward scales remain stable. I was wondering when PopArt was used whether the advantage is also scaled according in gradient estimate.

Our implementation again follows the official implementation of MAPPO. They use PopArt to normalize the learning process of the critic. When computing the advantage for the actor, the value prediction is first denormalized by the inverse PopArt transform and then the advantage is Z-score normalized for policy gradient.

Q4: MADDPG is not benefitting from the increased exploration is also a bit surprising to me. MADDPG has different exploration schemes than MAPPO and HAPPO. This direction comparison in terms of entropy bonus between these methods may not stand.

Thanks for the insightful comment. We agree that MADDPG’s exploration scheme differs from the entropy-based exploration used in MAPPO and HAPPO, and that direct comparisons must be interpreted with care.

In MAPPO/HAPPO, the entropy bonus is a soft constraint, which encourage explorration at the beginning, while the policy can still converge to a relatively deterministic policy which reach optimal cooperation.

In contrast, the exploratory noise of MADDPG is additive action noise, which is enforced throughout the training process and does not decay naturally. This can lead to suboptimal behavior, especially when large noise values perturb the agent into poor or unstable regions of the state space. Empirically, we find that increasing MADDPG’s action noise did not lead to better performance. We believe this is because excessive noise degraded the quality of the agent’s behavior, making it harder to maintain meaningful strategies.

We will add the analyzation of on-policy and off-policy exploration scheme above in final revised version for further analyzation.

Q5: In the resilience description “the system evolves without additional uncertainties”, not sure if there is a typo in the inner expectation for resilience.

Thanks for pointing this mistake. The inner expectation should not additionally condition on u. We will correct this in final revision.

Thanks for the response. I will keep my score unchanged.

We thank Reviewer bhkV for appreciating the scale, rigor, comprehensiveness, codebase contribution, statistical test and actionable guidance of our paper. A detailed response is provided below.

Q1: The paper do not introduce new algorithms and new theoretical guarantees.

Thank you for the comment. It is true that our paper does not propose a new algorithm or theoretical guarantee. However, the core aim of this work is to provide a comprehensive empirical study on robustness and resilience in MARL, which, to our knowledge, is still under-explored.

Rather than introducing a single new algorithm, we contribute a set of key empirical findings and actionable insights that can guide the development of an entire class of more robust and resilient MARL algorithms. We believe this kind of contribution can be equally, if not more, valuable to the community, as it identifies where and how current methods fail and what design principles matter most under different uncertainty conditions.

For example, in Section 4.3, we give suggestions on hyperparameters that improves cooperation, robustness and resilience together. In Section 4.4, we show applying these hyperparameters can improve MARL performance largely on both purely cooperative, and robust MARL. Similarly, in Section 4.2, we should robust MARL should be evaluated across uncertainty modalities and agent scopes, which highlight limitations of current evaluation and point to new algorithmic opportunities.

Q2: The results may not generalize to alternative MARL paradigms (e.g., value decomposition, graph-based, or distributional RL methods)

Thank you for the valuable feedback. While our focus is on policy-based MARL, many of our findings remain relevant to alternate MARL paradigms, and we identify this as a promising direction for future work.

First, our real-world environments requires continuous control without graph structure. Thus, mainstream value decomposition based method like QMIX are not applicable and the graph sturcture is not available. We acknowledge the importance of these works and will discuss them in the related work.

Second, while our study does not directly evaluate these methods, many of these paradigms are built on policy gradient frameworks. Thus, many insights should still transfer, such as the benefits of higher critic learning rates, parameter sharing, GAE, PopArt, and exploration strategies.

Third, value decomposition methods assume all agents contribute positively, which fails in adversarial settings with compromised agents. Graph-based MARL may be more vulnerable to critical agents due to inter-agent connectivity, while distributional RL may offer greater robustness via stochasticity. However, evaluating these would require ~80,000 additional experiments, which is beyond our current scope.

Finally, our codebase (Appendix A.4) support new algorithms that meets our interface requirement and can be used to evaluate alternative MARL paradigms under adversarial perturbations. Our codebase support QMIX which is based on value decomposition currently. We will include this as a limitation and an invitation for follow-up work in the final version.

Q3: Some conclusions may be specific to the chosen environments or implementation details.

We believe our findings are general enough for the following reasons. While no benchmark can fully represent MARL, our environments span over diverse and representative tasks of MARL.

Under diverse environments and uncertainties, our findings suggests certain tuning directions that are consistent and statistically significant across all settings, including higher critic LR, early stopping and Leaky ReLU. Moreover, while parameter sharing, GAE and PopArt are treat as default by previous researchers, our finding shows it is merely a design choice, revealing blind spots that researchers may overlook. While these findings may not help for any task, algorithm and random seed, they are a promising direction for optimization and helps at high probabilities.

Q4: There is limited presentation of raw numerical results, confidence intervals, or effect sizes in the main text. Per-task outcomes are relegated to the appendix.

We will share the raw reward score under each configurations and uncertainties after acceptance of our paper and include additional raw numerical results in Appendix for further analysis. Since we are sharing insights of our evaluation, the raw records may be messy and requires time and effort to understand.

As for confidence intervals and effect sizes, we will add them in main paper in final revision.

Q5: The empirical comparison does not include more recent robust MARL methods (e.g., distributional RL, adversarial or opponent modeling).

First, please allow us to clarify we have included comparison with ERNIE in Fig 9, a general-purposed robust MARL method. The result shows our selcted hyperparameters also works for ERNIE. Given the fact that many robust MARL algorithms are based on MARL backbones like MAPPO, our result helps these robust MARL methods as well.

Second, robust MARL is complex. Most algorithms are specific to their own type of perturbation and their backbone MARL algorithms. For example, EIR-MAPPO [11] works only on MAPPO backbone with action uncertainty applied to one agent, M3DDPG [24] works for MADDPG backbone only with action uncertainty applied to all agents. Similarly, in observation uncertainty, noise can be applied to each agent [8] or all agents [9]. Moreoverr, as reported by EIR-MAPPO [11], since their method is designed against action uncertainties, they are not robust to obervation uncertainties. To the best of our knowledge, there do not exist a single work on robust MARL that (1) can be applied to any backbone, (2) works for perturbations on one, several or all agents and (3) works for perturbations over observation, environment and actions. As such, we compare ERNIE, which is the robust MARL algorithm that have the highest potential to deal with all types of uncertainties.

Q6: The large breadth of experiments and the dense presentation may overwhelm readers, and some methodological choices (such as rationale for selected hyperparameter ranges or interaction effects) could be discussed more clearly for accessibility.

We will include a takeaway table for readers to summarize our main points below:

We select hyperparameters that are commonly used in existing codebases, including the codebase of MAPPO [1] and HAPPO [2]. These codebases includes a set carefully designed hyperparameters that works well for MPE, SMAC and MAMujoco, which varies in different tasks. Among all hyperparameters in these codebase, we conduct a pilot study to select hyperparameters that works well in most of our real-world tasks, and select other hyperparameters as alternate solutions.

[1] The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games. NeurIPS 2022.

[2] Heterogeneous-agent reinforcement learning. JMLR 2024.

For interaction effects, we conduct this anaysis to check if the hyperparameters are still effective when combining with each other. we will move some details in Appendix B.7 to main paper for increased clarity.

Q7: Have the authors considered whether noise effects accumulate in later-collected samples as training progresses?

The training progress is done in ideal simulation environment, mimicing current MARL training paradigm so we do not add noise in sample collection. Adding noise during training is often framed as backdoor or poisoning attacks, which follow a different theoretical paradigm and is out of the scope of our paper.

Q8: Would increasing random seeds affect statistical confidence?

Please allow us to clarify that all our results are averaged across multiple runs, tasks and algorithms. Thus, while we tested for 5 random seeds, the sample sizes are more than 100 in Section 4.2 and 4.3. Thus, while we admit more random seeds are always better, we believe the number of samples are sufficient to make statistically significant claims.

Q9: What are the computational requirements for reproducing the main findings?

Our full experiment takes ~230K GPU hours, measured in GTX 4090. For better reproducability, we will share the raw reward score under each configurations for further analysis.

Q10: repeated references to the code link and several instances of redundant information.

Thanks for the suggestion. We will reduce these redundant information in our final revision.

We thank Reviewer W4zq for appreciating the novelty, large-scale empirical study, codebase contribution, statistical contributions and findings of our paper. A detailed response is provided below.

Q1: The paper generalizes claims about concepts, while directly giving a contradictory example. Likely a more principled manner to make these claims, since the exceptions aren't always explained.

1. When and why parameter sharing do not work?

We believe parameter sharing helps in environments where agents are more homogeneous like SMAC, where agents are of the same type (eg, zealot, marine). However, it do harm performance for Voltage, Traffic, MPE and MAMujoco since in these environments, agents are heterogeneous (Line 307-311). For Dexhand and Quads, the effect is mixed and non-significant. While we do believe agents benefit from parameter sharing when they are homogeneous, there do not exist a quantitative way to measure how "homogeneous" the environment is since it is highly task-specific. A safe conclusion is to treat parameter sharing as a design choice, rather than default implementation, especially in environments with heterogeneous agents.

Our claim is reinforced in HATRPO [1], where they show parameter sharing may result in suboptimal actions since it pose an important constraint that the parameters of all agents are identical. The Proposition 1 made by HATRPO even show parameter sharing brings exponentially worse outcome by constructing a negative example. On MAMujoco environment, they also find parameter sharing "introduces extra variance to training, harms the monotonic improvement property, make algorithm converge to suboptimal policies, and the problem gets more severe in the task with more agents".

[1] TRUST REGION POLICY OPTIMISATION IN MULTI-AGENT REINFORCEMENT LEARNING. ICLR 2022.

2. GAE/Popart. is it just a design choice if we know the reward structure?

While we do find GAE and PopArt helps in MAMujoco with dense and stable reward structure, and can have negative impact in our 4 real world environment, SMAC and MPE with statistical significance, we shall additionally remind readers that such claims are made empirically and statistically, which will not hold for every case. However, the statistical significant result do suggest a potential direction for tuning, which have high probability (ie, statistically significant) to yield positive results.

3. While early stopping helps, at what point we should stop?

If we have a perfect measure of model performance, we should stop exactly at that point. However, in MARL community, a common way is to optimize cooperative performance greedily, which we find it will hurt robustness and resilience after a certain point due to overfitting. As such, while our focus on cooperation, robustness and resilience may not be a perfect measure, it can be a good enough starting point to start with in real world, and is much better than optimizing cooperation performance only. In practice, engineers can estimate the range of uncertainties in their own tasks, and perform early stop accordingly.

4. While high critic learning rates and leaky relu works better, are there any environment not the case?

In fact, there are environments and uncertainties that are not the case. For example, higher critic LR slightly hurts resilience in Dexhand and Leaky ReLU slightly hurts resilience for MADDPG. However, in all experiments we conduct, higher critic LR works better in 153/161 cases, and leaky relu works better in 148/161 cases. As a consequence, trying it first is likely to return positive result on average.

Q2: The hyperparameter search appears to just be a sweep over a certain range, is there not a more principled manner to do this? it will heavily depend on the environment and models.

To the best of our knowledge, there do not exist a theoretically rigorous way to define ideal hyperparameter sweeps (otherwise we can tune MARL optimally and analytically!). Thus, we select parameters that are commonly used in existing codebases, including the codebase of MAPPO [1] and HAPPO [2]. These codebases includes a set carefully designed hyperparameters that works well for MPE, SMAC and MAMujoco, which varies in different tasks. Among all hyperparameters in these codebase, we conduct a pilot study to select hyperparameters that works well in most of our real-world tasks, and select other hyperparameters as alternate solutions.

[1] The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games. NeurIPS 2022.

[2] Heterogeneous-agent reinforcement learning. JMLR 2024.

Q3: the related work is stated at the start of the paper to help frame the work.

Thanks for the suggestion. We will move that section in final version.

Q4: Can findings on general statements be generalized?

We believe our findings can be generalized as follows. Our findings suggests certain tuning directions that are statistically significant (i.e., likely to work on average), including higher critic LR, early stopping and Leaky ReLU. Moreover, while parameter sharing, GAE and PopArt are previously regarded as panacea to cooperative MARL, previous researchers may treat this as default. Our finding shows it is merely a design choice, revealing blind spots that previous researchers may overlook. We do not claim our findings to be a silver bullet that helps any task and any algorithm under any random seed, but it is a promising direction for optimization.

As for the claims in Section 4.1 and 4.2, we believe those evidence are general enough to guide the design of robust MARL algorithms. In Section 4.1, we suggest focusing on the more harmful attacks. In Section 4.2, we suggest focusing on attacks over all modality and agent scopes. We summarize our general suggestions as follows: