Variational Inequality Methods for Multi-Agent Reinforcement Learning: Performance and Stability Gains

Question 4: About "on the rewards as a metric in MARL": the authors claim that we need stronger evaluation metrics in MARL. While I agree that reward cannot fully reflect agent's behavior, I am not sure if other metrics could be called as "stronger" as the goal of multi-agent RL, like any RL, is to maximize reward.

In MARL, an agent's rewards depend on the policies learned by other agents—represented as $f_i(x_1, x_2, \dots, x_N)$. This interdependence can result in no discernible difference in rewards between agents that successfully learn to solve a problem and those that do not. For precise examples, please refer to the rock-paper-scissors game in Fig. 3 and the last paragraph of Section 5.2. Consequently, saturated rewards do not indicate that an algorithm has converged to a stable state (because the total sum can be constant while the learned policies rotate in parameter space) or that the overall performance is satisfactory. Furthermore, it is possible for some agents to fail to begin learning altogether. This issue is not unique to MARL but also applies to other multi-player machine learning setups. Unlike classification, where lower $f(x)$ loss implies better performance on training data, in games, this is not the case. For example, in GANs, a low generator loss (that depends on the discriminator) does not indicate good performance, which is why external metrics like FID or IS scores are commonly used. Thus, while we agree that the learning objective in multi-agent RL, like any RL, is to maximize reward, this cannot be used as a performance comparison between methods. Our findings and output inspections suggest the importance of using environments where external metrics can reliably quantify performance. For example, using games where it is possible to assess how effectively the agents solve a given task. A more detailed evaluation should also include measuring whether all agents contribute meaningfully to the task's completion.

---

Thank you again for your feedback and time. If your concerns have been addressed, we kindly ask you to consider revising your score. If there are any remaining questions or points requiring clarification, please do not hesitate to let us know—we would be happy to provide additional details.

Question 1: while it is well-known that MARL with gradient descent is hard to converge, is there any theoretical proof that other VI methods are better than gradient descent when it comes to this special case of VI problems?

Yes, under standard assumptions on the agents’ reward functions, such as convexity, the VI becomes monotone, and the methods we consider have convergence guarantees in this case. Thank you for bringing up this question; we have added the paragraph On the convergence in Section 4.2 to address your feedback.

Question 2: Can the authors explain why EG does not seem to improve performance as significant as LA, and EG-LA-MADDPG does not work well in the MPE-Predator-prey game and the rock-paper-scissor game in Fig. 5?

Thank you for raising this question. We included a comparison with the widely used Extragradient (EG) method for solving VIs, which is known for its convergence for monotone VIs (unlike GD). However, EG only introduces a minor local adjustment compared to GD. As such, the results align with expectations—while EG occasionally outperforms GD (the baseline), its performance is often similar. In contrast, nested LA applies a significantly stronger contraction, with the degree of contraction increasing as the number of nested levels increases. This leads to substantial performance gains, particularly in terms of stability, as it prevents the last iterate from diverging. However, there is a tradeoff: if the number of nested levels is too high, the steps can become overly conservative or slow. Based on our experiments, three levels of nested LA yielded the best results (see Fig. 1-a). It is also worth noting that similar trends have been observed in GANs: while EG performs slightly better than GD, more contractive methods consistently achieve better results. This suggests that the MARL setting involves a substantial rotational component, making it a promising area for further exploration. Thank you for pointing out that this is unclear; we have added the above explanation to Section 5.2 in the paragraph Comparison among [...] for further clarity.

Question 3: Is it possible to reach Nash equilibrium by tuning the hyperparameters more? It would be great if the authors could provide more attempts on achieving Nash equilibrium by tuning hyperparameters (which is also an ablation to hyperparameter sensitivity).

Thank you for this suggestion. With additional hyperparameter tuning, we observed significantly better performance; please refer to the updated Figure 1(a) for details. These results highlight the critical role of optimization in MARL. While this paper does not aim to address all aspects of the MARL training difficulties, we hope that these findings will encourage further exploration into automated hyperparameter selection specifically for MARL. To clarify, we do not claim to fully eliminate hyperparameter sensitivity. However, we show that by employing appropriate optimization methods, the high variability in performance—caused by different random seeds and hyperparameters—is significantly reduced. This is particularly important as the baseline exhibits substantial variance even with only changes to random seeds.

Thanks a lot for your time reviewing our paper and for your valuable feedback.

W1: contribution to the MARL community

MADDPG is largely outdated due to its difficulty to train and instability.

We compared MADDPG with MATD3 ([1] Ackermann et al., 2019); please see Figure $10$ in Appendix A.4 for details. Our observations show that the performance of both algorithms fluctuates between high and low rewards, indicating similar behavior. This suggests that our proposed methods, which address rotational dynamics, would be beneficial for both algorithms. Our experiments are designed to demonstrate relative improvements over the baseline. Therefore, the insights gained from our work are applicable to similar MARL algorithms that also involve competitive learning objectives.

[1-3] are not referenced in this paper.

Thanks, we added these.

In fact, instability can still be witnessed in Fig. 3 and I am curious whether changing an RL algorithm would fix such issue.

Competitive learning objectives (as described in Eq. $F_{MADDPG}$) naturally give rise to rotational dynamics, which necessitate optimization methods capable of addressing these challenges. While we do not claim that making learning objectives more ‘cooperative’ wouldn’t lead to more stable training, it could potentially reduce efficiency, as competition often facilitates a faster search process. Additionally, not all applications allow for cooperative adjustments. We acknowledge that changes to the RL algorithm itself could further improve performance. However, when competitive objectives are unavoidable, it remains essential to use methods designed to handle the resulting rotational dynamics effectively.

W2: It is unclear whether the property of being closer to equilibrium in simple tasks will bring better performance for more complicated mainstream MARL benchmarks such as SMAC.

We demonstrate improvements across multiple tasks where accurate comparisons between methods are possible. Based on our findings, we believe that if the agents’ learning objectives are competitive, the benefits of our approach will likely extend to SMAC as well. Otherwise, we anticipate that the methods will perform no worse than the baseline.

Thank you for your detailed response. Here are my follow-up questions:

1. When I write "I am curious whether changing an RL algorithm would fix such issue", I mean changing MADDPG to newer algorithms (e.g. MATD3), not changing reward function to make the agents cooperative.

2. I appreciate the authors' effect in introducing results of MATD3, but I feel this paper should more based on MATD3 instead of MADDPG as the latter (like other reviewers pointed out) is outdated. Currently, the story of the paper is still focused on MADDPG (e.g. "We introduce three algorithms that use nLA-VI, EG,and a combination of both, named LA-MADDPG, EG-MADDPG,and LA-EG-MADDPG,respectively" in the abstract).

3. I feel the "on the convergence" paragraph should be elaborated. More specifically:

a) Why does monotonicity ensures convergence? The paper claims that it is "known" in line 462-463 without reference. Can the authors add related theorems to make it more self-contained?

b) What is the "standard assumption" on the agent's reward functions, and why under this assumption VI becomes monotonic?

4. The current work does not seem to study scalability where the number of agents are much more than 3; for example, there could be over two dozens of agents in SMAC. Can the proposed method scale when there are more agents in the environment?

Thank you for taking the time to review our paper and for providing valuable feedback.

W1: The specific problem with MADDPG that the paper addresses is unclear. It appears the focus is primarily on a general optimization issue rather than a specialized challenge within MARL.

Our work focuses on a general MARL optimization issue rather than a specialized problem specific to a MARL algorithm, and it does not involve introducing changes to the reward functions. Instead, we demonstrate that by improving the optimization—while keeping all other components fixed—we achieve significant gains in both algorithmic stability across random seeds and performance, measured as the distance to equilibrium. For example, please refer to Figure 1 for evidence of these improvements. We view this as a key strength of our work, as it targets a fundamental challenge in MARL. This positions our approach uniquely, bridging concepts from two areas of machine learning to address a critical issue affecting both MARL research progress and deployment.

In response to this feedback, we have updated the introduction to better highlight our motivation and added references to emphasize the significance of the problem and its impact on MARL research and applications.

W2: The proposed methods seem straightforward, primarily involving a switch from the Adam optimizer to existing VI methods. The motivation for this switch is not adequately explained, and there is a lack of theoretical justification for how LA or EG methods can specifically benefit MADDPG or MARL algorithms.

Our work represents an initial step toward establishing a foundation for further development of VI methods in MARL. Our results highlight significant performance gains, demonstrating their timely relevance for MARL challenges.

The VI perspective we present frames MARL as a VI instance, revealing that it generally involves rotational learning dynamics. When addressed with standard optimization methods, these dynamics can lead to training instabilities. For theoretical justification of the proposed methods under standard assumptions, please refer to the paragraph On the convergence in Section 4.2. Please let us know if further clarifications could be helpful.

W3: simplistic experiments

We demonstrate improvements across multiple tasks where accurate comparisons between methods are possible using distance to the equilibrium. Our findings, consistent across the experiments, imply that if the agents’ learning objectives are competitive, the benefits of our approach will extend to other benchmarks as well.

Question: In Figures 2 and 6, the evaluation episodes provide limited information. I would appreciate seeing the learning curves for the two MPE experiments.

Thank you for the question. We added those results, please refer to figures $8$ and $9$ and section A.3.4 for the learning curves of MPE experiments.

---

Thanks again for your feedback. Please consider increasing your score if your concerns are resolved. Otherwise, please let us know and we will be happy to provide further clarifications.

Thanks a lot for your time reviewing our paper and for your feedback.

W1-motivation: high sensitivity to hyperparameters in RL

To clarify, our work does not aim to address hyperparameter tuning or the elimination of hyperparameters. Instead, our motivation lies in tackling the broader challenges of MARL training, commonly referred to as the MARL reproducibility crisis [1]. These challenges include:

• The well-documented difficulty of MARL training, where convergence may fail to initiate.
• Significant performance variability caused by small changes in hyperparameters or initial random seeds. Our results address challenge (1) and partially mitigate (2), as the variance across random seeds is notably reduced. Furthermore, the comparison of the proposed VI methods across benchmarks shows consistent and expected improvements; see the updated paragraph Comparison among [...] in Section 5.2 for more details.

Thank you for highlighting this point. We have updated the introduction to better address these aspects; please see Section 1, particularly the second paragraph.

W2: [....] it would be beneficial if the authors provided results of the proposed method applied to other algorithms

We compared MADDPG with MATD3 (Ackermann et al., 2019); please see Figure $10$ in Appendix A.4 for details. Our observations show that the performance of both algorithms fluctuates between high and low rewards, indicating similar behavior. This suggests that our proposed methods, which address rotational dynamics, would be beneficial for both algorithms. Our experiments demonstrate relative improvements over the baseline. Therefore, the insights gained from our work are applicable to similar MARL algorithms that also involve competitive learning objectives.

W2: Figure 2 does not effectively illustrate the benefits of the proposed method.

For the predator-prey benchmark, rewards are not indicative of performance as they depend on the policies of other agents (see On the rewards as a metric in MARL in Section 5.2). Instead, we use the win rate of one team as a metric in Figure 2 and provide reward values and algorithm snapshots in the appendix.

Metric in Fig. 2. If all agents develop strong policies, the adversary team’s win rate against the agent should approach $0.5$, as the agent has a speed advantage, but coordinated adversaries can improve their chances of capturing it. A win rate closer to $0.5$ indicates better adversarial coordination, observed with LA-MADDPG. In contrast, baseline MADDPG often results in only one adversary learning to chase, while the other moves away or remains stationary, leading to a lower win rate due to poor coordination.

Other metrics in Appendix. Visual snapshots in Figure 7 show that baseline MADDPG often diverges, moving away from targets, which highlights suboptimal behavior. Figure 8 provides the rewards during training for completeness.

---

[1] BenchMARL: Benchmarking Multi-Agent Reinforcement Learning, Bettini, Prorok, Moens, 2024.

---

Thank you again for your valuable feedback. If we addressed your concerns, please consider revising your score. Otherwise, if there are any remaining questions or clarifications needed, please let us know.

W2: controlling for NE convergence

Thank you for your perspective. While we understand your point, theoretical guarantees, as in GANs and deep learning, often do not fully apply to general neural networks. Our approach builds on recent VI optimization advancements, highlighting the role of rotational dynamics in MARL. Please see also paragraph On the convergence in Section 4.2 of the updated version. While we acknowledge Mazumdar et al., 2019, proposing a second-order method, such approaches are computationally prohibitive. Hence, we focus on first-order VI methods that are more practical yet effectively address rotational dynamics.

W3: results

The presentation of the results lacks clarity and does not convincingly demonstrate the efficacy of integrating MADDPG with VI techniques.

Thank you for raising this point. Our results consistently show improvements in equilibrium distance and reduced variance across random seeds. In summary:

• The baseline (in blue) exhibits high variance across initial random seeds and often diverges.
• EG, with only minor differences from the baseline, shows slight improvements but not always significant.
• Nested LA, which introduces a stronger contraction, achieves significant performance gains.

Therefore, we respectfully disagree with the above assessment, as our findings provide strong evidence of the impact of the proposed optimization methods on MARL performance. Figure 1-a illustrates that simply improving the optimization process leads to substantial performance gains.

Further details on the consistent outperformance of the methods across benchmarks are provided in Section 5.2, specifically in the paragraph Comparison among [...]. We added the paragraph Summary in Section 5.2 in response to your feedback on the confusion. Please let us know if additional clarification would be helpful.

What is point of using MADDPG + VI in games?

As mentioned above, we leverage VI-solving methods to address the rotational dynamics inherent in MARL training, which traditional gradient-based approaches struggle to handle effectively. This focus is core to our methodology, and we would be happy to provide further clarification if needed.

In Fig. 2, the proposed method does not show any improvements during learning. Why?

This behavior is expected due to the metric used. The $y$-axis in Fig. 2 represents the average win rate of adversarial agents. While both teams are improving their policies over time, the plotted win rate reflects performance relative to the opposing team at a given iteration ($x$-axis). As both teams learn and adapt simultaneously, the win rate fluctuates, even as the overall quality of policies improves.

Moreover, the high variance observed in the results casts further doubt on the method's reliability and robustness. Such discrepancies point to a need for more rigorous experimental design and a deeper analysis to genuinely assess the method's impact on performance in MARL settings.

We respectfully believe there may be a misunderstanding here. The high variance observed in the baseline results is precisely the motivation for our work, as this issue is well-documented in MARL literature (discussed in Sec. 1). Our methods aim to mitigate this variance by addressing the underlying rotational dynamics, which we show is a key challenge in MARL. Our approach has demonstrated significant improvements in stability and performance across benchmarks.

---

[1] BenchMARL: Benchmarking Multi-Agent Reinforcement Learning, Bettini, Prorok, Moens, 2024.

[2] Last-Iterate Convergence of Saddle Point Optimizers via High-Resolution Differential Equations, Chavdarova, Jordan, Zampetakis, 2023.

---

We would like to thank the reviewer again for their comments and time. Please consider increasing your score if you believe your concerns have been resolved. Otherwise, please let us know and we will be happy to provide further clarifications.