Multi-Agent Interpolated Policy Gradients

We express our sincere gratitude to Reviewer 4F8p for their invaluable insights, which have significantly enhanced the robustness of our work. In response to the specific concerns raised, we present a thorough clarification of each point.

1. Related works.

In the revised version of our paper, we have incorporated a discussion on references [a][b][c].

Reference [a] is particularly relevant to our work as it addresses variance, and we emphasize that there is no conflict between [a] and our method. It is possible to integrate an optimal baseline from [a] into our method by replacing the advantage estimator in Equation 10 with Equation 12 from [a]. However, this modification requires the estimation of joint Q-functions, and GAE is not compatible, potentially impacting the performance of PPO-based gradients.

Reference [b], a milestone work on value factorization, shares similar ideas when analyzing the Bellman operator of a factorized function class, and we have included it in the related works.

Reference [c] proposes a promising approach using switch control to dynamically switch from independent learning to centralized learning. We believe similar methods could be applied to dynamically adjust the weight parameter $\nu$ in our approach. However, achieving this requires further efforts in both theoretical and experimental aspects, as discussed in the related and future works section of our revised paper.

[a] Kuba, Jakub Grudzien, et al. "Settling the variance of multi-agent policy gradients." Advances in Neural Information Processing Systems 34 (2021): 13458-13470.

[b] Rashid, Tabish, et al. "Weighted qmix: Expanding monotonic value function factorisation for deep multi-agent reinforcement learning." Advances in neural information processing systems 33 (2020): 10199-10210.

[c] Mguni, David Henry, et al. "MANSA: learning fast and slow in multi-agent systems." International Conference on Machine Learning. PMLR, 2023.

2. Analytical Comparison of Bias and Variance.

Regarding the question on achieving reduced bias and variance, our method intentionally introduces bias to reduce variance compared to the standard unbiased objective.

In Equation 10, the first term, requiring a sample from the joint action space, contributes to high variance, while the second term does not. Consequently, in contrast to the standard objective ($\nu=0$), our method reduces the variance of the high variance term by a factor of $1-\nu$, simultaneously introducing a low variance term with a coefficient of $\nu$. For a more comprehensive understanding, refer to Equation 13, where the variance of the high variance term is explicitly reduced by $(1-\nu)^2$.

It's important to note that, as the standard objective is unbiased, our method does not achieve reduced bias compared to it. The term "low bias" in our context reflects that the additional bias introduced by the factorized function is comparatively modest. This acknowledgment is particularly pertinent considering that a factorized function has the potential to introduce significant bias.

3. Performance with Different Maxima of $Q^\pi$ and $Q^\mu$.

While we do not directly use $Q^\pi$, we understand the concern about the potential impact on policy performance if $Q^\pi$ and $Q^\mu$ have different maxima. Although combining two value functions with different maxima could lead to a suboptimal policy, we argue that such a scenario is unlikely to occur and is rarely observed. To demonstrate this, we conducted a matrix game, the details of which are provided in Appendix D3 of the revised paper. The nature of learning Q functions and our choice of on-policy implementation contribute to mitigating the likelihood of disparate maxima between $Q^\pi$ and $Q^\mu$.

Thanks.

1/Related works.

My main concern remains as it seems that reference [c] seems to do something similar to solving this problem but allows the value of $\nu$ to vary at each state which seems to be a more powerful approach. In light of this I would like to have seen some statements as to why this paper is a useful contribution given [c].

2 + 3/bias

My concern here is that the method introduced here seems to lead to biased solutions (and maybe even in the asymptotic training regime) - using this method it seems we lose convergence guarantees to any useful stable point. If so, because in general the stable points/equilibra of multi-agent systems are extremely sensitive to the objective parameters we could end up converging to a point a long way from any sort of local optimum (with arbitrarily bad solutions). This seems to be a significant issue.

I would like to see at the very least some analysis on the conditions when this bias would be relatively small. Alternatively, if the authors could show that the bias is small in a vast number of randomly generated games that could help.

Thanks. These responses have allayed my concerns.

If the authors could incorporate their responses, particularly, precise technical statements for point 2 in the script, I would be happy to raise my original score.

Apart from that, I believe that the empirical support for the authors' claim in point 2 would be greatly strengthened if the paper included supporting experiments across a range of (say, randomly generated) normal form games (for which standard methods converge).

We extend our sincere appreciation to Reviewer ZGxu for the thorough evaluation of our manuscript. We are grateful for the time and dedication you have invested in providing constructive feedback. Below, we address each of the reviewers' concerns individually.

1.The Incremental Nature of Results.

We would like to clarify certain aspects where there might be a misunderstanding regarding the novelty and significance of our work.

Firstly, our approach significantly deviates from single-agent papers referenced in Section 5. The integration of a joint Q-function and factorized function in our work is motivated by a distinct trade-off between expressive ability and learning difficulty. As detailed in Appendix A.1, our exploration of a convex combination highlighted its unsuitability for value-based methods, leading us to focus on policy-based methods. Although the resulted policy gradients share some similarity with IPG, our motivation is fundamentally different from any single-agent paper referenced.

Secondly, our theoretical results, particularly Proposition 4, provide distinctive insights into bounding the objective function. The analysis of bias induced by a factorized Q function, discussed in Section 4.2, represents a novel contribution absent in single-agent settings. Additionally, in Section 4.3, we introduced the concept of compatible function approximation in multi-agent reinforcement learning, exploring its relationship with value factorization.

Thirdly, the “different approach in the implementation (on-policy instead of off-policy)” is not an arbitrary choice. It aligns seamlessly with our theoretical findings, as used in Equation 14 and later on, and serves to prevent poor performance, as detailed in Appendix F. This approach is a deliberate and well-justified decision based on our theoretical framework.

In summary, while there may be similarities in addressing the bias-variance trade-off, our work represents a distinct and valuable contribution to the field. The strong relationship between the idea and bounds to value factorizations in the multi-agent settings sets our paper apart. Our implementation choices are grounded in the theory and have been carefully considered to yield consistent and meaningful experimental results. We believe these aspects collectively establish the novelty and significance of our proposed method.

2. Inconsistent Notations.

We appreciate the reviewer's observation regarding inconsistent notations, and we would like to provide further clarification. In our paper, $J$ denotes the standard unbiased objective for policy gradient, while $\hat{J}$ represents the objective of our proposed method. Additionally, $\hat{Q}$ is introduced in the third line of the second paragraph of section 3.2, serving as an estimation of $Q^\mu$. To enhance consistency, we have updated the notations in the paper. Specifically, in Equation 12, we replaced $J$ with $\hat{J}$, and in Equation 10, $Q^\mu$ has been replaced by $\hat{Q}$.

3. Mistake in the Proof of Proposition 4.

Certainly, we acknowledge the oversight in Proposition 4. In response, we have revised both the proposition and its proof in the paper. Notably, we have refined the definition of $\sigma$ for conciseness.

We express our gratitude to Reviewer JyH2 for recognizing the strength of our work. We now address the concerns raised by the reviewer:

1. Concerns about the Empirical Results.

While our method outperforms other baselines, it is essential to address the QMIX's comparable performance in several SMAC maps. Notably, QMIX is an off-policy method, incurring higher computational intensity and operating at about one-third the speed of our method. We have incorporated an additional plot illustrating compute time on the x-axis for a comprehensive comparison, as shown in Figure 8 of the revision of our paper.

2. Rationale for Utilizing Recurrent Neural Networks.

As indicated in the fourth line of the preliminary section, our approach is designed for Dec-POMDP, and the choice of full observation is made to simplify theoretical analysis. Given that SMAC is partially observable, the integration of RNNs proves beneficial for enhancing performance.

3. Clarification on Intermediate Steps from Equation 19 to Equation 10.

We can directly define $\mu(s) := E_{\pi(\xi)}[f(s,\xi)]$ , then E_{\pi(a,\xi|s)}$$[\nabla_\theta f_\theta(s,\xi)$$\nabla_aQ(s,a)]= E_{\pi(a|s)}[\nabla_\theta \mu(s)\nabla_aQ(s,a)]. In other words, in the context of a Gaussian policy, where $a= f_\theta(s,\xi)=\mu_\theta(s)+\Sigma_\theta(s)^{1/2}\xi$ and $\xi\sim\mathcal{N}(0,1)$, we have $E_{\pi(\xi)}[f(s,\xi)] = \mu_\theta(s)$.

4. Impact of $\mu$ Value on Performance Across Scenarios.

In our experiments, we observed that a $\mu$ value of 0.3 in SMAC and 0.5 in GRF yielded optimal performance, which can hardly be considered "low". However, in tasks with unnormalized reward, the choice of $\mu$ depends on the reward scale. Notably, PPO employs normalized advantage, resulting in $Q^\mu$ values in our methods that may be one order of magnitude larger than $A^\pi$, necessitating a smaller $\mu$. For instance, in MPE, we set $\mu=0.05$.

5. Exclusion of QMIX in GRF.

Addressing the absence of QMIX in GRF, as explained in the third paragraph of section 6.1, QMIX was not included due to its reported inferior performance in the MAPPO's paper.