$f$-Divergence Policy Optimization in Fully Decentralized Cooperative MARL

The application of f-divergence in policy optimization is not new; a comprehensive analysis of various distance constraints in policy gradients has been provided in [1].

[1] discusses the convergence of policy gradient iteration for a general utility instead of the original reward function in single-agent RL. Though the general utility may cover the $f$-divergence case, our analysis and theoretical results including the formulation of policy iteration of $f$-divergence policy optimization are novel in fully decentralized learning.

Extending existing single-agent analysis to the multi-agent setting is reasonable, but some assumptions are questionable. Specifically, the approach assumes full observability in MARL making the setting difficult to distinguish from single-agent reinforcement learning. Under full observability, what meaningful difference remains between centralized and decentralized control?

For the assumption of full observability or global state, we have a more detailed discussion in Appendix F.6. In conclusion, this is necessary for meaningful theoretical analysis. Even under full observability, decentralized control is still not able to observe the other agents' actions and use a centralized critic to settle the non-stationarity issue like centralized control. There are some studies on the decentralized control under full observability including [1,2] mentioned by Reviewer vfTk.

[1] Leonardos, Stefanos, et al. "Global convergence of multi-agent policy gradient in markov potential games." arXiv preprint arXiv:2106.01969 (2021).

[2] Fox, Roy, et al. "Independent natural policy gradient always converges in markov potential games." International Conference on Artificial Intelligence and Statistics. PMLR, 2022.

The performance improvement appears marginal. With full observability, IPPO has already demonstrated near-optimal performance on SMAC and Multi-Agent MuJoCo. Were the baseline hyperparameters tuned to achieve their optimal reported performance?

The main difference lies in the experiment setting. As we mentioned in Section 5.2, all the algorithms use the independent parameter to agree with the fully decentralized setting, and parameter sharing is banned. Moreover, the SMAC tasks and Multi-Agent MuJoCo tasks are partial observable in our experiments. In this setting, IPPO can't perform as well as in the CTDE setting. As for the hyperparameters, We are running extra experiments about the clip parameters of IPPO. We will update the empirical results as soon as possible.

Why is win rate not used as the evaluation metric for SMAC-v2 tasks?

As we mentioned in Section 5.2, These tasks are difficult for fully decentralized learning, so we also use the cumulative reward as the metric. All the algorithms can hardly win in SMAC-v2 tasks, so the win rates may not show any difference.

The relevant work of CTDE is incomplete and lacks recent work, such as HASAC[a] and MAT[b].

Thank you for providing more related works. HASAC combines the heterougeneous-agent decomposition with the entropy regularization in SAC. MAT introduce Transformer and sequencial modeling into the heterougeneous-agent decomposition. We will update these contents in the revision.

Assuming global information might influence the impact of this work. Due to the assumption of the global state, I suggest using Markov games [a] as the multi-agent framework.

Thank you for your advice. We have a more detailed discussion about the global state assumption in Appendix F.6. In conclusion, it has been proven that the problem under the partial observable framework may be too difficult to obtain any useful analysis. The Markov games may be a good framework to skip the gap between the global state, but the gap between Markov games and existing partial observable benchmarks arises (that's the opinion from previous reviewers). We will follow your advice to use the Markov game framework in the revision, but we need to point out that changing the framework might make it easier to understand while may not change the essence of the issue.

While the experiment results appear promising, the contribution is slightly insufficient compared with existing work[c,d].

Mirror learning [c] is a very good paper on the convergence of single-agent RL algorithms and has provided us with a lot of inspiration on introducing a general distance or divergence constraint in the policy iteration. However, when extending its conclusion into the fully decentralized learning, we find that its core results about the monotonic improvement fail because of the other agents' influence. Therefore we need to propose some novel methods for

As for our contribution compared with DPO [d], first we propose a general framework of policy iteration, $f$-divergence policy optimization, in fully decentralized learning. We also provide a detailed analysis and discussion about $f$-divergence policy optimization and propose an algorithm TVPO with a convergence guarantee in fully decentralized learning based on this framework, which shows its potential. For the comparison between TVPO and DPO, we provide a detailed discussion in the Remark part of Section 4.2 and Appendix F.5. In conclusion, TVPO and DPO are based on different objective, and the approximation of TVPO is more accurate which means TVPO can avoid more trivial solutions.

Why use different metrics for SMAC (win rate) and SMACv2 (return)?

As we mentioned in Section 5.2, These tasks are difficult for fully decentralized learning, so we also use the cumulative reward as the metric. All the algorithms can hardly win in SMAC-v2 tasks, so the win rates may not show any difference.

My major concern is that this paper seems to miss several relevant literature. For instance, [1], [2] both proposed algorithms for independent learning in potential Markov games, which include the cooperative Markov games investigated in this paper. Further, [1] proposed a policy gradient algorithm and [2] proposed a policy iteration algorithm, which is highly relevant to this paper.

Thank you for providing two more baseline algorithms. We are running additional experiments about these two algorithms and we will update the empirical results as soon as possible.

Is the  $V^*$ in Theorem 4.6 the stationary point instead of the value function corresponding to the optimal policy?

Yes, $V^*$ here represents the value function corresponding to the converged joint policy $\pi^*$ and $\pi^*$ is not guaranteed to be the optimal policy (it will be sub-optimal in most cases).

About second line of Eq (23)

Thank you for pointing out our lack of rigor. Here we can replace the "$>$" with "$\ge$" in the conclusion and proof, then they will be correct. This change will not influence the corollary 4.3 and the counterexample we constructed. We will rewrite our statement in the revision.