Policy Consistency in Multi-Agent Reinforcement Learning with Mixed Reward

We sincerely thank Reviewer LvJw for the recognition of our work and for providing constructive comments.

Question 1: If the rewards are inappropriate, how does the lower bound of this framework compare to the original training method without additional rewards?

Answer 1: Thanks for your insightful comments. Following the reviewer’s suggestion, we conducted additional experiments by setting individual rewards randomly from [-1,1] in the MPE environment. These results have been added to Section 5.4 of the revised paper:

Here, CMT(RD) represents CMT with random individual rewards. The results demonstrate that with inappropriate individual rewards, CMT(RD) performs similarly to MAPPO(Sparse), both showing limited effectiveness. This suggests that even in worst-case scenarios with poorly designed individual rewards, our approach maintains a performance floor equivalent to methods that operate without additional rewards.

Question 2: When the scale of individual rewards is too large, even exceeding that of team rewards, will the method in the paper still ensure policy consistency?

Answer 2: Thank the reviewer for this valuable question. In Table 3 of the revised manunscript, we added a new ablation study where the individual rewards are amplified three times and five times. Under such settings, the absolute value of individual rewards received in one episode has exceeded the team reward greatly.

CMT(3×) denotes the CMT developed with individual rewards amplified three times, while CMT(5×) indicates the CMT developed with individual rewards amplified five times. It can be observed that the maximum performance variation remains below 30% across all scenarios. These results demonstrate CMT's robustness to reward magnitude variations while maintaining consistent performance.

Question 5: What are the key differences between Equation 9 in this paper and Equation 7 in IRAT[1]?

Answer 5: There are two key differences between the two Equations.

• Firstly, IRAT utilizes a standard TD error, while our approach presents an extended TD error. The standard TD error incorporated in IRAT uses only individual rewards for policy evaluation, while CMT incorporates team rewards with a Lagrangian multiplier $\lambda$ into TD error. The team reward is a better metric (than the individual reward) to evaluate policy performance, and the introduction of the Lagrangian multiplier $\lambda$ represents the policy consistency constraints in the proposed Lagrangian dual problem. These innovative changes enable more accurate policy evaluation under consistency constraints, resulting in policies with higher team rewards and lower variance.

• Further, IRAT utilizes the individual reward to develop the individual policy $\pi_i$ and its state-value function $V$ for each agent $i$, while we utilize the mixed reward to develop the mixed policy $\pi_{E+i}$ and its state-value function $V_{E+i}$. Numerical results show that the use of mixed rewards can better balance individual skill execution and group collaboration.

Question 6: When comparing CMT with LAIES and MASER, does CMT use rule-based individual rewards of LAIES/MASER-constructed individual rewards?

Answer 6: When comparing CMT with LAIES and MASER, our CMT utilizes the individual rewards of LAIES presented. It is a heuristic reward, which is different from the rule-based reward we utilized when comparing CMT with IRAT, QMIX, and MAPPO. In the third paragraph of Section 5.1 of revised manuscript, we have revised the introduction of heuristic individual rewards:

“LAIES (Liu et al., 2023) and MASER (Jeon et al., 2022) implement their respective individual rewards, as introduced in Section 2. Since the individual reward in MASER relies on the mixing network of QMIX, which is incompatible with other types of MARL algorithms, our approach employs the same reward setting as LAIES.”

[1] Li Wang, Yupeng Zhang, Yujing Hu, Weixun Wang, Chongjie Zhang, Yang Gao, Jianye Hao, Tangjie Lv, and Changjie Fan. Individual reward assisted multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), pp. 23417–23432. PMLR, 2022.

We sincerely thank Reviewer c1Jb for the constructive and valuable comments. The concerns are addressed as follows.

Question 1: Clarify the main contributions of this paper, particularly compared with IRAT.

Answer 1: Thank you for the constructive suggestion, which is helpful in improving the quality of this paper. In the Introduction and Related Work sections of the revised manuscript, we emphasize the main contribution of this paper on the elimination of policy inconsistency, which is also one of the key differences between IRAT and our work.

While IRAT (Wang et al., 2022) mitigates the policy inconsistency issue through improving policy similarity between learned and team policies, our approach completely eliminates this issue by deriving exact policy objectives from a constrained Lagrangian dual optimization model. The standard TD error incorporated in IRAT is based only on individual rewards for policy evaluation, while CMT incorporates team rewards with a Lagrangian multiplier $\lambda$ into TD error. The team reward is a more comprehensive metric (than the individual reward) to evaluate policy performance, and the introduction of Lagrangian multiplier $\lambda$ enforces policy consistency constraints in the dual optimization problem. These innovations yield policies with higher team rewards and reduced variance.

Moreover, unlike IRAT’s focus on individual rewards, our approach incorporates mixed rewards during training, better balancing individual skill execution and group collaboration.

Question 2: An analysis of the key design elements contributing to the performance of CMT would strengthen the paper.

Answer 2: Thank the reviewer for this valuable suggestion. In Section 5.4 of the revised manuscript, we have added ablation studies to analyze the three key components of CMT: extended TD errors ($U_{E+i}$ and $U_{E}$), policy approximation based on similarity assumption, and KL divergence-based optimization objectives reconstruction.

Where i) CMT-TD(w) is CMT with the extended TD error replaced with the standard TD error (defined as Eq. 15) used by MAPPO and IRAT; ii) CMT-SP(w) is CMT with the policy approximation removed by initially using distinct parameters between mixed and team policy networks; iii) CMT-KL(w) represents CMT without the KL terms-based reconstruction module.

The results reveal that the inclusion of the extended TD error exerts the most significant impact, followed by the KL divergence-based optimization objective reconstruction model, with the policy approximation technique showing the least influence on algorithm performance. These findings align with our core contribution outlined in the last answer. The extended TD error, which originates from resolving the policy consistency-constrained Lagrangian dual problem, notably contributes to CMT’s enhanced performance.

Question 3: A more uniform evaluation across baselines when comparing CMT with LAIES and MASER would provide a clearer comparison. Could the authors elaborate on the rationale for selecting evaluation tasks?

Answer 3: Thank you for raising this question. In Table 2, we benchmark CMT against LAIES and MASER in 5 maps of SMAC environment. For the remaining 6 tasks (8m_vs_9m, 3s5z, 3s_vs_5z, 3s5z_vs_3s6z, 2s3z and 6h_vs_8z), both LAIES and MASER achieve a zero reward, and the proposed CMT performs poorly as well. We therefore did not include the numerical results on these 6 tasks.

Question 4: The notation $CD_{KL}$ should be explained upon first use, or the more commonly used $D_{KL}$ could be adopted for clarity.

Answer 4: Thank the reviewer for the constructive suggestion. We have adopted a more common notation $D_{KL}$ in Eqs. (10) and (13) of the revised manuscript.

Comment 3: Assumption 1. If $\pi_{E+i}$ and $\pi_E^i$ share the same policy space, Assumption 1 holds by default. I don't understand why this assumption is made explicitly.

Answer 3: Thank you for this insightful question. Assumption 1 is essential for two reasons:

• It ensures satisfaction of Slater's condition, which is crucial for establishing the equivalence between the original policy optimization problem and its Lagrangian dual formulation.

• While this assumption naturally holds when $\pi_{E+i}^i$ and $\pi_{E}^i$ share the same policy space, it remains necessary for analytical completeness, particularly when direct proofs are challenging. This approach aligns with similar assumptions in recent work [4].

Comment 4: I don't understand how two policies merge into one execution policy.

Answer 4: Thank you for noting this ambiguity. To clarify: During execution, agents exclusively use the mixed policy for environmental interactions. The team policy is only involved in training but not in actual execution.

We have revised the first paragraph of Section 4.3 to make this distinction explicit: “During policy execution phase, each agent utilizes the actor network of mixed policy to interact with environment."

[4] Yang Zhang, Bo Tang, Qingyu Yang, Dou An, Hongyin Tang, Chenyang Xi, Xueying Li, and Feiyu Xiong. Bcorle (λ): An offline reinforcement learning and evaluation framework for coupons allocation in e-commerce market. Advances in Neural Information Processing Systems, 34:20410–20422, 2021.

We thank Reviewer 1Pd6 for the valuable comments. We address the concerns as follows.

Comment 1: There is a gap between the actual implementation of CMT and its aim. The approximations made in CMT cause I am not convinced that CMT improves over previous methods in terms of fixing the policy inconsistency issue.

Answer 1: Thank you for highlighting the gap between the implementation of CMT and its intended goal. In the revised manuscript, we have added discissions to highlight that the approximations (resulting from the similarity assumption and Assumption 1) do not significantly impact CMT's effectiveness in addressing policy inconsistency.

• Firstly, we utilize a key implementation technique to ensure that the "similarity assumption" almost always holds in implementation. We initialize the mixed and team policies networks with the same parameters, contributing to the minimal disparity between the two types of policies.

• Secondly, we have added an ablation study to compare the performance between IRAT and CMT without shared parameter initialization in Section 5.4:

The experimental results demonstrate that even without shared initialization (CMT-SP(w)), our approach significantly outperforms IRAT, validating CMT's effectiveness in addressing policy inconsistency.

• Thirdly, Assumption 1 requires that there exists a mixed policy $\pi_{E+i}^i$, the performance of which can match that of optimal team policy ${\pi_E^i}^∗$ concerning the cumulative team rewards $J_E$ . As the reviewer has correctly noted, this assumption naturally holds in our setting where $\pi_{E+i}^i$ and $\pi_E^i$ share the same policy space. We believe that this assumption does not limit CMT's practical effectiveness.

Comment 2: Lack of theoretical results that ensure the optimization procedure of CMT leads to the optimal team policy.

Answer 2: We appreciate your observation regarding theoretical results. While we acknowledge this limitation, the challenge stems from two key factors:

• The inherent complexity of multi-agent environments;

• The general instability of MARL algorithms.

It's worth noting that theoretical guarantees are rare in contemporary MARL algorithms with sparse rewards, including well-established methods like IRAT[1], LAIES[2], and MASER[3]. While we recognize the importance of theoretical foundations, the current state of MARL with sparse rewards often prioritizes empirical validation. We are committed to developing theoretical guarantees in future work.

[1] Li Wang, Yupeng Zhang, Yujing Hu, Weixun Wang, Chongjie Zhang, Yang Gao, Jianye Hao, Tangjie Lv, and Changjie Fan. Individual reward assisted multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), pp. 23417–23432. PMLR, 2022.

[2] Boyin Liu, Zhiqiang Pu, Yi Pan, Jianqiang Yi, Yanyan Liang, and Du Zhang. Lazy agents: a new perspective on solving sparse reward problem in multi-agent reinforcement learning. In International Conference on Machine Learning (ICML), pp. 21937–21950. PMLR, 2023.

[3] Jeewon Jeon, Woojun Kim, Whiyoung Jung, and Youngchul Sung. Maser: Multi-agent reinforcement learning with subgoals generated from experience replay buffer. In International Conference on Machine Learning (ICML), pp. 10041–10052. PMLR, 2022.

We thank reviewer WpPH for the valuable and constructive comments. We address the concerns as follows.

Comment 1: The presentation on Introduction and Related work can be improved.

Answer 1: Thank you for your valuable suggestion. We have substantially revised the Introduction and Related Work sections to improve clarity and conciseness. Major improvements include:

1. Introduction Section:

• Add more discussion to distinguish this work from IRAT, by articulating two key advantages of the proposed approach in Paragraph 4:

$\qquad$① Fully address the policy inconsistency problem by solving a policy consistency-constrained Lagrangian dual optimization problem;

$\qquad$② Balance individual skill execution and group collaboration by incorporating mixed rewards instead of only using team rewards.

• Streamline the technical description in Paragraphs 5-6 to highlight our approach's three key components:

$\qquad$① Extended TD error derivation using performance difference Lemma;

$\qquad$② Policy approximation techniques;

$\qquad$③ Optimization objective reconstruction with KL terms.

2. Related Work Section:

• Consolidate similar works [1,2,3] into cohesive thematic groups for more efficient presentation.

• Eliminate redundant descriptions, particularly regarding IRAT, which have been covered in the Introduction.

The updated version is also available on OpenReview.

Comment 2: Including the algorithm in the main paper would facilitate readers' understanding of the algorithm.

Answer 2: Thank you for this constructive suggestion. We have incorporated the CMT algorithm's pseudo-code into the main paper. We also added more details about the algorithm implementation to improve readability.

Comment 3: Equation (7) in the main text lacks a definition for U. While it is defined in the appendix, it would be better to provide its meaning upon its first appearance in the main text.

Answer 3: Thank the reviewer for identifying this oversight. We have placed the definition of U at its first appearance in Eq. 7, adding its mathematical formulation and interpretation for clarity.

Comment 4: Clarify what the “performance” refers to and explain about “surpass ${\pi_{E}^i}_{*}$” in Assumption 1.

Answer 4: Thank you for noting this ambiguity. We have clarified that "performance" specifically refers to the cumulative team reward $J_E$. We have also replaced the imprecise term "surpass" with "approaches or even matches" for clarity.

To enhance readability, we have revised the interpretation of Assumption 1 as: " Assumption 1 requires that there exists a mixed policy $\pi_{E+i}^i$ (developed by the mixed reward in Eq. 2), the performance of which can match that of optimal team policy ${\pi _E^i}^*$ (defined in Eq. 1)) concerning the cumulative team rewards $J_E$."

[1] Iou-Jen Liu, Unnat Jain, Raymond A Yeh, and Alexander Schwing. Cooperative exploration for multi-agent deep reinforcement learning. In International Conference on Machine Learning (ICML), pp. 6826–6836. PMLR, 2021.

[2] Pei Xu, Junge Zhang, and Kaiqi Huang. Exploration via joint policy diversity for sparse-reward multi-agent tasks. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence (IJCAI), pp. 326–334, 2023a.

[3] Pei Xu, Junge Zhang, Qiyue Yin, Chao Yu, Yaodong Yang, and Kaiqi Huang. Subspace-aware ex- ploration for sparse-reward multi-agent tasks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 11717–11725, 2023b.