DRIMA: Differential Reward Interaction for Cooperative Multi-Agent Reinforcement Learning

The authors would like to sincerely thank the reviewer for the careful reading and positive comments on our paper. The following are responses to the Weaknesses and Questions.

To the Weaknesses:

1. Please review the updated PDF of our paper, more results of SMAC scenarios have been added. The wide range of scenarios shows that DRIMA exhibited relatively slower learning efficiency at the early stage in some scenarios, but it did not impair learning results when run until convergence. In fact, distributed learning approaches compared with independent and centralized ways inherently possess slower learning rate, since the distributed communication between agents aim to reconstruct the useful global information which is directly available to centralized algorithms and is not considered by independent algorithms. However, independent (or centralized) and fast learning efficiency do not guarantee the overall collaborative effectiveness and robustness of the strategy combinations. DRIMA is suitable for real-world multi-agent tasks, where centralized payoff information is difficult or expensive to obtain, or that demands system robustness.
2. We give an explanation in the 1st reply, and from our experiments, the relatively slower sample efficiency of DRIMA had no detrimental impact on the final convergence results.

To the Questions:

1. Following the reward construction of SMAC in MAPPO (Yu et al., 2022), the individual rewards are set as three parts, the first one is an Enemy Group reward that sums up all enemies' instant health value which allows the agent to grasp the overall battle progress, the second one is a reward relating to an Ally agent's own health, and the final part is a win reward given to all Ally agents if win the battle.

We hope that the above responses have addressed your questions. If you have any further confusion, we'd be glad to discuss them with you.

The reviewer thanks the authors for their efforts. After reading the rebuttal and the updated draft, I still have some questions about the experiments. The authors assert that the slower learning efficiency at the early stage of training is due to the absence of useful global information, and DRIMA is suitable for tasks in which centralized information is unavailable. But in experiments, global information like all enemies' instant health is still used for reward construction. Would this contradict the distributed setting? Is DRIMA only allowed to receive data from global but not to send messages? If so, why not provide global information for each agent in SMAC experiments to see if the learning efficiency can be improved?

The authors would like to gratefully acknowledge the reviewer for the valuable time during this review. We appreciate the strengths of your comment on our paper, and the following are our responses to your remaining concerns.

To the Weaknesses:

1. The current representative MARL cooperation often has limited generality. Such as COMA (Foerster et al., 2018) takes joint action $\mathbf{a}$ to calculate global-level advantage value $A^{a^i}\left( s, \mathbf{a} \right)$, possessing limited scalability. VDN (Sunehag et al., 2017) and QMIX (Rashid et al., 2020) respectively designs the NN structure which inputs N agents' local action-value function $Q^i\left( o^i, a^i \right) \left( i=1,...,N \right)$ and outputs the total joint value function $Q^{tot}\left( \mathbf{o}, \mathbf{a} \right)$. The two approaches are well applicable for the discrete action space, which can be represented as the one-hot vector and is discriminative enough as the NN inputs, to ensure the effective join action-value functions learning, while not quite fitting in the continuous action space. MFRL (Yang et al., 2018) learned through independent reward that is incapable of dealing with saddle equilibriums.
   Our approach is applicable for both value-based (e.g. DQN) and policy-based (e.g. A2C, DDPG, and PPO) RL algorithms, and for both discrete and continuous space environments. For discrete action space, we adopted MAPPO (Yu et al., 2022) as SOTA; for continuous action space, we selected the widely used benchmark MADDPG (Lowe et al., 2017). To make the experimental results straight and concise, we chose the 'Independent' and 'Centralized' forms of those algorithms as comparisons.
2. Thank you for this suggestion, we have refined Figure 2(a) to help identify various colored surfaces, and improve the related description under it for enhanced comprehensibility, please refer to the highlighted content in the updated PDF.
3. Thank you for the constructive suggestion, we have included the SMAC results in the main paper of the revision. Additionally, more results of battle scenarios have been included to improve the completeness.

To the Questions:

1. DRIMA is based on local information exchange, the scale of this approach does not increase with the number of agents in a system. Therefore, it can be equipped on almost every existing MARL, including MFRL (Yang et al., 2018), for saddle equilibrium avoidance and collaboration enhancement in large-scale environments.
2. The wide range of experiments in our paper shows that conflict-triggered DRI exhibited relatively slower learning efficiency at the early stage in some scenarios, but it did not impair convergence results. In fact, distributed learning algorithms compared with independent and centralized ones inherently possess slower learning rate, since the distributed communication between agents aim to reconstruct the useful global information which is directly available to centralized algorithms and is not considered by independent algorithms. However, independent (or centralized) and fast learning efficiency do not guarantee the overall collaborative effectiveness and robustness of the strategy combinations. This is a trade-off problem and an ongoing challenge in the distributed learning study.

We hope the above response has addressed your main concern about our paper, and we'd be glad to discuss any further questions you may have. The authors would greatly appreciate it if you could consider increasing your rating for the paper after reviewing the above response and the revised PDF.

The authors thank the reviewer for the valuable time and suggestions proposed to improve our work. If all your concerns have been addressed, we kindly ask you to consider raising your rating. If you still have any doubts or reservations about our work, please feel free to give us further feedback, we are more than willing to engage in further discussion with you.

The authors would like to sincerely thank the reviewer for the careful reading and positive comments on our paper. The following are responses to the Weaknesses and Questions.

To the Weaknesses:

1. The "differential reward" of an agent is $r^i-\mu ^i \left( i=1,...,N \right)$, and the "conflict" is the opposite sign between $r^i-\bar{\mu}^{N^i}$ and $\bar{r}^{N^{-i}}-\bar{\mu}^{N^i}$. We refined Section 2.2 and 3.1 with more clear definitions, please check the updated PDF.
2. We described the purpose of Equation 2 in the paragraph above the title of Section 3.1, and the intuition of this approach is derived from the "differential reward" concept in traditional (single-agent) RL, which was described in the last paragraph of Section 3.1. Thank you for this comment, we have refined the related description in Section 3.1 to be more clear in the revision.
3. As the ordinary definition of the differential value function in Section 2.2, the reshaped differential value function by Equation 2 can be obtained by analogy. Its formal definition was given in Lemma 1 and its updated rule was in Proposition 1, which we put in the (Convergence Analysis) Appendix A.2 considering the space limit.
4. The formula in Section 2.2 is correct, where $\mu_{\pi}^{i}$ represents the average reward of agent $i$ before DRI, and $\bar{\mu}_{\pi}$ in Section 3.2 denotes the average of the neighboring average rewards after DRI.
5. The $\beta_t$ should be the Actor step-size, thank you for pointing out this typing mistake.
6. Thank you for this comment, we have refined the related description and Figure 2(a) in the revision, please refer to the highlighted content under Figure 2.

To the Questions:

1. One of the differences between the advantage function ''$Q_t\left( s, a \right) -V_t\left( s \right)$'' and the differential reward ''$R_t\left( s, a \right) -\mu , \mu =\underset{T\rightarrow \infty}{\lim}\frac{1}{T}\sum_{t=0}^{T-1}{\mathbb{E} _{s\sim P, a\sim \pi}\left[ R_t\left( s,a \right) \right]}$'' is the former takes value function to compute, and in deep RL tasks the functions are approximated by NNs. In multi-agent scenarios, interactions using these approximations are subject to certain inaccuracies and inefficiencies, while the differential reward term is more precise. In fact, we had conducted experiments that combined the advantage function and our conflict-triggered idea, the results were in general inferior to the conflict-triggered DRI. Another advantage of DRI is that the average reward term ''$\mu$'' directly reflects the policy's expected reward that relates to the objective of the agent during learning, interactions utilizing it in the learning process help multiple agents keep goal consistency efficiently. Therefore, we believe the advantages of multiple dimensions work together that ensure the superiority of our approach.
2. The idea of our paper was derived from the average reward concept in the traditional (single-agent) reinforcement learning theory (Sutton & Barto, 2018). It is drawn from dynamic programming as one of the value function formulation methods in RL. The recent paper Reward Centering brings this concept back, restudy it with the Laurent-series decomposition theory, and gives it a new description as "Reward Centering". The main content of it is to explicate that the average reward concept plays an essential role in generally improving (single-agent) RL algorithms. Our study was accomplished before Reward Centering was published, and we hold a coincident view with it that "the simplest method can be quite effective", not only in single-agent learning, but we investigate it for cooperative multi-agent learning. (Note that all the compared algorithms in our experiments fairly utilized the average reward term.)

We hope that the above responses have addressed your questions. If you have any further confusion, we'd be glad to discuss them with you.

(continued from the previous response)

5. QMIX designed an extra NN structure, it takes N agents' local action-value function $Q^i\left( o^i, a^i \right) \left( i=1,...,N \right)$ as input and learns their weighted sum as the output that gain the total joint value function $Q^{tot}\left( \mathbf{o}, \mathbf{a} \right)$. This method is applicable for the discrete action space, since actions can be represented as one-hot vectors which are discriminative enough as the NN inputs to learn effective joint action-value functions. For the continuous action space scenario including MPE, QMIX is not well applicable. The paper of QMIX chose discrete action SMAC to conduct experiments, and the paper of MAPPO after it demonstrated that MAPPO overall outperforms QMIX, so we chose MAPPO as our baseline (labeled as PPO-Ctde in the experiments) in SMAC. For the continuous action MPE scenario, we choose the widely adopted MADDPG (labeled as DDPG-Ctde) as a baseline.
   The current cooperative learning approaches often have limited generality, our approach DRIMA is applicable for both value-based (DQN, etc.) and policy-based (A2C, DDPG, PPO, etc.) RL algorithms, and for both discrete and continuous space environments. To make the experimental results straight and concise, we chose the 'Independent' and 'Centralized' forms of those algorithms as comparisons, generally, DRIMA can be equipped on almost every existing MARL for saddle equilibrium avoidance and collaboration enhancement.

We hope the above response has addressed your main concern about our paper, and we'd glad to discuss any further questions you may have. The authors would greatly appreciate it if you could consider increasing your rating for the paper after reviewing the above response and the revised manuscript.

The authors would like to gratefully acknowledge the reviewer for the valuable time during this review. We appreciate the strengths of your comment on our article, and the following are our responses to your remaining concerns.

To the Weaknesses:

1. We revised the definition to make it formal and more direct, and Sections 2.2 and 3.1 are refined to improve the understandability, please check the highlighted content in our updated PDF.
2. We had done lots of experiments in SMAC, and we demonstrated the representative four results considering SMAC scenarios mainly belong to the four types (i.e., homogeneous and symmetrical, homogeneous and asymmetrical, heterogeneous and symmetrical, and heterogeneous and asymmetrical). Our algorithm and compared ones generally perform well in other scenarios. Considering your constructive suggestion, we have added more scenarios to improve the completeness, and we moved SMAC experiments from the Appendix to the main content of the revision.

To the Questions:

1. The formal term is ''differential reward interaction'', it means that multiple agents interacting their differential rewards $r^i-\mu^i \left( i=1,...,N \right)$ during learning. The ''differential reward'' is one of the traditional formulations of the value function in (single-agent) RL (Sutton & Barto, 2018). We have refined this definition, please refer to Sections 2.2 and 3.1.
2. At the beginning of this paragraph, we firstly explained the differential reward term $r\left( s_t, a_t \right)-\mu_t$ for single-agent RL. Since $\mu_t$ indicates the expected average reward $\mu_\pi$ (defined in Section 2.2, under the differential action-value function), and from the definition, one can see that it reflects the overall assessment of its current policy. For single-agent, at training step $t$, the differential reward term $r\left( s_t, a_t \right) -\mu_t$ represents a contribution value to current policy by taking action $a_t$ at state $s_t$, and the sign of this term represents this contribution is positive or negative. Correspondingly, in multi-agent setting, the opposite sign between $\bar{r}^i\left( s_t, \mathbf{a}\_t \right)- \bar{\mu}\_{t}^{N^i}$ and $\bar{r}^{N^{-i}}\left( s_t, \mathbf{a}\_t \right) -\bar{\mu}\_{t}^{N^i}$ represents the conflict contribution of the action decisions relating to the individual $i$ and its other averaged neighbors $N^{-i}$, to the current neighbor-joint policy $\boldsymbol{\pi}_{t}^{N^i}$ (valued by $\bar{\mu}\_{t}^{N^i}$). When the opposite sign occurs at step $t$, agent's personal reward $r^i\left( s_t, \mathbf{a}_t \right)$ was reshaped as neighbor-averaged reward $\bar{r}^{N^i}\left( s_t, \mathbf{a}_t \right)$, to keep joint goal consistency. Reward consensus at critical learning steps can efficiently trade off the joint goal consistency and individual diversity, avoid saddle equilibriums, and ensure robust collaborative learning. Thank you for the comment, we have clarified these descriptions in Section 3.1 of the revision.
3. In SMAC, all the learning agents of the Ally Group receive an Enemy Group reward that sums up all enemies' instant health value, it enables the agent to grasp the overall battle progress. Additionally, each individual receives their own health value as a reward for the 'Independent' and 'Drima' algorithms, while for the 'Ctde' algorithm, agents obtain a total Ally Group reward that summarizes all the members' health.
4. Based on the 3rd reply, each agent receives different reward $r^i$ and composes different $\bar{r}^{N^i}$ respecting their communication range (we set it as view range).