MACCA: Offline Multi-agent Reinforcement Learning with Causal Credit Assignment

Question 1: The proposed reward decomposition mechanism is not specific to offline MARL. The results will be more convincing if MACCA achieves performance gain in the online MARL (SMAC tasks).

Answer Q1: We appreciate your interest in the applicability of our MACCA framework to online MARL settings. Our decision to focus on offline credit assignment was driven by several key factors. Firstly, offline spatial credit assignment poses a more significant challenge compared to online settings due to restricted exploration opportunities and the inherent constraints of data distribution. When relying only on team rewards, bias in offline data or significant performance differences between agents can seriously affect the accuracy of the evaluation. In contrast to offline settings, online credit assignment algorithms like COMA or SQDDPG can iteratively refine their evaluations through continuous interaction with the environment. Therefore, the challenge that MACCA focuses on in offline environments is accurate agents' contribution based on offline data, which is crucial to understand the role of each agent and optimize overall team performance in scenarios with restricted exploration.

Secondly, the learning of causal models, which is a central component of MACCA, benefits from the more balanced data distribution typically found in offline reinforcement learning. In these settings, datasets often derive from policies with similar performance levels, resulting in a relatively uniform data distribution. This contrasts with online settings, particularly on-policy methods, where there's no inherent experience replay buffer to facilitate causal learning. Off-policy methods may offer some advantages, but they also come with the issue of potential shifts in the distribution within the experience replay buffer. Furthermore, while MACCA could theoretically be adapted for online off-policy settings—possibly through pre-training or the establishment of an independent causal learning buffer—such adaptations are beyond the current scope of our paper. Our focus on the offline setting is deliberate, as it naturally complements the learning of our causal model.

Furthermore, MACCA has demonstrated superior performance across three distinct environments. We believe that these results affirm the efficacy of MACCA in the offline setting and hope this addresses any concerns regarding our choice of focus and the potential of MACCA in different application scenarios.

We appreciate your review and provide point-by-point responses to your questions below.

Weakness 1: The target of credit assignment is not the immediate reward, but the cumulative reward of the whole trajectory, which is affected by the immediate reward and the transition (long-horizon rewards). However, this paper only considers the decomposition of immediate reward, ignoring the long-horizon causal relationship. For example, in the delayed reward setting (SMAC), the agents only receive the reward at the last timestep, the credit assignment of the whole trajectory in MACCA is only related to the state and actions of the last timestep, which is unreasonable.

Answer W1: Thank you for the question. In our understanding, the setting you mentioned is generally in the single agent setting, which usually refers to temporal credit assignment, like having a delayed or episodic reward and assigning to each time step. However, our work in this paper primarily addresses spatial credit assignment within a multi-agent context, which involves determining the distribution of team rewards among individual agents, a process central to accurately assessing each agent's contribution.

Furthermore, MACCA's primary objective is to learn an individual reward function to generate the $r_t^i$, distinct from directly apportioning the team reward $R_t$, like using the Shapley value [1][2]. Under our understanding, the reward itself, whether the team reward or individual reward, should not consider the long-term (long-horizon) effect, whereas, under the CTDE paradigm, the individual rewards derived are then utilized by the critic to calculate the corresponding $q_t^i$, ensuring that long-term effects are taken into account.

Regarding the sparse reward setting in SMAC, it indeed poses a more complex challenge, combining both spatial and temporal credit assignment aspects. While this forms an intriguing direction for future research, it falls beyond the scope of our current paper, which concentrates on spatial credit assignment in offline multi-agent environments.

• [1] Wang, Jianhong, et al. "Shapley Q-value: A local reward approach to solve global reward games." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 34. No. 05. 2020.
• [2] Wang, Jianhong, et al. "Shaq: Incorporating shapley value theory into multi-agent q-learning." Advances in Neural Information Processing Systems 35 (2022): 5941-5954.

Weakness 2: The states of the environments adopted by this paper are low-dimension vectors. It is hard to capture the causal relationship between image state and reward.

Answer W2: Thanks for the question. This work focuses mainly on state-based tasks. 1) For the setting where states are not defined, such as image-like input, our work can be applied by employed in the latent state space, which requires learning a latent vector representation. 2) For a large number of dimensions of state, this would increase the complexity of the causal modelling and thus require more efficient causal discovery methods. 3) As an advantage, by learning the causal structure, we can constrain the optimization of the models over a small subspace of state and action, resulting in a lower requirement of parameters of the neural network. 4) considering that we wanted a fairer comparison with other baselins under this setting, we followed the same data format and level of dimension of states with [1][2][3][4]. Overall, the main focus of this work is to address the team rewards credit assignment problem under the offline setting. Therefore, we consider the standard and general state setting.

• [1] Samvelyan, Mikayel, et al. "The starcraft multi-agent challenge." arXiv (2019).
• [2] Fu, Justin, et al. "D4rl: Datasets for deep data-driven reinforcement learning." arXiv (2020).
• [3] Pan, Ling, et al. "Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification." ICML. PMLR, 2022.
• [4] Yang, Yiqin, et al. "Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 10299-10312.

Thank you for your reply. Before answering your question, we would like to clarify the composition of the reward function in the offline dataset we use.

Team Reward Format in Offline Dataset: We would like to describe the offline datasets more, since it seems there might be a misunderstanding about the nature of the offline RL setting and the dataset we are using.

• SMAC environment reward (team reward): the team reward is generated based on the hit-point damage inflicted on enemy units and the number of enemy units eliminated at each timestep. To illustrate, consider the scenario in the 5m_vs_6m map within the Medium dataset. From timestep 0 to 10, the team rewards are as follows: (0, 0, 0, 0.9057, 0.2264,0.6792, 0.7170, 0.679,0.2264, 0.4528). Here the team reward is 0 at timesteps zero to three, 0.9057 at timestep four, 0.2264 at timestep five, 0.6792 at timestep six, 0.7170 at timestep seven, 0.679 at timestep eight, 0.2264 at timestep nine, and 0.4528 at timestep ten. This demonstrates that team rewards are evaluated based on the team's performance against the opponent at each timestep, rather than cumulatively.

• MA-MuJoCo enviornment reward (team reward): the reward function is formulated as $\frac{\Delta x}{\Delta t}+0.1 \alpha$. Here, $\Delta x$ denotes the change in the overall position or distance, $\Delta t$ signifies the change in time and $\alpha$ represents an action regularization term.

• MPE environment reward (team reward): For example, in Cooperative Navigation (CN), all agents are globally rewarded based on how far the closest agent is to each landmark (sum of the minimum distances) for each timestep. The team reward are penalized if some agents collide with other agents (-1 for each collision).

Q1: Apparently, policy learning is guided by return, but the return is the sum of decomposed individual rewards, which is different from the decomposed global return. This is what I mean the difference between your solution and the long-horizon solution I proposed.

Response: We are not arguing against your solution: your approach is indeed a valid one. However, it might require a reconstruction of the environmental dataset rewards; therefore, it is not suitable for our offline setting. Since you raised concerns about our method not considering return and long-horizon effects, we want to clarify that our solution does take return into account: the policy learning will learn a policy that maximizes the long-term return. In conclusion, our approach considers the setting of currently available offline benchmark datasets, where each time step represents an instantaneous team reward rather than an accumulated reward. Then we have adopted the proposed approach.

Q2: Analysis of toy case & Q3: In the training process, the agents will chose random actions for exploration. So this situation will occur in the training process and the agents will receive a wrong reward assignment

Response: Thank you for providing this example. According to the Team Reward Format we provide, unlike the scenario in this toy example, the team reward at each time step reflects the collective performance of the team at the current timestep rather than being cumulative. In your example:

• At timestep=1, Agent 1 attacks, env reward (i.e., team reward from the dataset) = 1. Agent 1 would receive a higher reward, let's say 0.99, and Agent 2 would receive 0.01.
• At timestep=2, Agent 1 attacks, env reward=1. Again, Agent 1 would receive 0.99, and Agent 2 would receive 0.01.
• At timestep=3, Agent 1 attacks, env reward=1. Once more, Agent 1 would receive 0.99, and Agent 2 would receive 0.01.
• At timestep=4, Agent 2 attacks, env reward=1. Now, Agent 2 would receive a higher reward, let's say 0.99, and Agent 1 would receive 0.01.

It's important to note that at timestep=4, the env reward is not 4 but 1, as provided directly by the environmental data. Our algorithm establishes a relationship between team reward and individual agent contributions, allowing it to better infer that Agent 1 should receive more rewards overall. Thank you for presenting this example.

Q4: It should work in both offline and online settings. Performing online experiments is the only way to demonstrate the effectiveness in online setting.

Response: Once again, we agree with your suggestion to try it online. However, what we would like to emphasize is that this paper, as indicated by our title, is focused on an offline setting. We opted for an offline setting because, as mentioned earlier, challenges in environments like the online SMAC environment have largely been addressed. Furthermore, tackling offline scenarios presents greater difficulty and room for improvement, which is why we conducted our experiments in an offline setting. If there is a desire to explore online scenarios, that could potentially be the subject of a separate paper. Nevertheless, given tha

We thank the reviewer for your positive support. Below, we provide a point-wise response to your questions.

Weakness 1: The presumed data-generating process is restrictive.

Answer W1:

Thank you for the question. We understand your concern regarding the expressive capability of our linear model in MACCA. Firstly, it is essential to emphasize that MACCA is adaptable for some specific nonlinear team reward functions. For instance, in cases where the team reward $R$ is defined as either the maximum or minimum of individual rewards $r_t^i$ in the team, the identifiability of individual rewards under this setting is retained, and our theoretical proofs remain valid. Secondly, our linear configuration does not directly impact the training of the network. Although our model assumes the team reward $R$ as the sum of individual rewards, the primary output of the $\psi_r$ is the estimated individual rewards $\hat{r}_t^i$, not directly generating the estimated team reward $\hat{R}_t$. Finally, team reward can be modelled as linear in many scenarios, particularly in cooperative games. A pertinent example is urban traffic management, where multiple agents control traffic signals at various intersections. Here, the traffic flow at each intersection and the overall throughput of the system exhibit a linear relationship. By aggregating the contributions from each intersection, the effectiveness of the entire traffic management system can be effectively quantified. Therefore, the linear model is sufficient so far, and we are considering discussing more complex settings and providing corresponding solutions in future work.

Questions 1: Have the authors ever considered extending the algorithm to scenarios where the team reward cannot be decomposed as the sum of individual rewards? What can be the possible solution?

Answer Q1: Thank you for the question. Yes, we will consider extending to such non-sum scenarios in future work. The possible direction is to use some nonlinear integration function to generate the team reward instead of the sum function.

We thank the reviewer for the constructive comments and provide our point-wise response below.

Weakness 1: The overall writing quality needs to be improved. There are some grammatical errors, and the details will follow.

Answer W1: Thanks for pointing it out. We will revise our paper carefully in the revision.

Weakness 2: Although the evaluation of MACCA has been conducted in multiple benchmarks to demonstrate its generalizability, it is questionable whether the method has top performance in all of the environments that belong to a specific benchmark. For example, the SMAC benchmark has more than 20 original battle scenarios, including a few Super Hard challenges. However, only three of the maps, including a single Super Hard challenge, are shown in the results.

Answer W2: Thank you for your concern. Our paper mentions that the primary reason for choosing the SMAC benchmark is to demonstrate performance in larger-scale scenarios (Section 5.2). Accordingly, we focused on maps that are challenging and involve more than five agents. However, considering your concern and acknowledging that the extensive experimentation may further substantiate MACCA's SOTA performance, we will include additional Super Hard map in our next revision.

Weakness 3: Regarding the MACCA architecture, clearly, the policy learning part is crucial for the model. However, this part is mostly taken from previously established methods such as I-CQL, OMAR and MA-ICQ, especially the term $J_\pi$. Also, after looking at the experiments in different environments, it is not clear whether MACCA-CQL, MACCA-OMAR, or MACCA-ICQ has the SOTA performance overall.

Answer W3: Thank you for highlighting this aspect of the MACCA architecture. Firstly, it's important to note that MACCA is designed to be a plug-and-play method, capable of being seamlessly integrated with other offline MARL frameworks. This adaptability is, in fact, one of MACCA's strengths, as it allows for versatile applications across a range of existing methods.

Secondly, regarding the performance of MACCA in different environments, we provide detailed data in the paper to prove its efficacy. Here, we summarize the results in the table above, including the average percentage improvement of all MACCA-based methods in these three environments. According to the results presented in our paper, MACCA-based algorithms consistently meet or exceed SOTA performance in all test environments and maps. This is demonstrated by the bold entries in Tables 1 and 2, which highlight the best performance in the experiments.

Weakness 4: The code for the proposed method is not included in the submission. The disclosed information about the hyper-parameters and environmental settings is limited.

Answer W4: We appreciate your feedback. The description of hyperparameters is in Appendix D, including the model structure and specific hyperparameters. The environment settings are in Appendix C, including the configuration and data set generation methods of three different environments. Also, to enhance clarity and reproducibility, we have uploaded the code for our proposed method to an anonymous repository, accessible at: https://anonymous.4open.science/r/MACCA_ICLR/.

Question 1 In Section 4.1, are the masks in the expression for the reward learned or manually set?

Answer Q1: The causal masks are learnable and predicted by networks, enjoying the advantage of addressing complex dynamic settings in MARL, described in equation 3, Section 4.2.

Question 2 There is a long and confusing sentence in Section 4.2: "Its primary objective is to ... reduce the number of features that a given depends on, ... mitigates the risk of overfitting." What does "reduce the number of features that a given depends on" mean?

Answer Q2: We will clarify this in the future version: the $L_{\mathrm{reg}}$ can reduce the difficulty of estimation of the reward function since it can clear the redundant features during training by reducing the sparsity of the causal structure $\mathcal{S}_{s r}$ and $\mathcal{S}_{a r}$ (defined in Section 5.3), in order to mitigate the risk of overfitting.

Question 3 Is the hyper-parameter $h$ learned?

Answer Q3: No, the design specifics of hyperparameter $h$ is elaborated in Section 4.3 Individual Rewards Assignment, and the value of $h$ is also elaborately in Appendix D.3 table A2, we set $h$ in all environments to 0.1.

Question 4 Can you mention how many independent tasks do the Multi-agent Particle Environment and the Multi-agent MuJoCo have, respectively?

Answer Q4: Yes, as described in Appendix C, there are three scenarios (PP, CN and World) in the Multi-agent Particle Environment and one scenario (Half-Cheetah) in Multi-agent MuJoCo.

Question 5 In Section 5.3, you mentioned that "It is important to note that our method is not highly sensitive to the hyperparameters despite using them to control the learned causal structure." Can you justify this argument?

Answer Q5: Thank you for the question. To investigate the impact of the hyperparameters, we refer to the results presented in Section 5.3 Table 4. This ablation study discusses how different values of $\lambda_1$ affect the sparsity of the learned causal structure. The results indicate that while varying $\lambda_1$ leads to changes in the sparsity of the causal structure, the impact on the convergence of policy learning is relatively minimal.

Additionally, we add an additional ablation study on $\lambda_2$ as shown in the above table, which presents the mean and the standard variance of the average normalized score with diverse $\lambda_2$ in the MPE-CN task. The value in brackets is the sparsity rate $\mathcal{S}_ {ar}$ of $\hat{\boldsymbol{c}}_ t^{i, a \rightarrow r}$, whose definition can be found in Section 5.3. For all values of $\lambda_2$, the sparsity rate $\mathcal{S}_ {a r}$ consistently begins from zero. Over time, there is a discernible increase in $\mathcal{S}_{a r}$, and the convergence rate slows down with the increase of $\lambda_2$. And, the convergence rate to an elevated sparsity rate decelerates with the increase of $\lambda_2$. This pattern intimates that higher $\lambda_2$ values engender a more measured modulation in the causal impact exerted by actions on individual rewards. More importantly, despite the variation in $\lambda_2$ values, the average normalized scores converged towards a similar range.

Question 6 You showed the visualizations of two causal structures in MPE. Have you done similar work in MA-Mujoco and SMAC?

Answer Q6: Thank you for the question. The primary reason we focused on visualizations in MPE is due to its comparatively smaller state and action spaces relative to MA-Mujoco and SMAC. This smaller scale facilitates a more straightforward presentation and more accessible human analysis. Taking into account your suggestions regarding extending these efforts to MA-Mujoco and SMAC, we recognize its value and will update it in future revision.

Question 7 In the sub-section "Visualization of Causal Structure." in 5.3, what do "S2R" and "A2R" stand for?

Answer Q7: Thanks for pointing out these typos, "S2R" refers to Figure 2.a, which is "Causal Structure: States to Individual Rewards", and "A2R" refers to Figure 2.b, which is "Causal Structure: Individual Reward Actions".

Thank you very much for your positive feedback. We sincerely appreciate your positive support and constructive comments. Below, we provide a point-wise response to your questions.

Weaknesses 1 and Question 1: While MACCA establishes causal relationships among states, actions, and individual rewards in offline datasets, this "causal relationship" doesn't seem to have a strong correlation with offline multi-agent reinforcement learning. It appears that this "causal relationship" might also be applicable in online reinforcement learning，rather than being specifically designed for offline multi-agent reinforcement learning.

Response to W1&Q1: We appreciate your interest in the potential application of our MACCA framework to online MARL settings. Our choice to focus on the offline credit assignment setting was driven by several primary considerations.

Firstly, the restricted exploration in offline settings poses unique challenges to this credit assignment problem. When relying only on team rewards, bias in offline data or significant performance differences between agents can seriously affect the accuracy of the evaluation. In contrast to offline settings, online credit assignment algorithms like COMA or SQDDPG can iteratively refine their evaluations through continuous interaction with the environment. Therefore, the challenge that MACCA focuses on in offline environments is accurate agents' contribution based on offline data, which is crucial to understanding the role of each agent and optimising overall team performance in scenarios with restricted exploration.

Secondly, an essential aspect of MACCA is the training of causal models, which significantly benefits from the stable and consistent data distribution typically found in offline reinforcement learning environments. In offline settings, data often comes from policies with comparable performance levels, leading to a more uniform distribution. This stability is advantageous for causal learning, as it reduces the variability and potential distribution shifts that can occur in online settings, particularly in online and on-policy scenarios where an experience replay buffer is not utilized. Off-policy methods in online settings may offer some mitigation against this issue, but they are not without their challenges, such as potential shifts in the experience replay buffer's distribution. Given these considerations, our focus on offline settings was a deliberate choice, aligning well with the objectives and strengths of the MACCA framework.

Furthermore, MACCA has demonstrated superior performance across three distinct environments. We believe that these results affirm the efficacy of MACCA in the offline setting and hope this addresses any concerns regarding our choice of focus and the potential of MACCA in different application scenarios.

Weaknesses 2: The design specifics of $c^{i, s \rightarrow r}$ and $c^{i, a \rightarrow r}$ are not elaborated upon, and the particular networks used are not clearly mentioned.

Answer w2: Apologies for any confusion. In fact, we have introduced the design specifics of $c^{i, s \rightarrow r}$ and $c^{i, a \rightarrow r}$ in Section 4.2 and describe the networks used in Appendix D.2. The masks $\boldsymbol{c}^{i, \boldsymbol{s} \rightarrow r} \in\{0,1\}^{D_s}$ and $\boldsymbol{c}^{i, \boldsymbol{a} \rightarrow r} \in\{0,1\}^{D_a}$ in equation 2 denote as the binary masks in which control if a specific dimension of the state $s$ and action $a$ impact the individual reward $r_t^i$, separately. To estimate them, we use $\psi_g^{s \rightarrow r}$ and $\psi_g^{a \rightarrow r}$ in equation 3, the network we used share the same architecture but the parameters. It includes three fully-connected layers with a hidden size of 256, followed by an output layer with a scalar output. Each hidden layer is activated using the rectified linear unit (ReLU) activation function.

Weaknesses 3 and Question 2: There isn't much explanation regarding the setting of the hyperparameter $h$, nor is there any mention of whether the value of $h$ remains consistent across different offline environments.

Answer W3&Q2: Thank you for your query. The design specifics of hyperparameter $h$ is elaborated in Section 4.3 Individual Rewards Assignment, and the value of $h$ is also elaborately in Appendix D.3 Table A2.

The hyperparameter $h$ is a threshold to determine the existence of the causal edges after we get the scalar output of $\psi_g^{s \rightarrow r}$ or $\psi_g^{a \rightarrow r}$ in equation 3. During the inference, i.e., predicting the individual rewards for policy optimization, the binary masks to capture the causal structure are obtained by $\boldsymbol{c} =I(ReLU(o)>h)$ where $o$ is the output of the last layer of $\psi_g^{s \rightarrow r}$ or $\psi_g^{a \rightarrow r}$, and $I$ is indicator function. The value of the $h$ is invariant across different experiments, which is $0.1$, as shown in Table A2.

Weaknesses 4: Concrete code has not been provided.

Answer w4: We appreciate your feedback and have uploaded the code to anonymous git: https://anonymous.4open.science/r/MACCA_ICLR/ to improve clarity and support reproducibility.

Question Q3:In the ablation study section, the impact of $\lambda_1$ on the causal structure, specifically its influence on the state's effect on individual rewards, was explored. However, has the impact of $\lambda_2$ on the causal structure been considered? I didn't see this part being researched in the paper.

Answer 3: Thank you for your question. We have conducted additional experiments on $\lambda_2$ and show the results in the table below.

This table shows the mean and the standard variance of the average normalized score with diverse $\lambda_2$ in the MPE-CN task. The value in brackets is the sparsity rate $\mathcal{S}_ {ar}$ of $\hat{\boldsymbol{c}}_ t^{i, a \rightarrow r}$, whose definition can be found in Section 5.3. For all values of $\lambda_2$, the sparsity rate $\mathcal{S}_ {a r}$ consistently begins from zero. Over time, there is a discernible increase in $\mathcal{S}_{a r}$, and the convergence speed slows down with the increase of $\lambda_2$. This pattern intimates that higher $\lambda_2$ values engender a more measured modulation in the causal impact exerted by actions on individual rewards. Furthermore, despite the variation in $\lambda_2$ values, the average normalized scores across different $\lambda_2$ settings eventually converge towards a similar level.

Question 4: In the related work section on multi-agent credit assignment, it seems that recent approaches in value decomposition, such as wqmix, qplex, resq, etc., have not been considered. These methods tackle the credit assignment challenge among agents using value decomposition techniques and, from what I understand, they have demonstrated commendable performance. Have you considered applying the methods from wqmix, qplex, resq, etc., to offline multi-agent environments?

Answer Q4: Thank you for the question. As reflected in Table 3, we have conducted a comprehensive comparison with SOTA online off-policy credit assignment baselines (including QMIX, SHAQ, and SQDDPG) and their combination with offline learning algorithms to maintain a fair and unbiased comparison. We noticed that the plethora of QMIX variants that employ diverse mixing networks for value decomposition, their application in offline settings is indeed worth considering for a more exhaustive comparison. Nevertheless, our comparative analysis primarily concentrates on explicit credit assignment methods. Upon a thorough evaluation of both the performance and robustness of these implicit credit assignment methods in online scenarios, we opted to utilize QMIX as a representative model for all value decomposition approaches.