MADiff: Offline Multi-agent Learning with Diffusion Models

Dear reviewer, thank you very much for your constructive comments! We try to address your concerns as follows.

Q1: "Given that these tasks were essentially solved by the community with in the easy maps that are compared with in the results, the results may not be as convincing as currently indicated."

A1: Thanks for your comment, but you might misunderstand our experimental settings. Our method and all baselines, including QMIX, are trained in a fully offline manner. Offline Policy learning is far more challenging than online learning due to limited data support and sub-optimal behavior policies used to collect the dataset.

One task can be solved by online RL methods does not mean it can be solved at the expert level with offline learning. For example, PPO [1], which was proposed in 2017, can easily achieve expert-level performances in single-agent mujoco environments (Hopper, HalfCheetah, ...). However, the widely-used D4RL benchmark [2], which was released in 2020, still uses those environments to build their dataset. And the SOTA algorithms today can only obtain about half of expert scores on D4RL's halfCheetah-medium dataset [3].

As a result, our results under offline MARL settings should not be compared with online MARL methods.

Q2: "Regarding the other experimental design items, it is unclear what hypothesis the MPE environments test that would not be tested by SMAC."

A2: Both MPE and SMAC datasets are widely adopted by existing works in offline MARL [4, 5]. These two classes of environments have distinct state/action spaces and require different cooperation patterns. We include both of them to ensure a thorough evaluation of our method. Two intuitive differences are that the action space of MPE is continuous, and the initial state distributions are more diverse than SMAC.

Q3: "It is unclear if the test games have a distribution of trajectories that are out of distribution of the training/validation set."

A3: Thanks for your good question! We adopt the standard training/validation/test split in MATP experiments, which allows for out-of-distribution (OOD) tests. On offline MARL tasks, our evaluation procedure follows the existing works on offline RL [6, 7] and MARL [4, 5] and does not deliberately test the generalization ability.

We want to highlight that different from static supervised learning tasks in MATP, evaluating offline-learned policies by interacting online naturally includes the requirement of OOD generalizations. Due to errors induced by functional approximations and stochasticity in environmental dynamics, agents will encounter OOD states even if they are started from in-distribution states. The learned policies must have the ability to recover from OOD states and prevent errors from accumulating over time.

Designing reasonable benchmarks for specifically evaluating the generalization ability of offline MARL models is less discussed yet, and we acknowledge that it is an important future research direction.

Q4: "How does the centralized control version scale with the number of agents? It is unclear if this is a useful avenue of exploration compared with the CTDE paradigm."

A4: Both versions of MADiff have their own values. MADiff-C is only applicable in cases where a centralized controller for all agents is allowed, such as traffic light control or fleet control. Since all agents' actions are made jointly, it is easier to keep their behaviors coordinated. Therefore, MADiff-C is supposed to perform better compared to MADiff-D in those cases, which is aligned with our experiment results. However, there are more scenarios where centralized controls are impossible, e.g., autonomous driving and distributed sensor networks [8]. MADiff-D, which is the CTDE version of MADiff, is designed for these scenarios.

MADiff-C and MADiff-D scale the same as the number of agents increases. We use a shared U-Net model for all agents, where different agents' data can be batched together and passed through the network. It does not cost much more inference time as the number of agents increases with GPU-accelerated computing. For self-attention among agents' embeddings, it can also scale with linear complexity [9].

When the number of agents is extremely large, we can modify the opponent/teammate modeling to operate in a projected low-dimensional space with a latent diffusion model [10]. We leave this direction to future work.

We are delighted to share with you some new ablation results!

As mentioned in A3, to test whether both diffusion modeling and attention networks are important in the success of MADiff, we compare our network design to ConcatDiff, which is an ablation variant that does not use attention networks but directly concatenates all agents' observations as inputs to a U-Net. ConcatDiff-C is used for centralized control, and ConcatDiff-D masks other agents' observations before concatenation during training. Results are presented as follows:

We can observe that ConcatDiff performs worse than MADiff in both CTCE and CTDE cases, and underperforms OMAR, which is the best baseline method in MPE datasets.

We sincerely hope that our responses and the additional experiments we conducted have addressed your concerns and enhanced the quality of the paper. We would greatly appreciate it if you could clarify any remaining concerns you may have.

Dear reviewer, we appreciate your time and effort in reviewing this paper. We try to address your concerns as follows and hope we can ease your concerns.

Q1: "The overall structure of the paper and algorithm is very similar to Decision Diffuser, but with an additional attention layer."

A1: It is indeed that many successful design choices of [1] have inspired us and guided us to derive our method. Similarly, much of [1] also referred to the design of diffuser [2], but that doesn't take away from the fact that it's an excellent research.

Compared with [1], MADiff, which is the first to apply diffusion models in offline MARL, is naturally designed for multi-agent problems, taking multi-agent coordination, opponent modeling, and trajectory planning in a unified framework. Our contributions include but beyond the model design, and the most important one is that the proposed framework allows the use of the same model structure to handle various tasks (CTCE, CTDE, MATP) by flexible conditioning during evaluation and achieves superior results on those tasks.

Q2: "The benefits of using diffusion policies mentioned in Decision Diffuser, ... are not addressed in MADiff in the context of MARL."

A2: Compared to single-agent learning, multi-agent learning problems themselves are more challenging and can benefit from diffusion modeling. Specifically, besides the variety of a single agent's behavior, the interplay between multiple agents exhibits a multi-modal and complex nature. Our goal is to introduce DM with powerful modeling ability of arbitrary distributions to MAL for better learning multi-agent interactions in offline datasets. Verifying whether skill composition in SAL still applies in MAL settings is an interesting trial, but it is out of the scope of our paper.

Q3: "The inverse dynamics loss in Eq. 7 looks a little strange since the observations s^i, s^{i'} are used, ... but may not translate to more complex tasks where the other agents’ actions impact the dynamics. Also, the notation s^i is confusing here since Section 2 uses o^i."

A3: Thanks for your careful checking. There is a typo in Eq.7, where $s^i$ and $s^{i'}$ should be replaced by $o^i$ and $o^{i'}$. We have made corrections in the revised manuscript.

We would like to justify our use of IDM $I(o^i, o^{i'})$ in multi-agent tasks in two aspects:

• First, since the inverse dynamics model is used rather than a forward dynamics model, we are not assuming "the next state observation only conditions on the current local observations and local actions", but that the current and next local observations determine the local actions. Since the next local observation is jointly generated with predictions of other agents' observations, it already considers the impact of other agents' actions.

• More importantly, the goal in Dec-POMDP is learning a policy for each agent, which can maximize cumulative rewards when all agents are controlled by learned policies [3]. Therefore, although actions are made in a decentralized way, each agent's policy is learned from the same dataset the IDM is trained on. Thus, $p(a^i|o^i, o^{i'})$ during evaluation should be similar to IDM's training distribution. Learning policies that can coordinate with any teammates requires zero-shot coordination [4], and is (to our knowledge) not yet discussed in offline settings and clearly out of the scope of this paper.

Q4: "Only a small number of agents (3-5) are considered in the experiments."

A4: Most existing works in offline MARL do not consider tasks with a large number of agents. For example, MADT-KD [5] and OMAR [6] are tested on environments with up to 8 agents. Moreover, OMAR uses eight multi-agent environments for benchmarking, yet only two of these consist of more than 5 agents. Despite the fact that we consider at most 5 agents in MARL experiments, our MATP experiments are conducted on NBA datasets with 10 agents, and MADiff demonstrates superior performance than the baseline method.

Q5: "While opponent modeling is mentioned as one of the contributions, previous works in opponent modeling are missing from the Related Work section."

A5: Thanks for your suggestions! We have included a discussion of previous opponent modeling literature in the related work section.

Dear Reviewer QCKf,

We deeply appreciate the valuable and constructive feedback you provided. As the rebuttal discussion between reviewers and authors is coming to a close on November 23rd, we would greatly appreciate it if you could clarify any remaining concerns you may have. This will ensure that we can adequately address them during this period.

Thank you for your attention to this matter. We are looking forward to your response.

Best regards,

The Authors

Dear reviewer, thanks for your sincere review and advice! Hereby we try to answer your questions.

Q1: "whether MADIFF can be applied to domains with high-dimensional inputs (e.g., image)"

A1: Thanks for your valuable suggestions! MADiff can be applied in environments with high-dimensional, e.g., image observations with more complicated encoder and decoder networks such as CNN or vision transformers. There are some successful cases of applying diffusion planning to single-agent and image-based tasks [1, 2], which can be incorporated into the MADiff framework. We leave this to future work and add a brief discussion to the conclusion of this paper.

Q2: "In Equation 9, inferring all other agents' observations could be very difficult when N is large. Could MADIFF scalability benefit by inferring only the agent i's neighbors instead of all N agents?"

A2: That is a good point, and your suggestion is valuable in boosting the scalability of MADiff. Selective opponent modeling is necessary in scaling to environments with very large N, and only inferring agent i's neighbors can reduce computation and modeling costs. However, neighbors defined under simple Euclidean distance may not be reasonable in some environments, while a learning-based distance metric would be more desirable. Another possible choice is to map the trajectories of all other agents to a low-dimensional embedding space and perform latent diffusion on it. Those modifications required are non-trivial, and we leave this direction as part of future work.

Q3: "Can MADIFF be applied to general-sum settings or would it be limited to cooperative settings only?"

A3: The MADiff framework can only be applied to cooperative tasks, which is the same as most prior works in offline MARL [3, 4]. Naive offline RL or generative modeling methods are likely to fail on non-cooperative tasks, since the optimal policy learned from static datasets can be exploited by other agents during evaluations if they change their policies to be different than behavior policies in datasets. Effective offline MARL in multi-player general-sum settings requires offline equilibrium finding (OEF), which is particularly challenging due to limited data support. Very little work has attempted to solve OEF and is limited to simple tasks with discrete state and action spaces [5].

References

[1] Chi, Cheng, et al. "Diffusion policy: Visuomotor policy learning via action diffusion." arXiv preprint arXiv:2303.04137 (2023).

[2] Du, Yilun, et al. "Learning universal policies via text-guided video generation." Thirty-seventh Conference on Neural Information Processing Systems. 2023.

[3] Pan, Ling, et al. "Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification." International Conference on Machine Learning. PMLR, 2022.

[4] Yang, Yiqin, et al. "Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning." Advances in Neural Information Processing Systems 34 (2021): 10299-10312.

[5] Li, Shuxin, et al. "Offline equilibrium finding." arXiv preprint arXiv:2207.05285 (2022).

Dear reviewer, thanks for your time! We have conducted additional experiments on Multi-Agent Mujoco datasets and hereby answer your questions below. Hope we can ease your concerns.

Q1: "What does it mean for actions to be 'high-frequency'?"

A1: We are sorry for the confusion. The term "frequency" refers to using the Fourier transform to transform the state or action sequences in the time domain to the frequency domain. Action sequences are less smooth across time steps and thus have a lot of high-frequency components after transformation.

To make it clearer, we have removed the statement about action frequency.

Q2: About "Returns" and "Other information" in Figure 1.

A2: We are sorry for the confusion. During training, "Returns" in Fig. 1 is set to the cumulative discounted reward starting from time step $t$. In some environments, agents can have different returns at some time step, e.g., penalties on the specific agent that collides with others in MPE, thus we use the plural form to denote returns for each agent. During evaluation, we set returns to the same high-enough value for all agents. Before input to the U-Net residual blocks, Returns are passed through an MLP and concatenated with the diffusion time step embedding.

"Other information" is task-related condition variables other than returns. For example, in the NBA dataset, we need player embeddings and ball positions as inputs. "Other information" is passed through a separate embedding network and concatenated with return embeddings. We have revised the caption of Fig. 1 to include descriptions of these two terms.

Q3: "it is unclear how the attention mechanism incorporates the information of other agents, and how this is supposed to help coordination"

A3: The attention mechanism helps coordination in both centralized control and decentralized execution settings. When centralized control is allowed, the attention module enables the diffusion model to jointly plan trajectories for all agents, which is essential in tasks that require coordination.

In a decentralized execution setting, although different agents make decisions independently, they can use the same attention network to jointly infer other agents' future trajectories and plan their own trajectories. Since the attention network is trained in a centralized manner, the generated trajectories for all agents should be consistent. Intuitively, the consistency constraint requires the ego agent to first think at a higher level, i.e., make a coordinated plan for all agents, and then place itself in that plan. The coordination among agents can be improved by performing "imagined" planning for others (i.e., opponent modeling).

Q4: "the term "opponent" or "opponent modelling" is used repeatedly, despite the work only dealing with cooperative settings."

A4: Thank you for pointing this out. We use the term "opponent" to refer to other teammates in cooperative games, which is also adopted in some prior works [1, 2]. Since most methods in modeling adversary agents in competitive games can be used in modeling teammates, using the same term "opponent" can ensure better coverage of related works.

Q5: "Section 5.4 as a whole is unclear to me, including exactly what the Valid Ratio is measuring."

A5: We are sorry for the confusion and have rewritten Section 5.4 to make it more clear.

Q6: "In SMAC, it is common to report the win rate ... would like the authors to provide the win rates".

A6: Most prior offline MARL works use self-collected datasets, which makes it hard to measure the progress in this field. Instead, we use the off-the-grid MARL dataset [3], which is open-sourced and released with comprehensive baseline results. For SMAC tasks, they did not report the win rates but only the scores of baseline algorithms. As a result, we do not compare the win rates of MADiff in our experiments. We want to highlight that there are some offline MARL works [4, 5] that also use the average score as the only metric for SMAC.

To make the readers better understand the magnitude of scores in SMAC, we have added Fig. 5 in the supplementary material to demonstrate the distribution of scores in each SMAC dataset we used.

Q7: Issues concerning Figure 3 and Table 1.

A7: Thank you for your careful check! Results are computed with 3 different seeds. In the revised paper, we mention the number of seeds in the captions of Fig. 3 and Tab. 1. The error bar is added to Fig. 3.

Q8: "The results for SMAC seem underwhelming, especially since SMAC lacks relevant stochasticity and partial observability."

A8: To our knowledge, each agent in SMACv1 can only receive local observations within a fixed sight range [6], which makes the environment partially observable. Although SMACv1 lacks relevant stochasticity, we highlight that both MA-ICQ and MA-CQL are strong baselines in offline MARL, and MADiff achieves competitive results.

Q10: Multi-agent Mujoco

A10: We are delighted that you acknowledge our added experiments. The naming of different Multi-agent Mujoco tasks follows the off-the-grid datasets (https://github.com/instadeepai/og-marl), where 2halfcheetah and 4ant refer to the 2-agent halfcheetah and 4-agent. The environment setting is the same as the default configurations from the repo of the Multi-agent Mujoco environment (https://github.com/schroederdewitt/multiagent_mujoco). As we mentioned in A2, no other information is used in Multi-agent Mujoco experiments. Results on Multi-agent Mujoco are added to the revised paper, accompanying necessary descriptions.

References

[1] Alcorn, Michael A., and Anh Nguyen. "baller2vec++: A look-ahead multi-entity transformer for modeling coordinated agents." arXiv preprint arXiv:2104.11980 (2021).

[2] Pan, Ling, et al. "Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification." International Conference on Machine Learning. PMLR, 2022.

Dear reviewer, thanks for your time reviewing this paper! We have conducted several additional experiments, and try to ease your concern below. Hope we can ease your concerns.

Q1: "The novelty of the work seems a bit limited. MADIFF heavily relies on the existing work DecisionDiffuser (Ajay et al., 2023), and it can be seen as its simple extension to the MARL setting ... Adopting an attention mechanism for multi-agents itself does not seem to be a new idea."

A1: It is indeed that many successful design choices of [1] have inspired us and guided us to derive our method. Similarly, much of [1] also referred to the design of diffuser [2], but that doesn't take away from the fact that it is an excellent work.

Compared with DM works in single-agent learning, MADiff is naturally designed for multi-agent problems. By using different conditioning during evaluation, the same framework can handle multi-agent coordination, opponent/teammate modeling, and joint trajectory prediction, which is supported by strong experimental results. We believe that the references in the single-agent learning domain do not diminish the novelty of our approach in the context of multi-agent learning.

We acknowledge that the attention mechanism has been used for years in MAL, and we verified that it is also effective in conditional generative modeling of multi-agent trajectories.

Q2: "More baseline (MADT-KD) could have been compared in the experiments."

A2: Thank you for your suggestion! To our knowledge, MADT-KD does not have an open-sourced implementation, and it requires non-trivial changes on top of MADT's training code. We include MADT [3] as an additional baseline on SMAC datasets, and the results are pasted below:

We also updated Table 1 to cover those results.

Q3: "In the top-middle of Figure 2, why is the planned trajectory of the planning agent (purple) different from the real sampled trajectory?"

A3: The plans on the top row are done at the early stage of an episode, and decisions are made in a decentralized way. Therefore, the purple agent at that time did not have much information about other agents' intentions. Then the purple agent may make conflicting plans with the red one, i.e., both want to cover the same landmark in the middle.

As the purple agent gathers more information during execution, it identifies the red agent's inclination toward the middle landmark. Thus, the purple agent's opponent U-Net model tends to generate a future trajectory for the red agent, which arrives at the middle landmark. And because of the attention mechanism, the purple agent made corrections when generating their own plans (bottom-middle subfigure). The plans of all the agents in the bottom-middle row become conflict-free, and the conflicts are eliminated in real sampled trajectories.

We have revised the explanation in Section 5.4 to make it clearer.

We have conducted addtional experiments to address reviewers' questions, and new results are summarized as follows:

1. We benchmark MADiff on Multi-agent Mujoco datasets:

   Dataset	BC	TD3+BC	OMAR	MADiff-D
   2halfcheetah-Good	6846 $\pm$ 574	7025 $\pm$ 439	1434 $\pm$ 1903	8254 $\pm$ 179
   2halfcheetah-Medium	1627 $\pm$ 187	2561 $\pm$ 82	1892 $\pm$ 220	2215 $\pm$ 27
   2halfcheetah-Poor	465 $\pm$ 59	736 $\pm$ 72	384 $\pm$ 420	751 $\pm$ 14
   4ant-Good	2802 $\pm$ 133	2628 $\pm$ 971	344 $\pm$ 631	3090 $\pm$ 26
   4ant-Medium	1617 $\pm$ 153	1843 $\pm$ 494	929 $\pm$ 349	1679 $\pm$ 93
   4ant-Poor	1033 $\pm$ 122	1075 $\pm$ 96	518 $\pm$ 112	1268 $\pm$ 51

2. We compare MADiff to ConcatDiff, a naive application of single-agent DM to MAL which concatenates all agents' observations:

   Dataset	OMAR	ConcatDiff-D	MADiff-D	ConcatDiff-C	MADiff-C
   Spread-Expert	114.9 $\pm$ 2.6	106.4 $\pm$ 3.5	98.4 $\pm$ 12.7	112.7 $\pm$ 4.3	114.7 $\pm$ 5.3
   Spread-Md-Replay	37.9 $\pm$ 12.3	31.5 $\pm$ 2.7	42.9 $\pm$ 11.6	32.2 $\pm$ 3.5	47.2 $\pm$ 6.6

3. We compare MADiff with MADT:

   Dataset	MADiff-D	MADiff-C	MADT
   3m-Good	18.8 $\pm$ 0.2	19.7 $\pm$ 0.1	19.0 $\pm$ 0.3
   3m-Medium	17.2 $\pm$ 0.3	18.4 $\pm$ 0.2	15.8 $\pm$ 0.5
   3m-Poor	11.2 $\pm$ 0.1	11.8 $\pm$ 1.0	4.2 $\pm$ 0.1
   5m6m-Good	16.5 $\pm$ 0.3	18.1 $\pm$ 0.1	16.8 $\pm$ 0.1
   5m6m-Medium	16.3 $\pm$ 0.1	17.6 $\pm$ 0.4	16.1 $\pm$ 0.2
   5m6m-Poor	10.3 $\pm$ 0.5	11.0 $\pm$ 0.3	7.6 $\pm$ 0.3