MADiff: Offline Multi-agent Learning with Diffusion Models

References

[1] Janner, Michael, et al. Planning with Diffusion for Flexible Behavior Synthesis.

[2] Iqbal, S., & Sha, F. Actor-attention-critic for multi-agent reinforcement learning.

[3] Si, C., et al. Freeu: Free lunch in diffusion u-net.

[4] Chi, C., et al. Diffusion policy: Visuomotor policy learning via action diffusion.

[5] Shen, Zhuoran, et al. Efficient attention: Attention with linear complexities.

Q1: "The novelty is limited ... which is not a novel concept in the multi-agent domain."

A1: Many successful design choices of the decision diffuser (DD) inspired and guided our method. DD also referred to the design of the diffuser [1], but it remains excellent work.

Compared with DM, which focuses on SAL, MADiff is naturally designed for multi-agent problems. By using different conditioning during evaluation, the same framework handles multi-agent coordination, teammate modeling, and joint trajectory prediction, supported by solid experimental results. We acknowledge the long-standing use of the attention mechanism in MAL and verified its effectiveness in conditional generative modeling of multi-agent trajectories.

Q2: "Why the win rate in SMAC is not reported, but instead the average return? "

A2: We adopted offline datasets and the performance of most baseline algorithms from OGMARL, which reported the average return of their baseline algorithms instead of the win rate. Therefore, we chose the average return as the evaluation metric in Table 1. We have included the win rate of MADiff and our self-implemented baseline, MADT, in three SMAC maps in Table 1 of the supplement PDF. We will re-run the other baseline algorithms and the 8m map and update the win rates to the main paper.

Q3: "Experiments use only three random seeds, whereas current MARL works typically use five or more to validate algorithm effectiveness."

A3: We conducted experiments with five seeds on three SMAC maps, as shown in Table 1 of the supplement PDF. Results generally align with our previous reported performances, showing MADiff's robustness to random seeds. We will update all experimental results to five random seeds once finished.

Q4: "Why was a diffusion model based on the U-Net architecture with an added attention mechanism chosen?"

A4: We use U-Net to model each agent's trajectory and the attention mechanism to coordinate across agents for three reasons:

• U-Net-based diffusion models are still predominantly used in offline RL due to the convolutional module's ability to fuse locally consistent subsequences into globally coherent trajectories [1].
• Attention modules applied to U-Net's skip-connected features, which are shown to constitute high-frequency information [3], allow agent interactions on the most critical parts of the trajectory, effectively steering the generation process.
• A prior study found that while transformers performed better on some imitation learning tasks, they were generally more sensitive to hyperparameters [4].

Q5: "Although the paper claims ... would help assess the source of MADiff's performance gains."

A5: Both diffusion modeling and attention networks are essential in our algorithm. Baseline Q learning algorithms use single-agent independent actors, and only their centralized critics can incorporate attention modules. To our knowledge, no prior work uses attention-based critics in offline MARL. In online MARL, attention-based critics [2] do not outperform QMIX's mixing value network. MADT frames MAL problems as sequence modeling with a fully attention-based network, and MADiff outperformed MADT in most datasets.

Q6: "As the number of agents increases, MARL faces the dimensionality curse of the joint action space, which becomes more pronounced with computationally heavy diffusion models."

A6: The potentially high computational cost of MADiff comes from two parts: the iterative denoising process and the modeling of teammate agents.

• Many off-the-shelf fast sampling techniques can be adopted to reduce the steps. For example, with DDIM samplers, we can reduce the sampling steps from 200 to 15 with negligible performance loss.
• As mentioned in Section 5.5, we choose to use a shared U-Net model for all agents, where different agents' trajectories can be batched together and passed through the network. It does not cost much more inference time as the number of agents increases with GPU-accelerated computing, as reported in Appendix Table 6. For self-attention among agents' embeddings, it can also scale with linear complexity [5].

When considering performance improvements over baseline algorithms, the higher computational cost of MADiff is still tolerable in many scenarios.

Q7: "Beyond being index-free, what other advantages does the attention mechanism provide? Can the agent observations be concatenated as a single input?"

A7: Attention is conducted on U-Net skip-connected features, allowing agent interactions on crucial trajectory parts, effectively steering the generation. We compare MADiff-C with concatenating all agents' observations (ConcatDiff) on three MPE spread datasets. Table 3 of the supplement PDF shows MADiff-C consistently performs better.

Q8: "The inverse dynamics model takes the local observations oi of the agents as input ... stably and effectively under these conditions?"

A8: The non-stationarity in the transition function $p(o'|o,a)$ does not necessarily imply non-stationary inverse dynamics $p(a|o, o')$. For example, in MPE tasks, the next observation depends on all agents' actions. However, knowing the current and next local observations allows easy determination of ego movement. Most of our benchmarked tasks don't show high non-stationarity in inverse dynamics.

If the inverse dynamics of certain tasks depend significantly on other agents, one can use DM-generated current and next observations of other agents as additional input to the inverse dynamics model.

Q9: "Can the authors elaborate on how DMs avoid extrapolation errors in offline RL?"

A9: Due to the character limitation, we response to this common question from you and Reviewer hh3p in A1 of the general response.

Q10: "Should E(si,ai,s′i) in Eq. (6) be modified to E(oi,ai,o′i)?"

A10: Thanks for your checking! We have fixed the typo.

Q1: "The paper lacks a detailed discussion on the computational complexity and scalability of the proposed approach, which is crucial for practical applications."

A1: The computational complexity of MADiff during sampling is $O(KN^3)$. The following three points warrant clarification:

• Within decentralized execution settings, each agent must make decisions independently. Consequently, one factor of $N$ in $KN^3$ is inevitable for all methods.
• As shown in Figure 1, we adopt one-layer attention only before each decoder layer. Most of the computations are still independently computed within each agent's U-Net. This majority of computation scales linearly with agent numbers. If the agent number is not very large, the attention operation will not cost much effort.
• For the diffusion step K, many off-the-shelf fast sampling techniques can be adopted. For example, with DDIM samplers, we can reduce the sampling steps from 200 to 15 with negligible performance loss.

To ensure we address your concern accurately, could you please provide more specific details on what aspects of scalability you would like us to focus on? For instance, are you interested in the scalability in terms of computational resources, dataset size, or large number of agents? Any additional guidance would be greatly appreciated.

Q2: "The motivation of this paper is unclear."

A2: In fact, you have summarized the motivation of our paper quite well in the summary and strengths of the review: "introduces an innovative approach by integrating attention-based diffusion models for multi-agent learning" to "effectively address coordination among multiple agents, which is a significant challenge in MARL". We are continually revising the paper, especially the abstract and introduction sections, to emphasize our motivation more prominently. We are a little confused regarding the unclear motivation issue you mentioned. We would appreciate it if you could clarify it more precisely.

Q3: "The evaluation primarily focuses on performance metrics; additional analysis on the robustness and adaptability of the model in diverse scenarios could provide more insights."

A3: Thanks for the suggestion. The online evaluation return is the only optimization objective in the problem formation of offline (MA)RL. Thus, we choose the return as the measure to compare with baseline algorithms. Regarding robustness, MADiff is not sensitive to hyperparameters. As listed in Appendix F.3, most of the hyperparameters are kept the same for all datasets. We only tune the reward discount, return scale, and diffusion horizons. Regarding adaptability, could you please specify what aspects of analysis or testing you would like to see?

Q4: "The motivation for using DMs to solve the extrapolation errors in offline MARL is unclear to me. Why using the DMs can solve it? What are the strengths of DMs compared to previous works?"

A4: As the similar question was proposed by Reviewer joCT, please refer to the general response A1 for our answer to this issue.

Q5: "The attention mechanism and DMs for single-agent reinforcement learning (RL) have been extensively studied in previous works. This paper appears to simply integrate these two methods into MARL. Could you please provide more details on how this approach differs from directly applying these methods in MARL settings?"

A5: Our contributions include but beyond the model design, and the most important one is that it allows the same model structure to handle various tasks (CTCE, CTDE, MATP) by flexible conditioning during evaluation and achieves superior results over baseline algorithms. We conducted ablation experiments on model design and case studies about teammate modelling with diffusion models in multiple environments, all of which are part of our contribution and are novel. We believe that our work, being the first to apply the diffusion model in multi-agent learning, is valuable. To better address your concern, could you specify what you mean by "directly applying these methods in MARL settings"? Thank you.

Q6: "How does MADIFF perform with an increasing number of agents in MARL systems? Additionally, I am very interested in how MADIFF addresses the challenges of adding a new agent to the existing MARL systems."

A6: As shown in Table 1 and Table 2, our experiments are done on various sets of MARL systems ranging from 2 to 10 agents, and achieve superior performances on most of them. Adding a new agent to the existing MARL systems is a different problem known as ad hoc teamwork [1], and is beyond the scope of our paper. To the best of our knowledge, there is no offline algorithm that tackles the ad hoc teamwork problem.

Q7: "Related works should discuss using diffusion for single-agent RL more."

A7: Thank you for your kind reminder. We have included an additional related work subsection to elaborate more on prior studies that using DMs in RL.

Q8: "It would be greater to add some discussions of real-world applications."

A8: MADiff has the potential to be applied in scenarios such as multi-robot collaboration, autonomous driving, and multi-player match data analysis (similar to our NBA experiments). We'll include a discussion of real-world applications in the paper.

References

[1] Chen, S., et al. Achieving the ad hoc teamwork by employing the attention mechanism.

Thank you for participating in the discussion phase.

Our work goes beyond just borrowing DMs from single-agent learning to MARL datasets. By using flexible evaluation-time conditioning, MADiff effectively handles various MAL tasks (CTCE, CTDE, and MATP). Note that our experiments show that during decentralized execution, using a centralized-trained DM to predict teammates' trajectories enhances each agent's decisions. This enhancement allows MADiff-D to outperform the model variant where each agent generates its own trajectory, which is a direct application of DMs in single-agent learning to MAL. Therefore, we consider our work a novel and non-trivial use of DMs in MAL, not incremental.

Q9: "Why does attention mechanism work?""Why you choose attention mechanism here?"

A9: We choose our model design, the U-Net with cross-agent attention, for three reasons:

• Not all generated teammate trajectories are equally important for ego planning. The attentional mechanism can learn to focus on interactions with important teammates, as demonstrated in Figure 1 of the supplement PDF.
• We compare our model design with concatenating all agents' observations (ConcatDiff, Table 3 of the supplement PDF) and independently generating ego trajectories (Section 5.5), with notably better performance.
• The attention modules are applied to the skip-connected features of the U-Net, which are shown to constitute high-frequency information. This allows the interactions among agents to happen on the most critical parts of the trajectory, most effectively steering the generation process.

Q10: "Are there any other possible solutions?"

A10: Our model design is not the only option. As mentioned in A9, we compared MADiff with other modeling choices and achieved better results. It is possible that some non-trivial modeling designs could surpass our current results. As the first to apply diffusion models in MAL, we will be thrilled if our work can inspire further research.

Thanks again for your time and effort in reviewing our paper!

In addition to the MPE experiments in Section 5.5, we compared MADiff with independent DMs in MAMujoco datasets. Since shared-parameter independent DMs performed significantly worse than non-shared ones in Section 5.5, here we compared MADiff (with shared U-Net parameters) to independent DMs without parameter sharing in the decentralized execution setting. The results are listed below:

To validate the advantages of the attention mechanism over direct concatenation (ConcatDiff) in modeling multi-agent interactions, we conducted ablation experiments on MAMujoco datasets in the decentralized execution setting. When performing distributed execution with ConcatDiff, we condition only on the dimensions corresponding to the ego agent in the current and historical observations, while the other dimensions are generated by diffusion. The results are listed below:

Q1: "The novelty of the work is somewhat limited, being seen as a simple extension of Decision Diffuser to the MARL setting with an additional attention layer."

A1: It is indeed that many successful design choices in single-agent diffusion learning have inspired us and guided us to derive our method. Similarly, much of Decision Diffuser (DD) [1] also referred to the design of Diffuser [2], but that does not take away from the fact that DD is excellent work.

Also, many of the well-acknowledged multi-agent RL methods inherit a lot from single-agent RL in both online (MADDPG, MAPPO) and offline (OMAR, MADT) settings. MADiff is the first to apply diffusion models in multi-agent settings, naturally designed for multi-agent problems and taking multi-agent coordination, opponent modeling, and trajectory planning in a unified framework.

Q2: "Writing quality needs improvement, especially to a table to explain the mathematical symbols would help."

A2: Thanks for pointing it out. We have made a table that lists and explains all the mathematical symbols used. Due to the space limit, we are unable to include it in the supplement PDF. We will add the table in the Appendix of our paper.

Q3: "Is it unclear how sensitive is the MADiff's performance to some critical hyperparameters, ..."

A3: Thanks for the question.

• The diffusion loss and inverse dynamics loss are independent and used to train different model parameters, so one does not need to balance between them.
• We did not tune the number of encoder and decoder layers for each experiment setting. We use three encoder layers and three decoder layers for all our experiments. Intuitively, for tasks with smaller observation dimensions (e.g., MPE), a smaller number of network layers may suffice.

Q4: "How are 'Decentralized policy and centralized controller' related to 'centralize training decentralize execution'?"

A4: Our proposed algorithm has two variants depending on the use case. Both MADiff-C and MADiff-D are trained in a centralized way.

• MADiff-C: If the task setting allows for a centralized controller to set actions of all agents, the diffusion model can be used to generate trajectories for all agents conditioned on all agents' current observations. This is the "centralized training with centralized execution" case.
• MADiff-D: If the agents are required to make decisions independently using a decentralized policy based on their local observations, the diffusion model is used to generate ego trajectory along with predictions of other teammate agents. This is the "centralized training with decentralized execution" case.

Q5: "What is condition y during training in Algorithm 1 and 2?"

A5: Condition $y$ refers to other conditions besides the historical (and current) observations used in trajectory generation .

• In offline MARL experiments, the condition $y$ is the cumulative discounted reward in the offline dataset. During online evaluation, it is set to a relatively large value to generate high-rewarded behaviors.
• In MATP experiments on the NBA dataset, the condition $y$ includes the ball's historical trajectories, player ids, and a binary variable indicating the side of each player's frontcourt, as described in Line 256-257.

Q6: "Please check whether the Classifier-free guided generation part in Equations 4 and 6 is represented correctly or explained intuitively. Although it is known that these equations are similar to (Ajay, 2023), the paper should ensure its self-containedness."

A6: Thanks for your suggestion! Regarding your questions:

• As mentioned in A3 to reviewer vCUA, there is a typo in Equation 4 and the correct form should be $\hat{\epsilon} := \epsilon_{\theta}(\hat{\tau}_k,\emptyset, k) + \omega(\epsilon\_\theta(\hat{\tau}_k, y(\tau), k)-\epsilon\_{\theta}(\hat{\tau}_k,\emptyset, k))$ , and $\omega$ is a scalar to balance the conditioned and unconditioned model output. If $\omega$ is set to 1, it reduces to pure conditioned generation. Prior diffusion model studies reveal that an $\omega$ larger than 1 can yield better condition quality [3], as it explicitly forces the sample to stay away from unconditioned regions.
• The empty set means the model is not conditioned on any extra information $y$. In practice, it is implemented as zero out the embedding of $y$.
• $\beta$ is not a constant, but a random variable which has the Bernoulli distribution (we choose $p(\beta=1)=0.25$ for all our experiments). It means during training, the embedding of $y$ has a probability of 0.25 to be masked with all zeros. Therefore, the trained diffusion model can be used to provide both conditioned and unconditioned outputs in Equation 4 with the same set of parameters.

We have revised the explanation of the above symbols in the paper to ensure sufficient self-containedness.

References

[1] Ajay, Anurag, et al. Is Conditional Generative Modeling all you need for Decision Making?.

[2] Janner, Michael, et al. Planning with Diffusion for Flexible Behavior Synthesis.

[3] Ho, J., & Salimans, T. Classifier-Free Diffusion Guidance.

Q1: "The proposed method is mostly evaluated across domains with a small number of agents (up to 8) and MADiff with decentralized execution requires each agent to predict all agents' trajectories, which is difficult to be applied to larger scales."

A1: Thanks for your comment!

• Most existing works in offline MARL do not consider tasks with a large number of agents. For example, MADT-KD [1] and OMAR [2] are tested on environments with up to 8 agents. Meanwhile, our MATP experiments are conducted on NBA datasets with 10 agents.
• As predicting all agents' trajectories introduces more computational cost, as shown in Appendix Table 6, the computation time does not increase much with the number of agents thanks to GPU-accelerated computing.
• When the number of agents is extremely large, teammate agents can modeled by a latent diffusion model [3] with the agent dimension reduced, thus avoiding the curse of dimensionality. We leave this direction to future work.

Q2: "It would be beneficial to provide some discussions about how attention help with the teammate modeling for decentralized execution."

A2: In a decentralized execution setting, although different agents make decisions independently, they can use the same attention network to jointly infer other agents' future trajectories and plan their own trajectories. Since the attention network is trained in a centralized manner, all agents' trajectories should be reasonable, and tend to be consistent given enough information. Intuitively, it requires the ego agent to first think at a higher level, i.e., make a coordinated plan for all agents, and then place itself in that plan. We will include more discussion of the role of attention modules in CTDE settings in our paper.

Q3: "The clarity of section 2.3 could be improved. For example, what is $w$ in Equation (4)?"

A3: Thanks for the careful checking! Equation 4 should be $\hat{\epsilon} := \epsilon_{\theta}(\hat{\tau}_k,\emptyset, k) + \omega(\epsilon\_\theta(\hat{\tau}_k, y(\tau), k)-\epsilon\_{\theta}(\hat{\tau}_k,\emptyset, k))$ .

Q4: "The subscripts used in Equation (7) and (8) are not consistent with the ones in Figure 1."

A4: We have revised the subscripts in Figure 1 to be the same as those in Equation 7 and 8.

Q5: "In decentralized execution with teammate modeling, agent cannot access other agents' current observations ... Some visualizations of the heatmap of the attention layer for communication and interaction among agents might be helpful."

A5: Great question, thanks! In decentralized execution, teammate modeling is most important for reasonableness given local information, not correctness (which is hard to achieve). Figure 2 shows that the purple agent trajectory predicted by the red agent differs from it's own plan. However, the three agents still cover different landmarks in the red agent's imagination, and is valid in completing the teamwork. The model also has the ability to adjust the prediction given more information from other agents is observed later.

Regarding the potential problem you mention, the most extreme case is when some teammates are completely outside the visible range, at which point their imagined current observations could be highly stochastic and the model could not use those estimated trajectories to make decisions. We verified on SMACv2 that the attention mechanism mitigates this problem. Figure 1 of the supplement PDF shows the attention scores in two environment states, and we found that the teammates outside the visible range do have lower attention scores.

Q6: "Is the centralized training procedure exactly the same for both centralized control and decentralized execution with teammate modeling?"

A6: Yes, the centralized training procedure is the same for CTCE and CTDE.

Q7: "In section 5.4, how do you define two paths are consistent?"

A7: In Section 5.4, consistency is defined on plans made by different agents. Two agents' plans are consistent if the ego trajectories in both plans cover different landmarks. Therefore, if inconsistent plans are executed without runtime correction, two agents will cover the same landmark, and the 'Spread' task will fail. We have revised the corresponding part of the paper for better clarity.

Q8: "In Line 307 and 308, it is said that 'the red agent and the purple agent generate inconsistent plans,' but I think the inconsistent plans are generated by purple and green agents (i.e., planning agents) right?"

A8: In Figure 2, at t = 0, the plans (dashed lines) in the first and second plots are generated by the red and purple agents, respectively (the planning agents are represented by triangular marks). The ego trajectories (starting from triangular marks) are both heading towards the middle landmark. Thus, the red and purple agents generate inconsistent plans.

Q9: "How will planning horizon $H$ affect the performance?"

A9: We did not delicately tune the planning horizon, and it is decided for each environment and kept the same for datasets from the same environment. The choice of planning horizon is a tradeoff between planning ability and computational efficiency. A short planning horizon can not take full advantage of diffusion planning, and an overlong planning horizon requires more computing resources and potentially more training data. We report the performance of MADiff-C with different planning horizons in SMAC 3m-medium dataset in Table 2 of the supplement PDF.

References

[1] Pan, Ling, et al. Plan better amid conservatism: Offline multi-agent reinforcement learning with actor rectification.

[2] Tseng, Wei-Cheng, et al. Offline Multi-Agent Reinforcement Learning with Knowledge Distillation.

[3] Rombach, Robin, et al. High-resolution image synthesis with latent diffusion models.