DoF: A Diffusion Factorization Framework for Offline Multi-Agent Reinforcement Learning

We would like to express our gratitude for your time and effort in reviewing our work. Thanks for recognizing the novelty of the IGD principle that generalizes IGM, the theoretical proof of DoF satisfying the IGD principle, and the solid empirical experiments. We address your reviews as follows.

Potential limitations of noise and data factorization functions - the primary options used in the paper are Concat and Wconcat, which might lack modeling power.

Thank you for your insightful comment. We agree that Concat and Wconcat have limited modeling power, they are preferred due to their ability to satisfy the IGD property and conform to the Centralized Training with Decentralized Execution (CTDE) paradigm. In addition to using Concat and Wconcat for noise and data factorization, we also explored alternative methods such as QMix, which has better modeling ability.

QMix is a famous value factorization method, it models the monotonic increasing relationship among individual value function $Q_i$ and global value function $Q_{tot}$. For the QMix-based DoF, it uses the QMix function to factorize the noise. Each individual noise $\epsilon_i$ is mixed with a learned monotonic increasing function into the global noise $\epsilon_{tot}$. QMix-based DoF performs poorly in diffusion tasks, as shown in Table 6.

When using the QMix function as the noise factorization function, it has two drawbacks: non-Gaussian distribution and non-decentralized execution. For the non-Gaussian distribution drawback, QMix models the monotonic relationship among noises. It does not restrict the monotonic relationship as linear additive relationship. Thus, the joint noise $\epsilon_{tot}$ modeled by QMix cannot guarantee to follow the Gaussian distribution. As DoF is built on the Diffusion model which assumes that the noise follows the Gaussian distribution, this violation of the Gaussian distribution assumption leads to poor performance, as shown in Table 6 (column QMix). The non-decentralized execution drawback is that the agent of QMix-based DoF cannot be decentrally executed. In each diffusion step, each agent relies on the noise and the data of other agents.

Therefore, we believe Concat and Wconcat offer a better trade-off between performance, scalability, and theoretical constraints. The IGD principle proposed in our paper highlights an important direction for future research. We hope this principle inspires further exploration, potentially leading to more powerful and scalable noise factorization and data factorization functions in the future.

Table 6 compares decentralized methods (WConcat) with 'centralized' methods (attention), showing similar performance. To properly interpret these results: How is the centralized attention method implemented during execution?

Thank you for your insightful question. Centralized attention adopts a centralized training and centralized execution Paradigm, as detailed in the explanation of the noise factorization method using the Atten implementation in Appendix C.1.3. The noise factorization function is expressed as:

$$\epsilon_{t o t}\left(\mathbf{x}^k, \mathbf{y}, k\right)[(i-1) \times d: i \times d]=\sum_{j=1}^N w_i^j \epsilon_j\left(\mathbf{x}_{\mathbf{j}}^k, \mathbf{y}, k\right)$$

Where $w_i^j$ represents a set of weights learned through an attention mechanism. Its implementation is similar to Wconcat.

While the 3m and 5m_vs_6m homogeneous environments (as shown in Table 6) are relatively simple, resulting in comparable performance between $f=Wconcat$ and $f=Atten$, We conducted additional experiments in more challenging map environments, including 2c_vs_64zg (homogeneous) and 3s5z_vs_3s6z (heterogeneous), to better illustrate the differences. The performance comparison is shown below：

From the table, it is evident that the centralized attention method ($f=Atten$) outperforms Wconcat ($f=Wconcat$) in both the homogeneous 2c_vs_64zg and the heterogeneous 3s5z_vs_3s6z environments.

What observations/information does centralized attention have access to versus decentralized methods? Is centralized attention meant to serve as an upper-bound baseline (with access to information not available in real deployments) rather than a competing method?

Thank you for your question. In the case of centralized attention, during centralized training, agent $i$ generates noise based on the actions and observations of other agents. However, during execution, due to the requirement for consistency between forward noise addition and denoising, agents must still access the observations of other agents. It means that agents must be executed centrally rather than decentrally.

In contrast, decentralized methods like Concat and WConcat require only the noise information from agent $i$ itself during execution, without the need for information from other agents.

DoF seeks to address the scalability issue of diffusion-based MARL. We agree with the reviewer's observation that centralized attention represents an upper bound on achievable reward, as it utilizes information beyond what decentralized methods can access. Although centralized attention could achieve higher rewards, it lacks the scalability of Concat and WConcat due to increased computational and memory demands, which makes Concat and WConcat the lower-bound in terms of scalability.

The authors claim in section 4.3.2 that 'the data factorization h could model more powerful data relationships than the noise factorization function f'. However, Table 14 shows minimal performance improvement when using attention (a more complex function) versus WConcat for data factorization h. Are there theoretical or empirical evidence that supports the claim that a more complex h can compensate for using simple noise factorization functions f (as is mainly used in the paper)?

Thank you for your question. Our claim that a more complex data factorization $h$ can compensate for the simpler noise factorization function $f$ is supported by an illustrative example and empirical evidence.

First, in Table 11, different data factorization $h$ are employed to reconstruct a one-step payoff matrix. The results show that $h=Atten$ reconstructs a better payoff matrix than other functions.

Second, Table 14 shows the results for the 3m and the 5m vs 6m scenarios, where agents are homogeneous. Although it shows that $h=Atten$ is slightly better than $h=Wconcat$, it is not significant. This is due to the fact that in such a homogeneous setting, a linear weighted relationship among agents could be enough. To further validate the effect of a more complex data factorization function, we test these methods in a heterogeneous 2s3z and 3s5z_vs_3s6z environment, where agents are heterogeneous. The results are presented below:

As shown in both Table 14 and the heterogeneous 2s3z and 3s5z_vs_3s6z environment, the choice of data factorization $h$ has a significant impact on performance. The trend $h = \text{Atten} > h = \text{WConcat} > h = \text{Concat}$ demonstrates that more complex data factorization methods provide a clear advantage, supporting our claim.

To make it easier for the reviewer to examine the results, we have included both Table 11 and Table 14 directly below.

Table 11: Payoff matrix of one-step matrix games and reconstructed value functions to approximate the optimal policy.

(a) Game Payoff Matrix 1

$b$ $h = \text{Concat}$

$c$ $h = \text{Atten}$

$d$ Game Payoff Matrix 2

$e$ $h = \text{Concat}$

$f$ $h = \text{Atten}$

Table 14: Ablation Study Data Factorization functions $h$

We hope this presentation helps clarify our results and makes it easier for the reviewer to assess the evidence.

Thanks for your time and effort to read our work. We highly appreciate your comments about the novelty of the IGD principle, the usefulness of the noise factorization and data factorization, and the extensitive experimental results.

The paper lacks clarity in some important parts. It introduces DoF-Policy and DoF-Trajectory but does not specify the suitable scenarios for each. making it difficult to assess the relative strengths of the two methods. The results section also fails to indicate which algorithm was used, and Figure 6 only includes DoF-Policy results.

Sorry for the confusion, most of the results reported are DoF-Trajectory. In the experimental section, we had written "the generated data are trajectories." in line 343 of the initial subsmission. It was written to indicate that most of the results are from DoF-Trajectory. However, it is not work as expected. We have updated the text as "in default, the results for DoF-Trajectory are reported." in the experiment section. Specifically, the experiments for SMAC, SMAC v2, and MPE in the main text use the DoF-Trajectory agent, while Appendix D.3, Table 13, and the corresponding figures present results with the DoF-Policy agent.

The core contribution of our work is the IGD principle and the diffusion factorization methods (noise factorization and data factorization). The DoF-Trajectory and DoF-Policy agents are developed to demonstrate that DoF can be used for both the trajectory-generation and the policy-generation cases. We do not present all the results of DoF-policy and DoF-trajectory as we want to focus on the factorization ability of DoF rather than the ability of a specific agent.

In essence, DoF-Policy is a combination of DoF factorization functions with the DiffusionQL[1] agent. And DoF-Trajectory is a combination of DoF factorization functions with the Decision Diffuser[2] agent. We have added the following sentence "To demonstrate the flexibility of DoF, we implement two agents based on agents of Decision Diffuser and DiffusionQL, respectively. " into Sec 4.4. DiffusionQL and Decision Diffuser are two seminar Diffusion-based approaches. DoF-Policy models the relationship between environment states and actions, making it suitable for continuous action spaces. DoF-Trajectory models trajectories and plans based on them. It is suitable for both continuous and discrete action environments. In most cases, DoF-trajectory is recommended. However, it needs to learn the inverse dynamic model which could be slower than DoF-policy.

The results section also fails to indicate which algorithm was used, and Figure 6 only includes DoF-Policy results.

Following your advice, we have conducted experiments comparing DoF-Trajectory and DoF-Policy in MPE tasks, the experimental results are shown in the following table. It will be included in the appendix E.4 (Table 20) for completeness.

Additionally, implementation details for the inverse dynamics model in DoF-Trajectory are missing.

The inverse dynamics model used in DoF-Trajectory is taken from the Decision Diffuser [2], with details provided in Appendix C.3 (C.44) and (C.45) of the initial submission. This specific model is not included in the main text as it is not the central contribution of this work. We have added "Please refer the details of the agents in Appendix C.3." in Sec 4.4.

Reference

[1] Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In ICLR 2023.

[2] Ajay, Anurag, et al. "Is Conditional Generative Modeling all you need for Decision Making?" ICLR 2023.

In Figure 2(a), there is a mislabeling of the reverse diffusion process: the output should be $x^{k-1}$ instead of $x^k$ , which may mislead the understanding of the process. Additionally, the caption of Figure 2 lacks necessary descriptions, which reduces its readability.

Thank you for pointing this out. There was a mislabeling in Figure 2(a), where the output should have been labeled as $x^{k-1}$ instead of $x^k$. We have corrected this error in the revised manuscript. Additionally, we have updated the caption of Figure 2 to provide more detailed descriptions to improve its clarity and readability. The updated caption is described as follows:

Figure 2: DoF Overview: (a) Diffusion Factorization: in each diffusion (forward and backward) step $1\le k \le K$, the noise factorization function $f$ is used to factorize the noises $\epsilon_i^k$ and intermediate data $x_i^{k-1}$. In the last backward step $k=0$, we apply the data factorization function $h$ to model the complex relationship among data generated by each agent. (b) DoF Trajectory Agent: generating trajectory for planning. $c$ DoF Policy Agent: generating actions for execution.

The paper's benchmarks are based on state vectors, making it unclear whether the proposed method can be applied to environments with high-dimensional observations. The Scalability Experiment section only compares GPU memory and time with MADIFF, without providing comparisons to other baseline algorithms. Since diffusion-based methods require multiple iterations to generate the final output, does this lead to significantly higher computational costs compared to traditional methods?

Thank you for raising this question. We fully recognize the importance of high-dimensional observation environments in practical applications and understand the challenges they present. In the field of multi-agent reinforcement learning, research typically focuses on learning cooperative strategies. The use of state vectors is a widely adopted approach, particularly in typical multi-agent environments such as SMAC, MPE. Compared to directly processing high-dimensional image inputs, state vectors indeed have lower dimensionality, which introduces certain limitations to the applicability of our approach. However, this is not only a limitation of our method but also a common challenge in the current state of the MARL field. We hope that there could be more advanced MARL environments that enable us to evaluate the performance of DoF with vision-based observations. We have added such discussion into the limitaion section in Appendix E.2.

We agree with the reviewer's concerns about computational costs. In our current scalability experiments, we mainly compared with MADIFF, another diffusion-based method, in terms of inference speed and GPU memory usage. Indeed, compared to traditional methods, diffusion-based approaches require multiple sampling iterations, which incur higher computational overhead. To provide a more comprehensive evaluation of performance, we have evaluated DoF and MADIFF in terms of the neural network parameters of their agents, and we have studied the impact of different accelerated sampling methods and compared them with two non-diffusion methods.

1. Parameter Efficiency: We measure network parameter counts per agent to evaluate the model's efficiency and scalability. The following table shows the parameter counts for DoF and MADIFF agent network with different numbers of agents.

2. DoF with enhanced sampling techniques and more baseline comparison: We explore sampling acceleration techniques, including DDIM[3] and Consistency Model[4], and added comparisons with non-diffusion methods (e.g., MACQL[1], MABCQ[2]) to better understand the computational trade-offs.

Inference Time Costs (in seconds) and reward：

These results demonstrate that, while diffusion-based methods generally have higher computational costs, techniques like DDIM and Consistency Model significantly reduce inference times of DoF with DDPM, enhancing their practical scalability. With fast sampling techniques such as DDIM and the consistency model, the rewards of DoF slightly decrease, but its inference time significantly decreases. For example, when DoF adopts the consistency model technique, it always performs better than the other non-diffusion methods (MACQL and MABCQ), at the cost of moderately higher inference time (2.4s vs 1.6s). The above results are included in Appendix E.4 .

Our choice of diffusion models for multi-agent decision making is primarily motivated by their powerful modeling ability, the avoidance of estimating value function, and the ability to generate novel behaviors. We have shown the powerful modeling ability of DoF in Section 5.1. We have shown that through modeling the trajectory rather than modeling the value function, DoF-Trajectory circumstances the need for value function estimation, which is error-prone. And DoF-Trajectory can obtain better performance than others (Table 2, 3, 4). Through a flexible combination of conditions, we can generate novel behaviors at test time, such as combining multiple skills or constraints. We validate this property through specific case studies in Appendix D.6. In Appendix D.6, two datasets, each containing a semi-circle, are used to train DoF-trajectory, and through the combination of multiple conditions, DoF-trajectory can generate a full circle.

Reference

[1] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In NeurIPS, 2020.

[2] Jiechuan Jiang, Zongqing Lu. Offline decentralized multi-agent reinforcement learning. In ECAI, 2023.

[3] Jiaming Song, Chenlin Meng, Stefano Ermon. Denoising Diffusion Implicit Models. In ICLR, 2021.

[4] Yang Song, Prafulla Dhariwal, Mark Chen, Ilya Sutskever. Consistency Models. In ICML, 2023

Given that QMIX is an online algorithm not designed for offline reinforcement learning, could you clarify the rationale for including it in the comparisons? What insights does this comparison provide?

Including QMIX in our comparisons follows the approach of prior offline MARL approaches such as MADiff [1] and OGMARL [2]. Although QMIX is primarily an online algorithm, its inclusion allows us to evaluate the effectiveness of famous online value-decomposition methods, in offline reinforcement learning scenarios. Following the advice of the reviewer, we have removed the results of QMIX from Table 3.

Can diffusion-based methods effectively handle out-of-distribution situations?

Thank you for the insightful question. We believe diffusion methods exhibit certain advantages in handling out-of-distribution (OOD) situations. When using DoF-Trajectory, from the perspective of trajectory modeling, diffusion methods bypass the traditional Q-value estimation step commonly used in reinforcement learning [3]. This characteristic naturally alleviates the issue of Q-value overestimation (e.g., extrapolation and bootstrapping) in offline scenarios, especially when learning from limited or biased data, thereby mitigating OOD problems. When using DoF-Policy, from the perspective of policy modeling, diffusion models model the distribution of the underlying offline policy well. In this way, it effectively reduces extrapolation errors caused by sampling out-of-distribution actions [4]. We have added this paragraph to the discussion section in Appendix E.3.1 of our submission.

Additionally, can these methods be applied to competitive or mixed-motivation scenarios?

Thank you for your question. As it is written in the abstract, in the introduction, and the background of the initial submission, We consider cooperative multi-agent decision-making tasks. We have not delved into competitive or mixed-motivation settings. However, we believe this is a highly valuable direction for future research. To address this, in future studies, we could extend our framework by explicitly incorporating motivations into the diffusion conditions. For instance, in competitive scenarios, the diffusion model could condition adversarial strategies or payoff structures during sampling. In mixed-motivation scenarios, it could integrate both individual motivations and shared objectives into the diffusion process. This enhancement would enable the model to better handle the complexity of competitive and mixed-motivation environments, paving the way for broader applications of our approach. We have added this paragraph to the discussion section in Appendix E.3.2 of our submission.

Reference

[1] Zhu, et al. "MADiff: Offline Multi-agent Learning with Diffusion Models." NeurIPS 2024.

[2] Formanek, et al. "Off-the-Grid MARL: Datasets and Baselines for Offline Multi-Agent Reinforcement Learning." AAMAS 2023.

[3] Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In ICLR 2023.

[4] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In ICLR, 2023

Dear Reviewer y4iS,

We have responded to your reviews 9 days ago. Can you please read our response? Are there still any remaining concerns?

Best regards,

Authors

Dear Review y4iS,

Thank you for your thoughtful feedback. We appreciate your recognition of the novelty of the IGD principle, the usefulness of noise and data factorization, and the extensive experiments.

In response, we have conducted scalability experiments, clarified scenarios for DoF-Policy and DoF-Trajectory, and revised the experimental section for clarity based on your suggestions.

We have received positive feedback from another reviewer, noting that our responses addressed the reviewer's concern and led to an improved rating. We hope our responses address your concerns and would appreciate it if you could reconsider your rating.

Best Regards,

Authors

If I understand correctly, Theorem 1 basically states that each component of isotropic Gaussian is independent. I feel this result is very limited and hardly holds in general. For instance, it’s unclear whether this would still apply even when the Gaussian has a non-diagonal covariance.

The understanding is not correct. We do not assume the data follow a Gaussian distribution. Theorem 1 does not state that each component of the isotropic Gaussian noise is independent. Instead, it establishes that a multi-agent diffusion model $p _ {\boldsymbol{\theta} _ {tot}}$ represents the data ($x _ {tot}^0$, rather than noise) following distribution (e.g., trajectories or actions), and can be factorized into individual data $x _ i$. Theorem 1 can be viewed as after training, it requires that each component $x _ i^0$ of a generated data $x _ {tot}^0$ is independent. The theorem focuses on modeling the global data distribution $p _ {\boldsymbol{\theta} _ {tot}}$ as a combination of locally optimal models $p _ {\boldsymbol{\theta} _ i}$, rather than making assumptions about the independence of Gaussian noise.

For diagonal covariance Gaussian noise, the DDPM and the majority of the diffusion model use diagonal covariance Gaussian distribution for both the noise and the transition probability. Our work is built based on these Diffusion models. The use of the Gaussian distribution with diagonal covariance is inherited from the DDPM framework [1]. Researchers have demonstrated that through using the diagonal Gaussian distribution, the diffusion model can generate very realistic images, and we have shown that through using the default Gaussian noise, DoF can generate very good decision.

If a diffusion model uses other forms of distributions instead of Gaussian distributions, it is unclear whether Theorem 1 still holds. We have added the following sentence in Theorem 1 "The noise $\epsilon_{tot}$ and $\epsilon_i$ and the transition probability ($p _ {\theta _ {tot}}(\mathbf{x^{k-1} _ {tot}}|\mathbf{x^k _ {tot}})$ and $p _ {\theta _ {i}}(x^{k-1} _ {i}|x^k_i)$) follow diagonal Gaussian distributions".

Minor comments:

• In eq (3), instead of $θ_i ⊂ θ_{tot}, (θ_1,...,θ_K) = θ_{tot}$ would be better, as the order of coordinate matters.
• $\epsilon_{1}^{\theta_1}$ in eq (5) is not clearly defined.

We do not agree with the reviewer. Although the Concat and WConcat functions implement $(θ_1,...,θ_K) = θ_{tot}$, in the definition of IGD, we expect there could be other functions satisfy $\theta_i \subset \theta_{tot}$.

$\epsilon_{1}^{\theta_1}$ in eq (5) was defined in lines 250-255 (around Equation 5) in the initial submission. We copy the text for the reviewer's reference "multiple small noise models $\epsilon_{i}^{\theta_i}(x_i^{k}, k)$, each parameterized by $\theta_i$. $\theta_i \subset \theta_{tot} \quad \forall i,\;\theta_i \cap \theta_j = \emptyset\;\; i\neq j$. During execution, agent $i$ uses the noise model $\epsilon_i^{\theta_i}$ to generate data". We will make it more clear.

Can the authors provide formal statements and proofs showing that IGD generalizes IGM, individual global optimal principle or risk-sensitive IGM principles?

Please refer to the above proofs.

Are there any nontrivial examples that satisfy IGD beyond the Gaussian case ?

We would like to clarify that our method, DoF, is built on the diffusion modeling approach introduced in the DDPM[1] framework, which uses Gaussian noise. We do not assume that the data follows the Gaussian distribution. Similarly, well-known works in reinforcement learning, like Diffuser[2], Diffusion-QL[3], and Decision Diffuser[4], also use Gaussian noise. Other types of noise distribution is possible, but it is out of the scope of our work.

The claim that IGD generalizes IGM is not well-defined. IGM needs a value function but does not necessarily have a distribution over action or even under a diffusion model. Please give an explicit statement on the generalization: For instance, given a value function $Q_{jt}$ satisfies IGM what is the corresponding $p_{\theta_{tot}}$ such that Eqn.(1) holds for any history $\tau_{tot}$.

At the end of section 4.2, there should be formal statements of IGD vs individual global optimal principle or risk-sensitive IGM principles.

We had shown the ability of DoF to generate value functions in Table 1 of the initial submission. DoF can be used to recover the optimal policy.

To show that the principle is a generalization of the IGM principle, we first show that the IGD principle is a generalization of the IGO principle. We have added these proofs into the Appendix E.1 .

The definition of the IGO Pinrciple [1]: For an optimal joint policy $\pi_{tot}^*(\mathbf{u_{tot}} \mid \mathbf{\tau_{tot}})$ : $\boldsymbol{\mathcal { T }} \times \boldsymbol{\mathcal { U }} \rightarrow[0,1]$, where $\mathbf{\tau_{tot}} \in \boldsymbol{\mathcal { T }}$ is a joint trajectory, if there exist individual optimal policies $\left[\pi_i^*\left(u_i \mid \tau_i\right): \mathcal{T} \times \mathcal{U} \rightarrow [0,1]\right]_{i=1}^N$, such that the following holds:

$$\pi_{tot}^*(\mathbf{u_{tot}} \mid \mathbf{\tau_{tot}})=\prod_{i=1}^N \pi_i^*\left(u_i \mid \tau_i\right),$$

then, we say that $\left[\pi _ i\right] _ {i=1}^N$ satisfy $\mathbf{I G O}$ for $\pi _ {tot}$ under $\boldsymbol{\tau _ {tot}}$.

The IGD principle requires that

$$\prod_{i=1}^N p_{\theta_i}(x_i^0) = p_{\theta_{tot}}(x_{tot}^{0})$$

where $x_i^0$ is the generated data of agent $i$, and $x_{tot}^0$ is the generated data of the whole multi-agent sytstem, and $p_{\theta_i}(x_i^0)$ is the probaility of $x_i^0$, and $p_{\theta_{tot}}(x_{tot}^{0})$ is the probability of $x_{tot}^0$.

We can use the IGD principle to generate multi-agent action. Let's treat the generated data $x_i^0$ as $u_i$, where $u_i$ is the action taken by agent $i$. Then, the probability $p_{\theta_i}(u_i)$ becomes $\pi_i(u_i\mid \tau_i)$, where $\pi_i$ is the policy of agent $i$ and $\tau_i$ is the local observation. Further, let's treat the generated total data $\mathbf{x_{tot}^0}$ as $\mathbf{u_{tot}}$. The probability $p_{\mathbf{\theta_{tot}}}(\mathbf{u_{tot}})$ becomes $\pi_{tot}(\mathbf{u_{tot}}\mid \mathbf{\tau_{tot}})$, where $\pi_{tot}$ is the policy of the multi-agent system, and $\tau_{tot}$ is the aggregated observations. Then the IGD principle becomes the following formulas.

$$$$ $$\prod_{i=1}^N \pi_i(u_i\mid \tau_i) = \pi_{tot}(\mathbf{u_{tot}}\mid \mathbf{\tau_{tot}})$$

The above formula requires that for any policy $\pi$, the total policy $\pi_{tot}$ is equal to the product of its per-agent policy. By substituting the optimal policy $\pi_i^*$ for the policy $\pi_i$ and the optimal total policy $\pi_{tot}^*$ as the policy $\pi_{tot}$, we can obtain the following formula.

$$$$ $$$$

The above formula is exactly the requirement of the IGO principle. Thus, we have shown that the IGD principle is a generalization of the IGO principle.

The IGD principle is a generalization of the IGM principle

We have shown that the IGD principle is a generalization of the IGO principle. And the IGO paper [1] shows that the IGO principle is a generalizatioin of the IGM principle, thus the IGD principle is a generalization of the IGM principle.

The IGD principle is a generalization of the RIGM principle

We first describe the RIGM principle [2], and we introduce a generation of the RIGM principle (GRIGM), in the end we show that the IGD principle is the generation of the GRIGM principle. Thus, the IGD principle is a generalization of the RIGM principle.

The definition of the RIGM principle: Given a risk metric $\psi_\alpha$, a set of individual return distribution utilities $\left[Z _ i\left(\tau _ i, u _ i\right)\right] _ {i=1}^N$, and a joint state-action return distribution $Z _ {tot}(\boldsymbol{\tau _ {tot}}, \boldsymbol{u _ {tot}})$, if the following conditions are satisfied:

$$\arg \max _ {\mathbf{u}} \psi _ \alpha\left(Z _ {tot}(\boldsymbol{\tau _ {tot}}, \mathbf{u _ {tot}})\right) = \left[ \arg \max _ {u _ 1} \psi _ \alpha\left(Z _ 1\left(\tau _1, u _1\right)\right), \ldots, \arg \max _ {u _ N} \psi _ \alpha\left(Z _ N\left(\tau _N, u _ N\right)\right) \right]$$

where $\psi _ \alpha: Z \times R \rightarrow R$ is a risk metric such as the VaR or a distorted risk measure, $\alpha$ is its risk level. Then, $\left[Z _ i\left(\tau _ i, u _ i\right)\right] _ {i=1}^N$ satisfy the RIGM principle with risk metric $\psi _ \alpha$ for $Z _ {j t}$ under under $\tau$. We can state that $Z _ {tot}(\boldsymbol{\tau _ {tot}}, \boldsymbol{u _ {tot}})$ can be distributionally factorized by $\left[Z _ i\left(\tau _ i, u _ i\right)\right] _ {i=1}^N$ with risk metric $\psi _ \alpha$.

Generalization of the RIGM principle

Let's define probablities function $\pi _ {tot}(\mathbf{u _ {tot}} \mid \mathbf{\tau _ {tot}})$ and $\pi _ i(u _ i \mid \tau_i)$. $\pi_{tot}(\mathbf{u_{tot}} \mid \mathbf{\tau_{tot}}) = 1$, when $u_{tot} = \arg \max _{\mathbf{u}} \psi _ \alpha\left(Z _ {tot}(\mathbf{\tau _ {tot}}, \mathbf{u _ {tot}})\right)$, and it is $0$, otherwise. For $\pi _ i(u _ i \mid \tau _ i)$, $\pi _ i(u _ i \mid \tau _ i)=1$, when $u _ i = \arg \max _ {u _1} \psi _ \alpha\left(Z_i\left(\tau_i, u_i\right)\right)$, otherwise 0. The RIGM principle becomes the following formula.

$$\pi_{tot}(\mathbf{u_{tot}}\mid \mathbf{\tau_{tot}})=\prod_{i=1}^N \pi_i(u_i\mid \tau_i)$$

The same as the IGO case, we can use a diffusion model to generate risk-sensitive multi-agent action. Let's treat the generated data $x_i^0$ as risk-sensitive action $u_i$ of agent $i$. Then, the probability $p_{\theta_i}(x_i^0) = p_{\theta_i}(u_i)$ becomes $\pi_i(u_i\mid \tau_i)$, where $\pi_i$ is the risk-sensitive policy of agent $i$ and $\tau_i$ is the local observation. Further, let's treat the generated total data $\mathbf{x_{tot}^0}$ as $\mathbf{u_{tot}}$. The probability $p_{\theta_{tot}}(\mathbf{x_{tot}^0}) = p_{\mathbf{\theta_{tot}}}(\mathbf{u_{tot}})$ becomes $\pi_{tot}(\mathbf{u_{tot}}\mid \mathbf{\tau_{tot}})$, where $\pi_{tot}$ is the risk-sensitive policy of the multi-agent system, and $\tau_{tot}$ is the aggregated observations.

\prod_{i=1}^N p_{\theta_i}(x_i^0) = p_{\theta_{tot}}(x_{tot}^{0})$$$$\prod_{i=1}^N p_{\theta_i}(u_i) = p_{\mathbf{\theta_{tot}}}(\mathbf{u_{tot}})$$$$\prod_{i=1}^N \pi_i(u_i\mid \tau_i) = \pi_{tot}(\mathbf{u_{tot}}\mid \mathbf{\tau_{tot}})$$$$\pi_{tot}(\mathbf{u_{tot}}\mid \mathbf{\tau_{tot}}) = \prod_{i=1}^N \pi_i(u_i\mid \tau_i)

We have shown that the IGD principle is a generalization of the general RIGM principle. Thus, the IGD principle is a generalization of the RIGM principle.

Reference

[1] Tianhao Zhang, Yueheng Li, Chen Wang, Guangming Xie, and Zongqing Lu. FOP: factorizing optimal joint policy of maximum-entropy multi-agent reinforcement learning. In ICML, 2021

[2] Siqi Shen, Chennan Ma, Chao Li, Weiquan Liu, Yongquan Fu, Songzhu Mei, Xinwang Liu, and Cheng Wang. Riskq: Risk-sensitive multi-agent reinforcement learning value factorization. In NeurIPS, 2023.

or a component-wise independent Markov chain with a stationary initial distribution

Regarding independent assumption, the IGM principle has a hidden independent assumption: It assumes that each agent can independently make decisions without communicating with others. Below we give a detailed description:

The IGM principle requires that $(\bar{u_1}, ..., \bar{u_N}) = \bar{u_{tot}}$, where $\bar{u_i} = argmax Q_u(\tau_i, u)$, $\bar{u_{tot}}= argmaxQ_{tot}(\tau_{tot}, u)$. There is a hidden independent assumption of the IGM principle. It implicitly assumes that each agent can make the optimal decision independently based on individual utility function $Q_i(\tau_i, u_i)$, where $\tau_i$ is the local observation-action history for agent $i$. However, the individual utility function does not guarantee to optimal decision. To obtain the true optimal action, the utility function should be $Q_i(\tau_{tot}, u_i)$, where $\tau_{tot}$ is the observation-action history for all the agents. Such an independent assumption for the IGM principle may not hold for multi-agent tasks that require communication among agents. The IGD principle has the independent assumption (limitation) as the IGM principle.

Reference

[1] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." Advances in neural information processing systems 33 (2020): 6840-6851.

[2] Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In ICML, volume 162, pp. 9902–9915, 2022

[3] Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In ICLR 2023.

[4] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In ICLR, 2023

Thanks for your honesty about the unfamiliarity with the diffusion and the diffusion-based RL domain. Thanks for your time and effort to read our paper. It is a pity that you cannot read our work thoroughly due to the lack of domain-specific expertise. We are eager to discuss with you to obtain a quality review and fair judgment. We will try our best to make you familiar with the diffusion methods, diffusion-based RL, and diffusion-based MARL domain to gain a better understanding of our work. We hope that after reading our rebuttal and related literature, you can gain the expertise to give a quality and in-depth review.

In this rebuttal, we will give the necessary background information, address the questions, and show that the IGD is an extension of the IGM and RIGM principles.

I am unfamiliar with this domain of literature, and my comment would mostly focus on mathematical writing. Overall, I have strong concerns about the claim that IGD generalizes IGM, and I feel the theory about diffusion factorization framework under Gaussian assumption satisfying IGD uninteresting. Thus, I suggest rejection.

An in-depth understanding of our work requires the literature such as diffusion models, diffusion-based RL, and diffusion-based MARL. Although we have discussed background knowledge in the submission (and in the appendix), here we present more details of this background knowledge and point out related literature.

It seems that one of the major concerns of the reviewer is about Gaussian distribution. We use the Gaussian Distribution the same as the seminar DDPM work [1]. The Gaussian distribution is used both in the forward process and the backward process. It is used to model the noise used in diffusion and it is used for modeling the transition probability of adding noise into data and for modeling the backward transition probability of removing noise from the data. Using the Gaussion distribution to model the noise and the transitions are not invented by us, we take such an approach from DDPM

1. Background Information about Diffusion Models

The seminal work DDPM [1] uses diffusion to generate images. It consists of the forward process and the backward (reverse) process. The backward processes of DDPM and ours are Markov chains. The forward step and the backward step of the diffusion process are both defined as Gaussian Distribution. Reference [2] is a very nice tutorial about diffusion models; it shows the math after diffusion models in excruciating detail. Please have a look if you have any doubts.

In the forward diffusion process, data $x^k$ is polluted by adding noise to $x^{k-1}$ according following formula (Gaussian Distribution).

$p(x^k|x^{k-1}) = \mathcal{N}(\sqrt{\alpha^k} x^{k-1}, \sqrt{1 - \alpha^k}\boldsymbol{\epsilon^{k-1}})$

where $\epsilon^{k-1} \sim \mathcal{N}(0, **I**)$ is a Gaussian Noise, $k$ is the diffusion time step, $\alpha^k$ is a pre-specified hyper-parameter. $\mathcal{N}(\sqrt{\alpha^k} x^{k-1}, \sqrt{1 - \alpha^k}\boldsymbol{\epsilon^{k-1}})$ is a Gaussian Distribution with mean $\sqrt{\alpha^k} x^{k-1}$, and variance $\sqrt{1 - \alpha^k}\boldsymbol{\epsilon^{k-1}}$. In essense, $p(x^k|x^{k-1})$ is a probability, that the probability of sampling $x^k$ condition on $x^{k-1}$. Thanks to the property of Gaussian distribution, it can be written as $x^k = \sqrt{\alpha^k} x^{k-1} + \sqrt{1 - \alpha^k}\boldsymbol{\epsilon}^{k-1}$.

The backward diffusion process （denoising process）is a Markov chain. It is used to sample data based on Gaussian Distribution. The probability $p_{\boldsymbol{\theta}}(\boldsymbol{x}^{k-1}|\boldsymbol{x}^k)$ of generating $\boldsymbol{x}^{k-1}$ based on $\boldsymbol{x}^{k}$ is defined as a Gaussian distribution, which is defined in the following formula.

$p _ {\boldsymbol{\theta _ i}}(\boldsymbol{x}^{k-1}|\boldsymbol{x}^k) = \mathcal{N}(\boldsymbol{x}^{k-1}; \boldsymbol{\mu} _ {\boldsymbol{\theta}}(\boldsymbol{x}^k, k), \boldsymbol{\Sigma}(k))$, where $\boldsymbol{\mu} _ {\boldsymbol{\theta}}(\boldsymbol{x}^k, k)$ and $\boldsymbol{\Sigma}(k)$ are the mean and variance of the Gaussian distribution. They are trained to maximize the true data distribution.

We would like to point out that defining the transition probability of the Markov Chain (backward process) as Gaussion Distribution and modeling the noise as Gaussian Distribution is not invented or proposed by us. We just follow the approach of typical diffusion models such as DDPM.

2. Background Information about Diffusion-based RL

Besides using diffusion models to generate images or videos, researchers have applied diffusion models to generate decisions for offline RL. There are two representative ways for diffusion-RL: trajectory-based and policy-based methods.

Trajectory-based methods use diffusion to generate trajectories and plan based on the generated trajectory. The Diffuser [3] and Decision Diffuser [4] are the seminar work utilizing the diffuion model to generate trajectories. After the trajectories are generated, an inverse dynamic model is used to plan action based on the trajectories. This allows for more accurate long-horizon decision-making, especially in offline settings with limited data. They circumstance the need to estimate value functions, which are difficult to approximate in an offline-RL setting.

Policy-based methods use diffusion to generate policies (actions). DiffusionQL [5] is the first diffusion-based RL method that generates actions through diffusion. It makes use of the powerful modeling ability of diffusion to model policy/action distribution.

3. Background Information about Diffusion-based MADM (MARL)

Due to the promising performance and flexible modeling ability of diffusion-based RL, researchers have applied diffusion-based methods to offline multi-agent decision-making (MADM). These approaches can be categories into trajectory-based and policy-based methods. As far as we know, as discussed in the introduction, the related work, the experiments, and the appendix, the trajectory-based diffusion MADM method [6] suffers from scalability issues, and the policy-based diffusion MADM method [7] suffers from poor cooperation issues.

For the trajectory-based method, the first approach that uses a diffusion-based method for MADM is MADIFF [6]. It is built based on a decision diffuser, with the introduction of an attention-based mechanism to model the relationship among agents. Thus, it achieves promising performance. As we had discussed in the introduction, the related work, the experiments (Table 5), and the appendix (A.2), suffers from scalability issues. Our work enjoys better scalability than it (Table 5). Its performance can be further improved (Table 2, 3, 4). Moreover, we show that our work can be used to generate data beyond trajectories. Our work can be used to generate actions (DoF-policy), generate Q values (Table 1), and generate novel behaviors through flexible conditioning (Appendix, page 34, Figure 10).

The policy-based diffusion MADM methods, such as [7], ignore the relationship among agents. It uses either DiffusionQL to model each agent without considering their cooperation. We call these diffusion-based approaches, which do not consider cooperation, as independent diffusion. They lead to poor performances, as it is shown in Figure 1, and Figure 3.

In this work, we show that DoF can enjoy better scalability with better performance than existing diffusion-based MADM methods. Besides trajectories and actions, it can generate Q values. Moreover, it can generate novel behaviors through flexible conditioning.

Reference

[1] Ho, Jonathan, Ajay Jain, and Pieter Abbeel. "Denoising diffusion probabilistic models." Advances in neural information processing systems 33 (2020): 6840-6851.

[2] Calvin Luo. Understanding diffusion models: A unified perspective. CoRR, abs/2208.11970, 2022

[3] Michael Janner, Yilun Du, Joshua B. Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In ICML, volume 162, pp. 9902–9915, 2022

[4] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B. Tenenbaum, Tommi S. Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision making? In ICLR, 2023

[5] Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In ICLR 2023.

[6] Zhengbang Zhu, Minghuan Liu, Liyuan Mao, Bingyi Kang, Minkai Xu, Yong Yu, Stefano Ermon, and Weinan Zhang. MADIFF: Offline multi-agent learning with diffusion models. NeurIPS 2024

[7] Zhuoran Li, Ling Pan, and Longbo Huang. Beyond conservatism: Diffusion policies in offline multi-agent reinforcement learning. arXiv preprint arXiv:2307.01472, 2023.

The empirical results show their method perform slightly better than other methods.

The empirical results show that our method advance the state-of-the-art methods not just in terms of rewards, but also in terms of data generation ability, in terms of scalability.

Data Generation Ability

DoF demonstrates strong modeling capabilities by generating data-consistent distributions, as shown in Section 5.1. This capability is beyond the reach of traditional offline multi-agent reinforcement decision making approaches. Moreover, it generates data much better than other diffusion-based MADM methods.

DoF can be used to generate novel behaviors by flexible conditions. We conducted experiments in Appendix D.6. DoF successfully generated a complete circle, whereas its training data consisted of only half circles. Through using flexble conditions, DoF agents can collaborate to meet complex constraints and achieve novel behaviors.

Scalability for Diffusion-based MADM

As discussed in Appendix A.2, the input complexity of DoF is $O(1)$ with respect to the number of agents, whereas the input complexity of MADIFF [NeurIPS 24] is $O(N)$, where $N$ is the number of agents. We have demonstrated that the inference time of MADIFF can be 1.7 times than DoF in Table 5.

Rewards

SMAC and SMACv2 Experiments (Section 5.2.1, Tables 2 and 3): Our method, DoF, has achieved a 16% improvement over another diffusion-based method, MADIFF. DoF achieves a reward improvement of 0.8 over the second-best algorithm, which does not have flexible generation ability. MPE Experiments (Table 4): DoF outperforms the second-best algorithm by an average reward improvement of 6 and shows a 19% improvement compared to MADIFF.

These results collectively demonstrate that the improvement of DoF over existing approaches is substantial. Reviewer y4iS comments DoF outperforms competing methods and performs well with an increasing number of agents. Reviewer YHzy regards the performance of DoF as strong. We would like to discuss this point with you.

Does IGD property apply to the initial distribution $p _ {\theta _ {tot}}(x^{0} _ {tot})$ or the whole series $p _ {\theta _ {tot}}(x^{0:K} _ {tot})$? Does $x^K _ {tot}$ need to be sampled from Gaussian? Does the reverse process need to be a Markov chain?

1. The IGD property applies to neither $p _ {\theta _ {tot}}(x^{0} _ {tot})$ nor the whole series $p _ {\theta _ {tot}}(x^{0:K} _ {tot})$, it apply to both the generated data $[p _ {\theta _ {i}}(x^{0} _ {i})] _ {i=1}^N$ and $p _ {\theta _ {tot}}(x^{0} _ {tot})$ as it is written in Line 200 and 201 of the initial submission.

2. $x^K_{tot}$ must be sampled from Gaussian Noise. We had already written in Line 194 of the initial submission "Gaussian distribution starting as $p(x_{tot}^K) = \mathcal{N}(0, **I**) \in \mathcal{R}^{N \times d}$"

3. The reverse process is a Markov chain. We had already written in Line 193 of the initial submission "the reverse process, defined as a Markov chain ...".

If I understand it correctly, it says a joint distribution $p_{\theta_{tot}}(x^0_{tot})$ can be decomposed into $[p_{\theta_i}(x^0_{i})]^N_{i = 1}$. Note that without loss of generality, the IGD property only applies $p_{\theta_{tot}}(x^0_{tot})$ as

Thanks for reading our response. This understanding is close to the meaning of the IGD principle, but the generative diffusion model part of the IGD is missing.

The diffusion model is a type of generative model. It aims to model the true probability of data $q(x_{tot})$ through using $p_{\theta}(x_{tot}^0)$. $p_{\theta}(x_{tot}^0)$ is parameterized by $\theta$. The goal of the diffusion model is to maximize the likelihood of $p_{\theta}(x_{tot}^0)$. Once a model satisfies the IGD is trained, the data $x_{tot}^0$ can be sampled through the Markov Chain. IGD requires that sampled data $x_{tot}^0$ should match the true data distribution $q(x_{tot})$ well. It is described in Line 195-196 "$p_{\theta}(x_{tot}^0)$ learned to model ground truth distribution". This is achieved by requiring the generated data (or sampled data) $x_{tot}^0$ be generated by the diffusion model. Note that $x_{tot}^0$ is the generated data, and $x_{tot}$ is the true data we tried to model. They are not the same.

Besides requiring the generated data to match true data distribution, it requires factorization.

Is IGD just saying that each coordinate of $x^0_{tot} = (x^0_1, ... , x^0_N)$ is mutually independent?

Not, in line 197 of the initial submission, $x^0_i \in \mathcal{R}^d$ is $d$-dimensional instead of just one-dimensional. IGD requires that $x^0_{tot}$ should close to true data distribution by using the diffusion models, as described in lines 195-196 of the initial submission.

What is the point of introducing the whole series $x_{tot}^{0:K}$? Those Markovian and Gaussian assumptions seem to be related to diffusion model, while IGD is still defined under general random variable $x^0_{tot} = (x^0_1,..., x^0_N)$.

This diffusion-related concept is introduced because IGD is developed based on diffusion models. In line 204 of the initial submission, we had stated clearly that $p_{\theta}(x_{tot}^0)$ and $[p_{\theta_i}(x_i^0)]_{i=1}^N$ are diffusion models.

$x^0_{tot} = (x^0_1,..., x^0_N)$ are not general random variables; they are random variables that match closely with the true data distribution thanks to diffusion models.

Finally, if you really want to focus on diffusion models, can you just say a diffusion model has IGD if each coordinate of initial state is mutually independent.

Not, the IGD does not require that each coordinate (or each compenent) of the original data (state) is mutually independent. Instead, it requires that the generated data (rather than the true data) should match the true data distribution, and each component $x_i^0$ of the generated data $x_{tot}^0$ is mutually independent.

For diffusion models that each coordinate (component) of the initial state is mutually independent, described by the reviewer, is called independent diffusion. It holds the oversimplified assumption that each component of the true data (rather than the generated data) is mutually independent. In independent diffusion, for each component of the jointed data, a distinct independent diffusion model is trained. It performs poorly, as demonstrated by experiments and discussed in our submission. Please refer to Appendix A.2 for further discussion.

We agree that the data generated through satisfying the IGD principle may not fail to model the complex relationship among each component of the data. Thus, after data are generated, we use the data factorization function (proposed by us) to model the complex relationship among data.

We will make the introduction of the diffusion model and the definitions better by introducing all necessary math backgrounds.

Again, I am asking for formal statement showing that IGD generalizes IGM. Table 1 seems to be special cases.

We have presented the formal statement proofs in the initial response regarding Table 1 (we had also emphasized this point in our revised paper in Appendix E.1). Could you please read it? We are eager to hear your opinion.

I should clarify my point. In my view, the proof technique relies heavily on the assumption that the process is Gaussian. Since I am not particularly interested in diffusion processes, I find the approach lacks broader insight.

Thanks for reading the proofs. We understand the research flavor of the reviewer towards the diffusion processes is different from ours. However, we would like to point out the super powerful modeling ability of the diffusion models have already changed the world, especially in image and video generation. Our work wants to leverage such modeling ability into multi-agent decision making.

Dear Reviewer zeea,

Thanks for your response.

I assume you are referring to the IGO as the individual global max principle, which was referred to as IGM in your paper.

No, in lines 205 to 212 of the initial submission, we show that IGD is a generalization of the IGM principle by using the property that the probability of the argmax operator $p(u_i)=1$ when $u_i=argmax(\tau_i, u_i)$, it is 0, otherwise.

By the way, the IGO is shorted for the individual-global-optimal principle, which is a generalization of the individual-global-max (IGM) principle. The IGO principle requires that the product of the probability $\prod[p(u_i)]_{i=1}^N$ of each agent's optimal action is equal to the joint probability $p(u_1, u_2, ..., u_N)$ of the multi-agent system's optimal action.

IGM is a property of the state-action value function and does not depend on a diffusion model. How can IGD, defined under the diffusion model, be considered a generalization of IGM?

The diffusion model can be used to generate images, videos, actions, and state-action value functions. The IGM does not restrict the way how the state-action value function can be implemented or generated. We have shown that the Diffusion model can generate actions and state-action value functions in this work. We have shown that the diffusion models can generate four value functions in Table 1 and Table 11. Moreover, we have shown that the diffusion models can generate actions through multiple experiments. Other diffusion RL research , such as DiffusionQL [1] and LatentDiffusion [2], also confirmed this.

We have provided formal proof of how the IGD generalizes the IGM principle in the initial response. It is located in the same page as this reply, its title is "The IGD principle is a generalization of the IGM principle".

We understand that the reviewer is still confused about how IGD generalizes IGM.

May I ask which part of the proofs confuse you?

Are there any steps you think are not correct?

Are you unclear on how the diffusion model generates actions or state-action value functions?

We are eager to discuss these questions with you.

Best Regards,

Authors

Reference:

[1] Zhendong Wang, Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In ICLR 2023.

[2] Siddarth Venkatraman, Shivesh Khaitan, Ravi Tej Akella, John Dolan, Jeff Schneider, Glen Berseth. Reasoning with Latent Diffusion in Offline Reinforcement Learning. In ICLR 2024.

Dear Reviewer zeea,

Thanks for reading our responses. We are happy that the reviewer does not think there are any wrong steps in our proofs about how IGD generalizes IGM. We are happy that there is only one remaining concern. The reviewer is concerned about the implementation of the diffusion model regarding how to generate actions or value functions given an observation or history $\tau$. As we mentioned in the very first response, we use the standard diffusion approach. We describe them first in an intuitive way and then describe the math details. After introducing this background knowledge, we answer the reviewer's questions in the end.

Assuming that we have trained a diffusion model to generate images, we can sample a noise $x_{tot}^K$ from a Gaussian distribution. And then iteratively sample a new data $x_{tot}^{k-1}$ from $x_{tot}^{k}$ following a Gaussian Distribution. For this procedure, when sampling (generating) an image, we do not have control over which content to generate. If we want to control the content of an image through the class description (prompt) $y$ (e.g., dog, cat), there could be three ways to impose control:

1. Train a diffusion model for each possible $y$; this is very expensive.
2. Classifier-guidance diffusion: once a diffusion model is trained. Additionally, train a classifier $p(y|x_{tot})$, given an input image $x_{tot}$, the classifier outputs the class probability. The classifier will control the content it generates.
3. Classifier-free diffusion: It learns an unconditional diffusion model $p(x)$ and controls the content using condition score (a function of $y$).

Approach 1 is the approach mentioned by the reviewer that trains a separate model for each $\tau_{tot}$. This is expensive; we do not adopt such an approach. Researchers have found that using the classifier-free approach [1] can lead to better performance than the classifier-guidance approach. Thus, we adopt the classifier-free diffusion approach. We have described them in lines 315--319 and 1816--1817 of the initial submission.

Similarly, for the Individual-Global-Optimal (IGO) principle, we can use classifier-free diffusion to generate action following $\pi_{\theta_{tot}}(u_{tot} \mid \tau_{tot})$. Once a diffusion model is trained to generate actions. We can use a condition $\tau_{tot}$ to guide the diffusion process to generate the required action $u_{tot} \sim \pi_{\theta_{tot}}(u_{tot} \mid \tau_{tot})$.

Specifically, to generate actions using a diffusion model, we follow a similar process as we would for image generation, but instead of generating images, we generate actions $u_{tot}$ for multi-agent.

1. Forward Diffusion Process: We begin with an initial variable $x_{tot}^0 \sim q(u_{tot})$, where $q(u_{tot})$ is the true distribution of actions. data $x^k_{tot}$ is polluted by adding noise to $x^{k-1}_{tot}$ according following formula (Gaussian Distribution).

$p(x^k_{tot} \mid x_{tot}^{k-1}) = \mathcal{N}(\sqrt{\alpha^k} x^{k-1}_{tot}, \sqrt{1 - \alpha^k} \boldsymbol{\epsilon^{k-1}})$

where $\epsilon^{k-1} \sim \mathcal{N}(0, **I**)$ is a Gaussian Noise, $k$ is the diffusion time step, $\alpha^k$ is a pre-specified hyper-parameter. $\mathcal{N} (\sqrt{\alpha^k} x^{k-1} _ {tot}, \sqrt{1 - \alpha^k} \boldsymbol{\epsilon^{k-1}})$ is a Gaussian Distribution with mean $\sqrt{\alpha^k} x^{k-1}$, and variance $\sqrt{1 - \alpha^k} \boldsymbol{\epsilon^{k-1}}$. In essense, $p(x^k_{tot} \mid x^{k-1}_ {tot})$ is a probability, that the probability of sampling $x_{tot}^k$ condition on $x^{k-1}_{tot}$.

2. Reverse Diffusion Process (Recovering Action following true action distribution): The goal of the reverse diffusion process is to learn how to gradually denoise the latent variable $x_{\text{tot}}^k$ to recover the final action $u_{tot} = x_{\text{tot}}^0$. The probability p _ { \boldsymbol {\theta}}$$( \boldsymbol{x} _ {tot}^ {k-1} \mid \boldsymbol{x} _ {tot} ^ k) of generating $\boldsymbol{x}_ {tot}^{k-1}$ based on $\boldsymbol{x} _ {tot}^{k}$ is defined as a Gaussian distribution, which is defined in the following formula.

p _ {\boldsymbol{\theta_ {tot}}}(\boldsymbol{x} _ {tot}^{k-1}$$ \mid \boldsymbol{x} _ {tot}^k)$$ = \mathcal{N} (\boldsymbol{x}_ {tot}^{k-1}; \boldsymbol{ \mu}$$ _ {\boldsymbol{\theta _ {tot}}}$$( \boldsymbol{x}^k_{tot}, k), \boldsymbol{ \Sigma}(k)).

$\boldsymbol{\mu} _ {\theta _{tot}}(\boldsymbol{x} _{tot}^k, k)$ and $\boldsymbol{\Sigma}(k)$ are the mean and variance of the Gaussian distribution. They are trained to maximize the true data distribution.

3. Reverse Diffusion Process (Generating the Action for a specific $\tau_{tot}$): The reverse diffusion can be guided by the agent’s history $\tau_{\text{tot}}$ through classifier-free guidance to generate $u_{tot}$. In this way, the action $u_{tot} = \pi(\tau_{tot})$. The process follows the standard classifier-free approach [1]; to see the exact details of such a process, please refer to [1].

To generate the individual action $u_i$ for agent $i$, the IGD principle requires that the parameters $\theta_{tot}$ can be factored into $\theta_i$, which can be used to sample individual action $u_i$ by using the diffusion process parameterized by $\theta_i$.

1. Reverse Diffusion Process (Recovering Action following true action distribution): The goal of the reverse diffusion process is to learn how to gradually denoise the latent variable $x_{i}^k$ to recover the final action $u_{i} = x_{\text{i}}^0$. The probability p _ { \boldsymbol {\theta}}$$( \boldsymbol{x} _ {i}^ {k-1} \mid \boldsymbol{x} _{i}^k) of generating $\boldsymbol{x} _ {i}^{k-1}$ based on $\boldsymbol{x} _ {i}^{k}$ is defined as a Gaussian distribution, which is defined in the following formula.

p _ {\boldsymbol{\theta_ {i}}}(\boldsymbol{x} _ {i}^{k-1}$$ \mid \boldsymbol{x} _ {i}^k)$$ = \mathcal{N} (\boldsymbol{x}_ {i}^{k-1}; \boldsymbol{ \mu}$$ _ {\boldsymbol{\theta _ {i}}}$$( \boldsymbol{x}^k_{i}, k), \boldsymbol{ \Sigma}(k)).

$\boldsymbol{\mu} _ {\theta _{i}}(\boldsymbol{x} _{i}^k, k)$ and $\boldsymbol{\Sigma}(k)$ are the mean and variance of the Gaussian distribution. They are trained to maximize the true data distribution.

2. Reverse Diffusion Process (Generating the Action for a specific $\tau_{i}$ or $\tau_{tot}$): The reverse diffusion can be guided by the agent’s history $\tau_{\text{i}}$ or aggregated history $\tau_{tot}$ through classifier-free guidance to generate $u_{i}$. The process follows the standard classifier-free approach [1]; to see the exact details of such a process, please refer to [1]. In this work, we follow the Centralized Training with Decentralized Execution (CTDE) paradigm. Thus, we choose to generate action $u_i$ conditions on local observation $\tau_i$ rather than $\tau_{tot}$.

We hope that the reviewer can have a better understanding of how we can use diffusion models following the IGD principle to generate actions.

For instance, given a value function satisfies IGM what is the corresponding diffusion process?

The diffusion process is the same as the above process to generate actions with only one difference.

The IGD principle requires that $x_{tot} \in \mathcal{R}^{N \times d}$ and $x_i \in \mathcal{R}^d$. The dimension of $x_{tot}$ must be bigger than that of $x_i$. However, the value function $Q_{tot}$ and $Q_i$ are one-dimensional. For a $Q_{tot}(\tau_{tot})$ and $Q_i(\tau_i)$ that satisfy the IGM, a diffusion process $i$, parameterized by $\theta_i$, can be used to generated $Q_i(\tau_i)$ through condition on $\tau_i$. And then the aggregated results $[Q_i(\tau_i)] _ {i=0}^N$ is passed to the data factorization function $h$ (described in Sec.4.3.2) to obtained $Q_{tot}(\tau_{tot})$. The data factorization function $h$ can be any factorization that satisfies the IGM principle.

Given $Q_{tot}$, what is the transition matrix for $p_{\theta_{tot}}(x_{tot}^{k-1}|x_{tot}^k)$ for each $k$?

These transition matrices are Gaussian Distributions, which are described in the above text and in the very first response to the reviewer's review. We follow the DDPM approach. Regardless of the type of data to be generated, the transition matrixes are the same.

Do you use different diffusion process for different history $\tau_{tot}$?

No, we have described the reason in the above text.

REFERENCES

[1] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS, 2021

Thank you for the response. The above argument is not rigors. The existence of $\mu_{\theta_{tot}}$ in the reverse process is not proved, and $K$ is not specified. For instance, if the optimal action is deterministic so that $|argmax_u Q_{jt}(\tau_{tot}, u)| = 1$, it seems highly unlikely to perfectly denoise it from a Markov chain with $\mu_{\theta_{tot}}$ and initial state Gaussian $x_{tot}^K$. While it may be possible to approximate it, achieving exact recovery seems implausible.

This may be due to the limitations of the diffusion model itself, but based on this discussion, I am not convinced the statement is true.

We are glad to see that the reviewer has no further questions and appreciates the feedback provided. Given our detailed responses, we believe that some of the initial concerns may have stemmed from misunderstandings, particularly regarding the nature of the diffusion process, the conditions under which exact recovery is possible, and the implementation of diffusion processes for deterministic actions. We notice the reviewer rated our work as "Reject" and would like to respectfully ask if this rating was influenced by such misunderstandings. If clarity was a major issue, we have already addressed this in the responses. We kindly ask that you reconsider your evaluation in light of these improvements.

At the same time, we hope that through this process, the authors and the reviewer can contribute to the ICLR community by providing an example of how to offer thorough, constructive feedback and how to effectively respond to reviewers' concerns. We deeply value every piece of feedback in the review process and aim to further improve and advance the field.

Thank you for your in-depth emergency review, especially for providing thoughtful and constructive feedback under such a limited time. We greatly appreciate the effort you invested in carefully evaluating our work. Below, we address your main points:

While the ideas are interesting, presentation of the paper makes it difficult to fully appreciate the contribution. For example, the algorithms are missing from the main manuscript.

Thank you for pointing this out. We included the pseudocode for DoF-Trajectory and DoF-Policy in Appendix C.3 and C.4, respectively. In the revised version, we have revised Section 4.4 to enhance clarity by adding the sentence: "Please refer to the details of the agents in Appendix." The organization of this paper will be reorganized to describe the algorithms better.

Related to the previous point, it is unclear how to formulate a diffusion algorithm based on Theorem 1.

I was a bit confused by the overall setup, are the agents doing global training and then individual sampling of the noise after doing the factorization? Is there a way for the agents to train simultaneously?

During the training phase, we globally sample an overall noise vector. Each agent learns a noise predictor (Denoiser), and their outputs are combined through an noise factorization function $f$ to approximate this overall noise, enabling efficient collaborative training.

During the inference phase, the diffusion method supports each agent independently sampling and generating its own trajectory or action, ensuring the capability for decentralized execution while maintaining alignment with the design philosophy of the CTDE framework.

We have provided pseudocode specifically for the training phase and execution phase of the DoF diffusion model, which elaborates on how the diffusion process is implemented. The pseudocode below offers a step-by-step description of centralized training and decentralized execution (generation). The full pseudocodes of DoF-trajectory and DoF-policy are presented in Appendix C.3 and C.4.

Centralized Training

1: repeat:
2:     $\mathbf{x}^0_{\text{tot}} \sim q(\mathbf{x}_{\text{tot}})$ (sample global data)

3:     $k \sim \text{Uniform}(\{1, \dots, K\})$ ($k$ is the diffusion time step)

4:     $\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I}) \in \mathcal{R}^{d \times N}$ (sample global noise)

5:     $x^k_{tot}=\sqrt{\bar{\alpha}^k} x^{k-1}_{tot} + \sqrt{1 - \bar{\alpha}^k}\epsilon$

6:     $x^k_i=x^k_{tot}[(i-1) \times d: i\times d]\;$ $i \in [1,..N]$

7:     $\epsilon_{\text{tot}} = f(\epsilon_{\theta_1}^1(\mathbf{x}^k_1, k), \epsilon_{\theta_2}^2(\mathbf{x}^k_2, k), \dots, \epsilon_{\theta_N}^N(\mathbf{x}^k_N, k))$ ($f$ is the noise factorization function)

8:     Take gradient descent step on:     $\nabla_\theta \|\epsilon - \epsilon_{\text{tot}}\|^2$

9: until convergence.

$\epsilon_{\theta_i}^i(\mathbf{x}^k_i, k)$ is a noise predictor for agent $i$ that predicts the $(i-1) \times d$ to the $i \times d$ index of the noise $\epsilon$.

Decentralized Execution (generation)

1: $\mathbf{x}_i^K \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ (for each agent $i$)

2: for $k = K, \dots, 1$ do

3:    $\epsilon _ \theta^i(\mathbf{x}^k _ i, k)$ (noise prediction by each agent $i$)

4:    Update state for each agent $i$:
   $\mathbf{x}^{k-1} _ i = \frac{1}{\sqrt{\alpha_k}} \left(\mathbf{x}^k _ i - \frac{1 - \alpha_k}{\sqrt{1 - \bar{\alpha} _ k}} \epsilon _ \theta^i(\mathbf{x}^k _ i, k)\right) + \sigma _ k \mathbf{z},$    where $\mathbf{z} \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$ if $k > 1$, else $\mathbf{z} = \mathbf{0}$

5: end for

6: return $\mathbf{x}^0 _ i$ (final trajectory or action for each agent $i$)

We will add the above algorithms into the main text.

Some of the notations are not well defined, such as $\epsilon$ in the theorem, and what role $\theta$ play in Eq (2).

$\epsilon$ is a random noise. $\epsilon^{\theta_i}_i(\mathbf{x}^k_i, k)$ is the noise predictor of agent $i$ to predict the noise that added at time $k$ for the data $x_i$. The noise predictor $\epsilon^{\theta_i}_i(\mathbf{x}^k_i, k)$ is parameterized by $\theta_i$. In this work, we assume that $\theta_i \subset \theta$. $\theta$ is the aggregated parameters of all the noise predictors. We will describe them more clearly in the paper.

The POMDP setting should be described a bit more in the main text, for instance, there is one environment that all agents are interacting with and the transition probability depends on the product space of all agent's actions.

We appreciate this suggestion. We will expand the description of the POMDP setting in the main text, including details about shared environments, joint actions, and how agents interact with the system.