ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization

We thank the reviewer for the comments. Please find below our responses to your concerns.

Although AlberDICE is briefly mentioned in Section 2, the paper does not adequately discuss the theoretical or empirical advantages of adopting the value decomposition instead of the resampling approach. In addition, the AlberDICE paper presents a simple multi-agent task where value function decomposition leads to a substantial performance loss.

We thank the reviewer for the insightful comments, which we greatly value and have made every effort to address. Here we would like to clarify that the key distinction between our algorithm and AlberDICE lies in the learning approach: our method learns the occupancy ratio at the global level, utilizing centralized learning and decentralized execution. In contrast, AlberDICE focuses on learning individual occupancy ratios, which restricts its ability to capture the interconnections between local agents. We have added clarifications on Page 3 to emphasize these differences.

(1) A detailed discussion comparing ComaDICE and AlberDICE should be added, (2) AlberDICE should be included as a baseline in all experiments, (3) ComaDICE should be tested on the XOR game in the AlberDICE paper

In the updated version, we have included comparisons with AlberDICE and OptDICE and updated our experiments accordingly. Additionally, we tested ComaDICE on the XOR game, with the results reported in Section B8 of the appendix. These results demonstrate that, for this simple game, ComaDICE achieves a similar best policy value as AlberDICE.

In Sections 6.1 and 6.3, SMACv1 environment is discussed; however, corresponding experimental results for SMACv1 are not included in the main manuscript...

Thank you for the suggestion. We have now moved the results for SMACV1 to the main paper.

Do the authors assume that $s$ can be sufficiently represented from $o$? Notations for states and observation are confusingly used.

In the POMDP setting, global states are not fully accessible and are instead represented by the joint observations from local agents. For simplicity, we use global state notation, but it actually refers to the corresponding joint observations. We have added a clarification on Page 7.

Figure 2 lacks information on which task was selected for each benchmark. (e.g. 5_vs_5 or 10_vs_10 in protoss, or expert or medium data quality in Hopper) Could you clarify?

For this figure, we used all the tasks and calculated the average win rates or returns for comparison. We have added a sentence to clarify this point.

We hope our revisions and our response address the reviewers’ concerns and further clarify our contributions. If there are any additional questions or comments, we would be happy to address them.

We thank the reviewer for the comments. Please find below our responses to your concerns.

The experimental results for hybrid networks contradict mainstream research findings (single-layer outperforms double-layers) ...

The low performance of the 2-layer mixing network clearly indicates that, in our offline setting, the 2-layer structure is overly complex for capturing the interdependencies between agents, leading to overfitting. Increasing the amount of offline data might reveal more information and potentially improve the performance of the 2-layer mixing network. However, this is not practical, as it would require significantly more storage and make training computationally expensive and infeasible. We have added an additional discussion to Section B.6 in the appendix to clarify this point. If our explanation remains unclear or if you have any further questions, we would be happy to provide additional clarification.

The discussion of the DICE method and its goals in the Related Work and Preliminary Knowledge section is not sufficiently in-depth to clearly locate the innovations of this paper with respect to existing work.

In the updated version, we have made every effort to position our contributions in relation to existing works and techniques. The updates have been highlighted in magenta for clarity. If the reviewer has specific points that require more in-depth discussion or analysis, we would be happy to explore them further and provide additional clarification.

The proof section of DICE skips some key steps, has a confusingly organized derivation and notation that does not clearly state the purpose and rationale for each step of mathematical transformation. ...

Due to space constraints, we had to omit some steps in our derivations and instead referenced prior work on which our approach builds. Specifically, the derivations up to Proposition 4.1 closely follow those in the OptDICE paper for single-agent RL. To clarify this, we have added some lines after Proposition 4.1 to acknowledge this fact.

The paper fails to explore the common sparse reward scenario in offline reinforcement learning, and the analysis of the quality requirements of behavioral strategies is insufficient, limiting the feasibility of the method in practical applications.

We acknowledge that our work did not address the sparse reward scenario or the quality of the behavioral policy. These are indeed important aspects; however, tackling these issues would require significant additional effort, which we believe warrants a separate paper. We will acknowledge this as a limitation of our work and highlight it as a promising direction for future research.

We hope our revisions and our response address the reviewers’ concerns and further clarify our contributions. If there are any additional questions or comments, we would be happy to address them

However, I have some observations regarding the formulation section of ComaDICE. The current presentation lacks a professional and cohesive mathematical structure. As it stands, the section appears more like a draft or homework exercise rather than a polished and rigorous exposition expected in a professional context. I recommend reorganizing this section to present the derivations and explanations with greater clarity and structure, adhering to standard mathematical writing conventions.

We sincerely thank the reviewer for their valuable feedback, which we greatly appreciate. Regarding the exposition, we have made every effort to present our formulation clearly. However, due to space constraints, some details have been omitted, and we have provided references to relevant papers for further information. If the reviewer could kindly point out specific parts of the section that require revision or improvement, we would be more than happy to address them and make the necessary revisions to enhance the clarity and quality of our work.

Additionally, the notations in this section require refinement for consistency and precision. For instance, the notation for the occupancy measure ...

We thank the reviewer for their valuable suggestion, which we have carefully considered. In response, we have now updated the notation to ensure consistency throughout the paper. Specifically, we now use global state notation consistently, with a clarification that, in practice, global state information is not fully available. Instead, we rely on joint observations to address this limitation.

Once again, we sincerely appreciate the reviewer’s additional feedback, which has helped us further improve the paper. We hope our responses and updates adequately address your remaining concerns.

Should you have any further questions or require additional clarification, we would be more than happy to provide further explanations or revisions.

We thank the reviewer for the comments. Please find below our responses to your concerns.

A very optimistic reading of the paper would interpret that the decomposition procedure is actually learning optimal value functions over decentralized policies. ... However, it is worth noting that this is more of a statement of the potential of the paper rather than a strength of its current version, as these points are not specifically addressed in the current draft. Furthermore, ComaDICE can be more scalable in comparison to AlberDICE [1] which requires alternating optimization.

In this paper, we propose a method to learn a globally optimal state-action occupancy by integrating the DICE framework with decentralized learning principles. While the DICE framework provides an effective approach for addressing out-of-distribution (OOD) issues, as demonstrated in prior DICE-based offline RL studies, the decentralized learning principle ensures that our algorithm remains scalable and efficient. Compared to AlberDICE, in addition to being more scalable, as you mentioned, our algorithm is better in capturing the interdependencies between local agents and handling credit assignment across agents through the use of a mixing architecture.

It is not clear what the main problem is that ComaDICE is solving. ....

Our algorithm addresses the OOD issue in a manner similar to other DICE-based offline RL algorithms, such as OptDICE and AlberDICE. Specifically, alongside maximizing the global reward, we incorporate a divergence term into the objective function to ensure the learned policy remains close to the behavior policy. The main distinction between our algorithm and AlberDICE lies in the learning approach: our method learns the occupancy ratio at the global level, leveraging centralized learning and decentralized execution. In contrast, AlberDICE focuses on learning individual occupancy ratios, which limits its ability to capture the interconnections between local agents. We have added clarifying details on Page 3 to highlight these differences.

The derivation is based on OptiDICE [2] but this is not explicitly mentioned, ... Furthermore, the final algorithm closely resembles OMIGA [3].

We would like to note that while OptiDICE is designed for the single-agent setting, our work extends its DICE-based approach to the multi-agent setting, which requires an extensive analysis on the connection between local policies and the global objective. Theoretically, we have included a closed-form expression for the first-order derivative of the objective function in our Proposition 4.1, which we believe is not available in the OptiDICE paper. To clarify this, we have added a detailed explanation after our proposition on Page 5.

Additionally, we have included discussions on Page 3 to highlight the differences between our algorithm, AlberDICE, and OMIGA. Furthermore, new numerical experiments have been provided to compare our algorithm with AlberDICE and OptiDICE.

What is the main purpose for the value decomposition and learning individual Q-functions?

The primary motivation for decomposing the Q-function is to enable the decomposition of the advantage function, which is a key component of our objective. In our multi-agent setting, local rewards are unavailable, making it challenging to directly compute local advantage functions. To address this, we learn local Q-functions and use them to derive the local advantage functions. Simultaneously, the mean squared error (MSE) is optimized to ensure that the global Q-function and state-value function align well with the global rewards. An explanation has been added to Page 7 to clarify this point.

AlberDICE and OptiDICE are missing as the main baselines.

We have included OptiDICE and AlberDICE for comparison and updated our experiments.

What is the purpose of Theorem 4.2, ..

Our theorem demonstrates that the loss function is concave with respect to νν when any mixing network with non-negative weights and convex activations is employed. This guarantees that the training process will remain stable when optimizing over $\nu$. Consequently, under our mixing architecture, minimizing the loss with respect to $\nu$ ensures that the global optimum will be achieved.

Can ComaDICE solve the XOR game and Bridge mentioned in AlberDICE?

Yes, ComaDICE is capable. In fact, these games are relatively small, with significantly lower state and action dimensions, allowing our algorithm to quickly converge to the optimal policy within just a few training epochs. For the XOR game, we observed that ComaDICE not only achieves the maximum possible reward of 100 but also converges much faster than AlberDICE. We have added such results to Section B.7. in appendix.

Related to “Strengths”, can solving ComaDICE learn the global optimum in the underlying MDP even with the mixing network?

Yes, under our mixing architecture, ComaDICE can theoretically learn the global optimum for $\nu_i$ (for all $i$), as guaranteed by the convexity with respect to $\nu$ established in Theorem 4.2.

Is the Individual Global Max (IGM) assumption required in order to introduce the mixing network? In other words, does ComaDICE assume that the underlying optimal Q function assume IGM?

We do not assume IGM in our approach. Instead, we demonstrate that our learning framework guarantees consistency between the global and local optimal policies, as shown in Proposition 4.3—a property that shares similarities with IGM. Our mixing network architecture is primarily designed to ensure the convexity of the loss function, enabling a stable and efficient training process.

What is the main difference in the final algorithm with OMIGA [3] ?

The fundamental difference is that OMIGA directly produces Q or V functions, which can then be used to extract a policy. In contrast, ComaDICE uses the Q and $\nu$ functions solely to learn the occupancy ratio between the learning policy and the behavioral policy. We have added a discussion on Page 3 to clarify this distinction.

Why are the notations mixed between using joint observations and global states

In the POMDP setting, global states are not fully accessible and are instead represented by the joint observations from local agents. For simplicity, we use global state notation, but it actually refers to the corresponding joint observations. We have added a clarification on Page 4 at the beginning of Section 4.1.

Please write in a different color during the rebuttal which part is the novel part. In particular, which part of the proofs are a novel contribution and which part is coming from OptiDICE [2]

Thank you for the suggestion. We have added some lines to the proof of our Proposition 4.1, acknowledging that the first part of the proof is a straightforward extension from the OptDICE paper, while the second part introduces some novel findings.

We hope our revisions effectively respond to the reviewer’s critiques and provide greater clarity on our contributions. If you have any additional questions or comments, we would be happy to address them.

I'd like to thank all of the authors for the hard work during the rebuttal, especially in providing extensive experiments with additional baselines and extra toy example (XOR Game) results. It seems pretty clear that ComaDICE performs well across a variety of environments and outperforms baselines, including AlberDICE and OptiDICE. I believe these new results strengthen the paper.

However, I still have some remaining concerns and questions so I hope you can clarify them:

Compared to AlberDICE, in addition to being more scalable, as you mentioned, our algorithm is better in capturing the interdependencies between local agents and handling credit assignment across agents through the use of a mixing architecture.

In contrast, AlberDICE focuses on learning individual occupancy ratios, which limits its ability to capture the interconnections between local agents.

While AlberDICE does learn individual occupancy ratios, they are solving a reduced MDP which also considers the other agents' current policy and their training procedure also falls under CTDE.

In detail, what are the core differences which allow ComaDICE to learn interconnections between local agents and handling credit assignment? It would be helpful if you can write a detailed comparison using both formulations.

Also on a related note, I would suggest emphasizing the ability for ComaDICE to learn a factorized policy which is a global optimum. This would also clarify the strength of ComaDICE over AlberDICE (which learns Nash policies).

Yes, under our mixing architecture, ComaDICE can theoretically learn the global optimum for $\nu_i$ (for all $i$), as guaranteed by the convexity with respect to established in Theorem 4.2.

However, I still have concerns whether it does indeed a globally optimum $\nu$. Theorem 4.2 is used to claim that $\mathcal{\tilde L}$ is convex in $\nu$, but does solving this lead to the same $\nu$ as in the original problem without value decomposition? My understanding is that this is not the case, and either an assumption (e.g. IGM) or a separate theorem is required.

Finally, regarding notation, I think it would be much simpler if states are used throughout the paper rather than joint observations. The main reason is that, as far as I understand, the derivations and the proof actually require the global state rather than joint observations (which are not equivalent to the state). Of course, partial observability can be introduced in the practical algorithm in Section 5.

We thank the reviewer for the prompt additional feedback, which we highly appreciate.

Can you confirm whether the following statement in my original review is true? ...

We believe your observations align well with the high-level perspective of our method. To clarify further, our approach involves learning a global policy in the form of an occupancy measure by decomposing the Lagrange multiplier in the DICE framework, i.e., $\nu$, which can be interpreted as a value function. Subsequently, our policy extraction step focuses on learning decentralized policies derived from the outcome of the DICE approach, specifically $w^{tot}$.

If we view the goal of MARL as $\max L(\pi^{tot})$ , where is the space of factorized policies, is this what the objective of $\max \widetilde{L}$ (Line 262) is doing?

Thank you for the question. To clarify, the objective of $\min \widetilde{L}$ is also to find an optimal $\pi^{tot}$, but it does so within the space of occupancy measures. Here, instead of factorizing the global policy directly, we factorize the Lagrange multiplier $\nu$. As a result, $\min \widetilde{L}$ does not output a globally optimal policy directly, but rather the occupancy ratio between the optimal policy and the behavior policy. The optimal policy is then extracted from this occupancy ratio through weighted behavior cloning (BC), where we propose learning decentralized policies that match with the resulting occupancy ratio.

I am having trouble understanding clearly how ComaDICE is able to do this, and solve the XOR Game for example. It would also be helpful if you can provide what the converged and values (both global and individual values) were for the XOR Game for better understanding.

We thank the reviewer for the question. We believe that, for the XOR game, our algorithm operates in a manner similar to AlberDICE. Both approaches aim to learn policies that maximize the reward while aligning with the behavioral policies. However, whereas AlberDICE performs this alignment at the level of individual agents, our approach operates at the global level.

For the XOR game, we have all numerical results demonstrating how ComaDICE achieves the optimal values reported in the paper. However, let us explain intuitively how ComaDICE solves the XOR game. Consider the dataset {AB}, where this observation yields a high reward (i.e., 1). When ComaDICE solves $\min \mathcal{L}$, it seeks a policy that maximizes the reward across the dataset while aligning with the behavioral policy represented by the dataset {AB}. Consequently, it will return a global optimal policy (in the form of an occupancy ratio) that assigns the highest possible probabilities to the joint actions {AB}. Subsequently, our weighted behavior cloning (BC) step learns decentralized policies that also assign the highest possible probabilities to the joint actions {AB}, returning the desired optimal policy observed in our experiments. The same reasoning applies to other datasets, such as {AA, AB, BA}, ensuring that ComaDICE learns the correct optimal policies across different scenarios.

We hope our responses address your concerns. If the reviewer would like a more detailed explanation of the policy values returned by our algorithm for the XOR game, we would be happy to provide additional clarifications and discussions.

Regarding whether $\mathcal{\tilde L}$ is learning the optimal policy over factorized policies, I understand the optimization is over occupancy measures. I am asking whether the implicitly learned optimal policy is over factorized policies (before policy extraction). For instance, perhaps it is possible to say that since $\rho (s, a) = \rho(s) \pi(a|s)$ where $\rho(s)$ is the state marginal, and we want $\pi(a|s)$ to have a factorized form, factorizing the value is actually learning an optimal stationary distribution over factorized policies. This is not exact and just a general idea but something similar would help understand intuitively what value factorization is doing here.

Regarding the Matrix Game, can you explain intuitively how ComaDICE is able to solve the XOR Game using the ${AB, BA}$ dataset? For instance looking at Table 26, it seems like $w(AB)$ would converge to some positive value while $w(BA) = 0$ in order to deterministically choose AB. How is ComaDICE able to do this despite both AB and BA being in the dataset?

We thank the reviewer for the comments. Please find below our responses to your concerns.

ComaDICE’s theoretical analysis relies on non-negative weights and convex activations in the mixing network for stability. While this aids convergence, it may limit the model’s ability to capture complex inter-agent dynamics...

We would like to emphasize that non-negative weights and convex activations have been widely used in prior MARL algorithms with mixing architectures, such as QMIX, QTRAN. Our choice to adopt this setting is driven by its crucial role for achieving strong algorithmic performance that was experimentally demonstrated in previous studies. For instance, [1] shows that mixing networks with negative weights or non-convex activations result in significantly worse performance. We added some lines on Page 6 to clarify this point.

[1] The Viet Bui, Tien Mai, and Thanh Hong Nguyen. Inverse factorized q-learning for cooperative multi-agent imitation learning. Advances in Neural Information Processing Systems, 38, 2024.

High Computational Cost: ComaDICE incurs a significant computational cost ...

In our algorithm, the use of f-divergence is particularly advantageous. Its closed-form formulation and convexity enable a closed-form solution for the inner maximization problem, allowing us to reformulate the min-max problem into a non-adversarial one. This significantly enhances the training process. Importantly, our experiments clearly demonstrate that the algorithm is highly scalable and efficient, even for SMACv2, which is widely regarded as one of the most high-dimensional and challenging MARL benchmarks.

The ablation study shows that performance decreases when using a more complex network. Why is that the case? If more data were available or the model was improved, could the results potentially differ?

The low performance of the 2-layer mixing network shows that, in our offline setting, the 2-layer structure is overly complex for capturing the interdependencies between agents, leading to overfitting. Increasing the amount of offline data might reveal more information and potentially improve the performance of the 2-layer mixing network. However, this is not practical, as it would require significantly more storage and make training computationally expensive and infeasible.

We have added an additional discussion to Section B.6 in the appendix to clarify this point. If our explanation remains unclear or if you have any further questions, we would be happy to provide additional clarification.

We hope our revisions address the reviewers’ critiques and further clarify our contributions. If there are any additional questions or comments, we would be happy to address them.