Inverse Factorized Soft Q-Learning for Cooperative Multi-agent Imitation Learning

How do the agent have access to the global state information. If this is the case, why does the paper even define observations? Is the global state information available only in training or after deployment, too? In what settings is this applicable?

Thank you for the question! We would like to clarify that our model does not assume access to global state information of the entire environment. As briefly described in Section 4.2.1, each agent in our MARL setting only has local observations of other agents (enemies or allies) in the agent’s neighborhood. The state information notion S in our MIFQ model is simply a combination of these local observations of our agents. We will make this distinction clearer in the revised version of the paper. We note that this setting of local observations is standard in many previous MARL studies. Such local observations are available in both training and deployment and, we believe, are highly realistic in practical applications.

How could one adapt this algorithm for non-cooperative settings? Is there a straightforward way or does it require completely new approaches?

We thank the reviewer for the question. Theoretically, our approach can be applied to non-cooperative settings. However, in practice, it would require a completely new algorithm. With conflicting rewards, the non-cooperative setting is much more challenging to train compared to the cooperative setting. We will definitely explore this in future work.

We also thank the reviewer for pointing out the typos, which we highly appreciate and will correct.

We hope that the above responses address your concerns. If you have any other comments or concerns, we are more than happy to address them.

We thank the reviewer for carefully reading our paper and providing us with valuable questions and suggestions.

I wonder if the authors could discuss whether a simple state-only extension of IQ Learn ...

Our argument in lines 143-148 simply means that directly using the global Q, V, and global state would be impractical in multi-agent settings. This is not a limitation of IQ-Learn but a well-known challenge when extending single-agent models to multi-agent settings. This is also why the centralized training decentralized execution (CTDE) approach has become appealing for MARL. The state-only approach in Section 5.4 of the IQ-Learn paper is only useful when actions are not available. Applying this in our context is unsuitable because action observations are available.

Can the authors more explicitly discuss what the shortcomings of an independent version of IQ-learn ...

If we learn the local Q independently by solving (4) for each agent, it implies that we neglect the interactions between agents. This approach ensures convergence to individual local policies but does not maintain consistency between local and global policies, as required by well-known principles such as IGO and IGM, which are necessary for a successful MARL algorithm.

The original IQ learn paper plots the rewards to validate that their method recovers the ground truth reward. Can the same be done here? 

Visualizing rewards in multi-agent settings is much more challenging compared to the single-agent setting due to the vast joint state and action space. So far, we are unsure how to obtain meaningful visualizations for rewards in multi-agent tasks. Therefore, we will keep this for future investigation.

Why does MIFQ perform worse than BC on MPE, particularly the reference and spread tasks?

As mentioned in the paper, MPEs are deterministic environments (i.e., no dynamics), and BC typically performs well on such deterministic tasks.

What is the number of trials for each of the results?

We briefly mentioned these numbers in Table 2 of the appendix. Each number in Table 1 is computed based on 4 seeds and 32 evaluation runs per seed. We will add this information to the main paper.

What is the number of demonstrations used to train each of the methods? What does the shaded region mean? ...

The number of trajectories is 128 for MPEs and 4096 for Miner and SMAC-v2. We will clarify this in the caption of Figure 2. Additionally, the reviewer's point regarding the 95% confidence interval is well taken. We will compute these and update the paper accordingly.

IGC is used in line 192, but is only explained in the following Section 4.2.2. Definition 4.2 - this definition is not specific enough to be useful. ...

We appreciate the reviewer for pointing these out. We will remove the mention of IGC in line 192. In Definition 4.2, equivalence means that the joint policy is equal to the product of local policies. We will clarify this.

What are some reasons why MIFQ does not achieve expert level performance? ...

This was stated as a limitation of our approach (and other existing multi-agent IL algorithms as well). Multi-agent tasks are much more complex, making it difficult to recover the expert policy. Increasing the amount of expert demonstrations might help, but it also leads to an excessively large replay buffer, causing out-of-memory issues. Addressing this limitation will require further efforts, which we plan to pursue in future work.

How does the method perform with demonstrations not sourced from MAPPO ? ...  

In our context, MAPPO achieves the best policy in MARL, so we use it as an expert. The main reason is that it is not reasonable to use a sub-optimal policy as an expert for imitation, as a sub-optimal solution to the imitation learning problem could yield better rewards than the expert, thus biasing the evaluation.

Why does the method need an order of magnitude more demonstrations than IQ Learn needs on complex single-agent tasks?

The main reason is that multi-agent tasks are much more complex than single-agent tasks, with much larger action and state spaces. Therefore, much more data is needed to understand the environment, requiring significantly more demonstrations for the imitation learning.

Why is it necessary to maintain Q and V networks separately? Why not derive the global V function by computing the softmax of the Q functions as described in line 163-164? 

We have discussed this in Section B.4 of the appendix. The main reason for our approach is to make the algorithm practical. Directly computing $V$ through the global $Q$ is generally impractical because it requires sampling over a joint action space, which is exponentially large. We actually attempted this approach, but it did not work at all—the algorithm couldn't learn anything, and the win rates were always zero. Therefore, we did not include this approach in the comparison. In our approach, we compute the global $V$ using local $Q$ values (which only require sampling over the local action space, making it much more feasible) and then aggregate the global $V$ using the mixing network.

Why is it necessary to compute Q^tot via Q^tot = -M (-Q)?

The main reason for this double negation is not only for the monotonicity, as $Q^{tot} = M(Q)$ is sufficient for that purpose. We use this approach to ensure that the global objective function is concave in $Q$ (Theorem 4.5),

Did the authors consider continuous action space settings?

So far, our algorithm is generally not suitable for continuous action spaces. However, all the environments under consideration have discrete action spaces and are taken from prior SOTA MARL works. Extending the approach to continuous action spaces would require further investigation. We plan to explore this in future work.

We hope that the above responses address your concerns. If you have any other comments or concerns, we are more than happy to address them

We thank the reviewer for reading our paper and for the positive feedback.

In Figure 2, the semi-transparent curves are not standardly explained. If these do not represent standard deviations, what statistical measure do they depict?

They present standard deviations. We will clarify this in the paper.

Minor Error: On Line 62, the term "generation" is used where "generalization" might be intended. Could the authors clarify or correct this in the context?

This is “generalization” and it is a typo. We will correct this.

We hope that the above responses address your concerns. If you have any other comments or concerns, we are more than happy to address them.

We thank the reviewer for reading our paper and the insightful comments and questions.

The paper's organization could be improved.

Thank you for the feedback! We will revise our writing and improve our exposition.

The similarity between IGC and IGO[1] requires further clarification.

Thank you for mentioning this. IGO and our IGC are indeed similar. However, IGC is more specific in that it requires the local policies, obtained by solving the local objective functions, to be equivalent to the global policy obtained by the global objective function.

The objective function (6) introduces sub-optimality compared to the original objective (3) due to the restriction that  and  must be monotonic. Additionally, since  and  use different mixing networks, the relationship between them violates Equation (2). This indicates that Equation (6) does not represent the same objective as Equation (3), even without considering the sub-optimality introduced by factorization. These issues need further theoretical exploration and discussion.

Thank you for the insightful comments. We agree that the factorized Q-learning objective violates Eq. (3). The main reason we follow this approach is that the relationship between $V$ and $Q$ cannot simultaneously hold at both the global and local levels, i.e., (2) and (5) cannot hold simultaneously (we discussed this in Section B2 in appendix). On the other hand, maintaining the relationship as in (2) is impractical because it requires computing $V^{tot}$ via a global Q-function and global policy $\Pi$. Therefore, we choose to keep (5) valid and build our factorization approach based on this (Computing local V functions via (5) , and compute $Q^{tot}$ and $V^{tot}$ by the mixing networks is indeed less challenging and more practical)

Furthermore, since (2) holds, each individual objective helps match the individual learning policy with the corresponding individual expert agent. Since IGC holds, our training can ensure global convergence and consistency across all agents.

Although the experimental results are promising, the superior performance seems to stem from the QMIX algorithm's advantage over other MARL algorithms. An important missing baseline is the soft actor-critic version of IQ-Learn, which uses a centralized Q function with decentralized critics and does not seem to violate the original objective.

Thank you for the comment and suggestion. There are two reasons we did not extend the soft actor-critic version of IQ-Learn to our multi-agent setting. First, as mentioned, it requires computing the global V function via the global Q function and global $\Pi$, which is impractical for multi-agent settings. Second, soft-actor-critic (SAC) methods only work well for continuous-action-space environments. All the environments we considered (following prior SOTA MARL papers) have discrete action spaces, making direct Q-learning algorithms more suitable. To support this argument, we have conducted an additional experiment, detailed in the attached 1-page PDF, where we compare a SAC IQ-learn adapted to multi-agent tasks. The results generally show that SAC-IQ performs worse than our algorithm, MIFQ. We will include these results in the paper.

Is BC trained online, given that it shows learning curves with environment steps? If so, why not use DAGGER?

Our BC was trained offline. The learning curves actually reflect our evaluations after certain training steps. We will clarify this in our paper.

Could the authors explain why MIFQ significantly outperforms IQVND? Is it solely due to the factorization structure?

Yes, IQVDN is simply a linear combination of local functions, while MIFQ leverages our two-layer mixing networks with learnable parameters. Previous work has also shown that QMIX generally outperforms VDN for this same reason.

Why does the paper state that QPLEX is unsuitable for the proposed method? QPLEX also has .

The monotonicity of the Q function is simply a corollary of our mixing structure and is not a key target when constructing our learning objective. Additionally, while QPLEX utilizes the advantage function (A = Q - V), our objective is different and such an advantage function is unsuitable to use. We will elaborate more on this point in the updated paper.

We hope that the above responses address your concerns. If you have any other comments or concerns, we are more than happy to address them.