Multi-Agent Imitation by Learning and Sampling from Factorized Soft Q-Function

We sincerely thank you for your supportive and insightful feedback on our work. Please find our detailed responses to your concerns and questions below.

I think the paper could better discuss the trade-offs of using value decomposition...discussion of when the factorization assumption might break down... Could you elaborate on the limitations of value decomposition in high-coordination tasks?

The Value Decomposition (VD) paradigm, including seminal works like VDN and QMIX, represents a mainstream and highly successful approach in cooperative multi-agent reinforcement learning. Its prominence is rooted in several key strengths that effectively address core challenges in MARL:

First and foremost, VD methods elegantly implement the Centralized Training for Decentralized Execution (CTDE) framework. This allows them to leverage global information during the training phase to solve the difficult credit assignment problem, while producing highly efficient, decentralized policies for execution where each agent acts based on its local observations. This contributes significantly to their scalability, avoiding the exponential complexity of the joint action space.

Furthermore, methods like QMIX introduce a crucial inductive bias through the Individual-Global-Max (IGM) principle. This constraint, which enforces a monotonic relationship between individual agent utilities and the global team value, is not merely a limitation. Rather, it is a powerful mechanism that guarantees consistency between local and global optimal actions, greatly simplifying policy extraction and promoting stable learning. The strong empirical performance of these methods across numerous benchmarks validates the effectiveness of this approach.

While it is well-recognized that the structural assumptions of VD methods (such as IGM) define their application scope—potentially limiting their expressiveness on tasks requiring highly complex, non-monotonic agent coordination—this is often viewed as a strategic trade-off between expressiveness and learnability.

It is precisely because the value decomposition paradigm provides such a solid and effective foundation that our work builds upon it. We believe that exploring advancements within this powerful framework is a crucial and promising direction for pushing the frontiers of multi-agent coordination.

While I appreciate that MAFIS is positioned within imitation learning, I believe it should be evaluated against recent offline MARL methods such as OMAR and ICQ-MA, which are also designed to learn from fixed online datasets withou exploration...Why was there no comparison to recent offline MARL methods such as OMAR or ICQ-MA?

Although offline MARL methods like OMAR and ICQ-MA also learn from offline datasets, the offline dataset considered in offline MARL is fundamentally different from the one we examine. Specifically, our offline dataset is of expert quality but contains only state-action transitions without reward signals. In contrast, offline MARL does not require expert-quality data but necessitates the inclusion of reward signals. As a result, the two prominent offline MARL methods, OMAR and ICQ-MA, cannot operate on our expert dataset (due to the lack of rewards). Additionally, during the training process, offline MARL methods do not allow agents to interact with the environment, whereas we consider not only offline learning but also online learning, where agents can learn to collaborate through online interactions (though they still cannot obtain reward signals). Hence, we did not include comparisons with them in our study.

How sensitive is MAFIS to the number and quality of expert trajectories?

With more data, MAFIS can perform better. The performance of MAFIS under different amount of expert data is shown in the following table.

Like common imitation learning algorithms such as BC, MAGAIL, etc., MAFIS currently requires high-quality expert data. However, exploring how to perform imitation learning with suboptimal expert data is an interesting direction that we will investigate in future work.

Thank you for your positive assessment of our work. Please find our detailed responses to your concerns below.

The sensitivity to the amount of expert data is not explored.

With more data, MAFIS can perform better. The performance of MAFIS under different amount of expert data is shown in the following table.

The computational overhead compared to baselines is not discussed.

Thanks to community-optimized deep learning packages like PyTorch and TensorFlow, as well as engineering techniques such as Just-in-Time (JIT) compilation, sampling an action via SGLD can be completed within milliseconds on widely used commercial GPUs (e.g., NVIDIA's A100 or 4090). Both MAFIS and baselines (like MAGAIL) can finish training within a few hours on a single GPU.

While Appendix B attempts to clarify differences from MIFQ, the distinction could be clearer in the main text.

Thank you for your advice. Due to space limitations, we have to put the full description of the differences in Appendix B and present a concise discussion of the differences between MIFQ and our approach at the end of Section 3.1. We will continue to optimize the article layout in the future.

MAFIS creates a fundamental mismatch between training and execution policies.

We argue that it is not an issue. Inspired by QMIX, we introduced $k^i(s)$ into the joint Q-function factorization. MAFIS samples actions that maximize each agent's local Q-function through Equation (9), which has been adopted and recommended by widely-used open-source toolkits such as Stable-Baselines3 and Tianshou in the reinforcement learning community. Taken the maximum entropy RL method SAC as an example, they require deterministic=True to ensure actions that maximize the Q-function are sampled for evaluation. Our key observation is that although the state $s$ is unavailable to each agent during the execution phase in Dec-POMDPs, the action that maximizes $k^i(s) Q^i(\tau, a)$ is the same as the one that maximizes $Q^i(\tau, a)$ due to that $k^i(s)$ is always greater than 0. This implies that even if we sample actions solely based on $Q^i(\tau, a)$, we can still make optimal decisions.

I have concerns about: a) In scenarios requiring exploration or stochasticity (e.g., multi-modal action distributions), the two policies will sample different actions with different probabilities. Has this been considered? b) Did you consider learning Q-functions directly for the execution policy? For instance, by reparameterizing $Q^i$ to absorb the effect of $k^i(s)$?

For your first concern, by treating the Q-function as an energy function and sampling actions, instead of directly modeling the policy with a Gaussian distribution, we are in fact considering that the policy distribution may be multi-modal. Sampling from an energy-based model via SGLD naturally supports obtaining samples from different modes, where the $k^i(s)$ does not change the local optima (as claimed in our paper).

For your second concern, the performance comparison of MAFIS using value factorization methods QMIX or VDN is shown in the following table. MAFIS(QMIX) consistently performs better than MAFIS(VDN), which verifies the effectiveness of introducing the coefficients $k^i(s)$ in Equation (4).

Can you provide analysis on why MAFIS still lags behind expert performance, particularly in continuous control? Have you tried: Varying the number of expert demonstrations (currently only 20 for continuous)? Alternative sampling methods (e.g., HMC)?

With only 20 expert trajectories, MAFIS achieved scores exceeding 4000 on both the Ant (2x4) and Walker2d (2x3) tasks. In contrast, the best baseline, Behavioral Cloning (BC), could not surpass a score of 2000, and MAGAIL failed to learn altogether. This demonstrates the significant effectiveness of our method. On the HalfCheetah (2x3) task, our method also outperformed the best baseline, but there is still a gap compared to the expert's performance. We attribute this primarily to the following reasons:

1. The task itself is difficult, as the continuous action space presents new challenges for agent exploration and learning.

2. The number of expert trajectories is limited. MAFIS would be able to perform better with more expert data. However, considering that expert data may not be easily obtainable in real-world applications, we chose to use only 20 expert trajectories in the paper. As can be seen, MAFIS still performs well even under this condition.

Have you considered comparing with other recent MAIL methods, particularly: SQIL which also builds on soft Q-learning or any offline MARL methods that can work without rewards (e.g., multi-agent behavior cloning with transformers)?

Thanks for your valuable feedback. In our experiments, to ensure a fair comparison, the network used for BC is a multi-layer perceptron (MLP) and a GRU, with the number of learnable parameters being almost equal to those of all other methods. Since a Transformer would introduce significantly more parameters, using it as the backbone for BC in our comparison might be somewhat unfair. However, exploring the use of a Transformer as the backbone within MAFIS is an interesting direction that we will consider for future work. The performance of MAFIS and MASQIL is shown in the following table. As the results indicate, our method MAFIS achieves a higher average score and outperforms the MASQIL baseline in a majority of the test environments.

Thank you for your time in reviewing our paper. Please find our responses below. We hope our responses address your concerns and questions.

The main weakness is that the idea of extending IQ-Learn to the multi-agent setting is not novel...There is a brief discussion on the difference between MFIQ and MAFIS, but the difference seems marginal, and the core idea remains essentially the same...Some theoretical results are trivial and can be easily deduced from prior work.

We appreciate the contributions of MIFQ, and we do not claim that MAFIS is the first work to extend IQ-Learn to the multi-agent setting. However, we would like to argue that MAFIS has made non-trivial contributions compared to MIFQ and IQ-Learn:

1. For discrete control tasks, we found that after decomposing the joint Q-function into the form expressed in Equation (4), the correct form of the optimization objective $\mathcal{J}$ should be as shown in our Equation (7) in Proposition 3.1, rather than the Equation (7) in Proposition 4.4 of MIFQ.

2. For continuous control tasks, a widely criticized problem of IQ-Learn is its instability. Starting from its adversarial optimization problem, we discovered a new optimization approach that achieves non-adversarial optimization even in continuous control tasks. This has led to performance significantly superior to adversarial methods like MAGAIL, whereas MIFQ cannot even be applied to continuous control tasks! We believe that our Propositions 3.1 and 3.2 are not simple extensions of MIFQ's propositions and theorems. On the contrary, we have not only made non-trivial contributions compared to MIFQ but have also solved the optimization problem of the original IQ-Learn in continuous control tasks.

Compared to [1], several relevant baselines are missing, such as VDN and value factorization with non-linear mixing networks...How does your factorization method perform when using non-linear mixing structures, such as a 2-layer mixing network?

The derivation of MAFIS does not require the mixing network to be a one-layer linear network. That is, MAFIS is compatiable with multi-layer non-linear networks. In our submitted code, we implement the mixing network as a non-linear three-layer mlp with ReLU activation functions as below (in line 75 of mafis/mafis_cont/mafis/algorithms/critics/twin_continuous_q_critic.py):

self.mixer = nn.Sequential(nn.Linear(share_obs_space.shape[0], 64), nn.ReLU(), nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 1)).to(device)

Can you say anything about convexity in the Q-function space — an important property of IQ-Learn that has been discussed in both the original IQ-Learn paper and in MFIQ?

Similar to the original IQ-Learn, the objective $\mathcal{J}(\boldsymbol{\pi}_{Q^{tot}}, Q^{tot})$ is concave in the joint Q-function space. Our factorization of the joint Q-function (as shown in Equation (4)) does not change the concavity of the objective.

How does your method compare with other standard baselines, such as VDN or independent IQ-Learn (where you perform IQ-Learn independently for each agent)?

Independent IQ-Learn fails to learn on the continuous control tasks. The performance comparison of MAFIS using value factorization methods QMIX or VDN is shown in the following table. MAFIS(QMIX) consistently performs better than MAFIS(VDN), which verifies the effectiveness of introducing the coefficients $k^i(s)$ in Equation (4).

Thank you for your positive and encouraging comments. Please find our detailed responses to your concerns below.

My primary concern lies in the factorization of the Q-function and the policy. Since each agent selects actions and evaluates its Q-function almost independently...This suggests that the current factorization may fail to capture essential inter-agent dependencies.

During training, although each agent selects actions indenpendently, their Q-functions are evaluated in a joint manner to make sure that the learned Q-functions capture the inter-agent dependencies. To be concise, MFIS updates the Q-functions of all agents simultaneously via Equation (7) or (8), in which the Q-functions of all agents interact to maximize the shared objective.

Another concern arises from the use of the state-dependent coefficients $k^i(s)$, which are used to compose the joint Q-function...it suggests a discrepancy between the training and deployment objectives that could impact theoretical guarantees or worst-case behavior.

Inspired by QMIX, we introduced $k^i(s)$ into the joint Q-function factorization. MAFIS samples actions that maximize each agent's local Q-function through Equation (9), which has been adopted and recommended by widely-used open-source toolkits such as Stable-Baselines3 and Tianshou in the reinforcement learning community. Taken the maximum entropy RL method SAC as an example, they require deterministic=True to ensure actions that maximize the Q-function are sampled for evaluation. Our key observation is that although the state $s$ is unavailable to each agent during the execution phase in Dec-POMDPs, the action that maximizes $k^i(s) Q^i(\tau, a)$ is the same as the one that maximizes $Q^i(\tau, a)$ due to that $k^i(s)$ is always greater than 0. This implies that even if we sample actions solely based on $Q^i(\tau, a)$, we can still make optimal decisions.

The use of SGLD for sampling in continuous action spaces can be computationally expensive, potentially limiting the method’s scalability.

Thanks to community-optimized deep learning packages like PyTorch and TensorFlow, as well as engineering techniques such as Just-in-Time (JIT) compilation, sampling an action via SGLD can be completed within milliseconds on widely used commercial GPUs (e.g., NVIDIA's A100 or 4090). In terms of scalability, MAFIS samples each agent's action independently, so the computational cost increases linearly with the number of agents rather than exponentially. We believe this design ensures strong scalability.

Moreover, employing SGLD for Q-function sampling is a well-established technique and, by itself, does not constitute a significant contribution.

We do not regard samping via SGLD as our contribution. However, to our best knowledge, MAFIS is the first multi-agent imitation learning method that employs SGLD for sampling from the Q-function.

Though factorization is standard, I want to understand the design choice that why not considering learning a centralized q function and then factorized it by learning the expectation of each agent’s Q value?

Your idea is quite interesting, but we haven’t yet found a suitable approach to implement it—we will explore it in future work. Additionally, learning a factorized Q-function has already achieved many successful applications in the field of multi-agent reinforcement learning, which inspires us to consider attempting it in multi-agent imitation learning.

It seems that if we want multi-agent collaborates in the beginning, it would be necessary to introduce some hidden variables before agents make decision.

MAFIS does not explicitly introduce any hidden variables. Instead, during training, we consider there are demonstrations provided by a group of experts. And by saying "a group of experts" here, we mean that they collaborate well. We believe that learning from demonstrations can teach agents how to collaborate, just like we human beings do. During execution, although agents does not commuicate with each other, their learned Q-functions will help them make decisions for a good collaboration.