MisoDICE: Multi-Agent Imitation from Mixed-Quality Demonstrations

We thank the reviewer for the insightful comments. Below, we provide our detailed responses to each of your questions and concerns.

Dependence on LLMs and Generality of the Framework: ... This undermines the claim that LLM-based labeling is a core, generalizable part of the contribution and raises questions about the framework's applicability to common imitation learning domains like robotic manipulation. The two stages feel disconnected, and the paper might be stronger if it focused on the more robust contribution of the MisoDICE algorithm itself.

We appreciate the reviewer’s insightful observation regarding the broader applicability of our algorithmic contributions. We agree that the LLM-based component is currently more suited to domains with interpretable features. Nonetheless, we believe that both components of our framework represent meaningful advancements. Our use of LLMs to identify expert-like demonstrations from fully unlabeled datasets is, to the best of our knowledge, a novel and impactful contribution that opens promising avenues for preference-based learning. In particular, it paves the way for future work on leveraging LLMs to identify expert behavior in more challenging and less interpretable environments, such as MAMuJoCo.

Importantly, our core algorithmic design in Phase 2 is general and agnostic to how expert demonstrations are obtained, making it broadly applicable across various settings. In the revised version of the paper, we will refine our contribution statement to more clearly articulate the generality and standalone value of this component.

Clarity of Positioning and Presentation: The paper's positioning of imitation learning (IL) as a subfield of reinforcement learning (RL) could be more nuanced....

We thank the reviewer for the insightful feedback. We recognize the importance of clearly distinguishing between IL and RL, especially given the nuanced differences in their problem settings, learning objectives, and assumptions. While RL focuses on learning optimal behavior through interaction with an environment and explicit reward signals, IL seeks to learn policies by mimicking expert behavior, often in the absence of reward functions. In the revised version of the paper, we will revise the presentation to explicitly highlight this distinction—both in the introduction and background sections.

... Although methods like "VDN" in Figure 1 look like baseline methods, they are clearly ablation studies of the proposed method. ...

We thank the reviewer for the insightful observation. You are correct that VDN, as used in our experiments, is a simplified adaptation of our main algorithm, MisoDICE, rather than a standalone existing baseline. We will revise the manuscript to clarify this distinction and avoid any potential confusion.

Clarification on BC Performance: In Figure 1, the performance of Behavioral Cloning (BC) appears to increase over the course of training. ...

We thank the reviewer for the insightful question. The performance curves shown in our experiments reflect evaluation during the policy update process. Specifically, we use batch samples from the offline dataset to update the policy using BC, and after a certain number of epochs, we evaluate the policy’s performance. We will clarify this evaluation procedure more explicitly in the experimental section.

Justification of the Two-Stage Framework: The paper's main theoretical contribution appears to be the MisoDICE algorithm (Stage 2), which is elegant and well-motivated. However, it is tightly coupled with the LLM-based labeling pipeline (Stage 1), which seems less general and brittle, as evidenced by its failure on MAMuJoCo. Could the authors elaborate ...? Would the paper's contribution be clearer and more robust if Stage 2 (MisoDICE) were presented as the core method, with the LLM-based approach being just one possible (and domain-dependent) method for generating the required labeled data?

We thank the reviewer for the insightful comments and thoughtful question, with which we fully agree. The close coupling between Phase 1 and Phase 2 in our framework stems from the original motivation of the work—tackling imitation learning in the challenging setting where the dataset is entirely unlabeled. In this context, Phase 1 plays a critical role, as effective imitation learning cannot proceed without first identifying expert-like demonstrations.

While we agree that Phase 2 is more general and may have broader standalone applicability—making it particularly relevant to some readers—we believe that Phase 1 also offers important contributions. To the best of our knowledge, it is the first approach that leverages LLMs to identify expert-like trajectories from noisy, unlabeled data. This opens promising future directions, including extending such techniques to less interpretable domains like MuJoCo, potentially leading to broader impact in preference-based and offline IL.

---

We hope the above responses satisfactorily address your concerns. We will update the paper to include the discussion described above. If you have any further comments or questions, we would be happy to address them.

We thank the reviewer for the insightful comments. Below, we provide our detailed responses to each of your questions and concerns.

The labeling phase relies on domain-specific semantic features ... How was the rule-based oracle designed for MaMuJoCo? ...If it requires domain-specific heuristics, does this undermine the framework’s generality?

We thank the reviewer for the insightful question. First, to clarify, the "rule-based oracle" used for the MAMuJoCo tasks is derived from the ground-truth reward function. In this domain, we assume access to expert feedback to generate preference labels, as LLMs are not well-suited for interpreting raw low-level state representations typical of MuJoCo environments.

In our framework, Phase 2 is more general, as it can be applied to settings where expert and suboptimal demonstrations are available, regardless of how the expert data is identified—either manually or via LLMs. In contrast, Phase 1 is somewhat less general, as it relies on the availability of contextual and interpretable features that can be understood by LLMs. However, we believe Phase 1 still offers independent value: to the best of our knowledge, this is the first approach that leverages LLMs to identify expert-like demonstrations from fully unlabeled and potentially noisy datasets. This opens promising directions for future work, particularly in exploring how to adapt LLM-based preference labeling to less interpretable domains like MuJoCo through better feature engineering or learned state abstractions.

How sensitive is the labeling phase to the quality and specificity of the LLM prompts? Could low-quality or overly generic LLM outputs significantly degrade O-MAPL refinement and subsequent IL performance?

We thank the reviewer for the insightful question. As noted in the appendix, we have already conducted an ablation study to analyze the sensitivity of MisoDICE to the quality of preference feedback in Phase 1. Specifically, in Section C.4 of the appendix, we compare two versions of the preference labeling pipeline: one using GPT-4o and the other using GPT-4o-mini. Since GPT-4o-mini is a smaller model with limited reasoning and contextual understanding capabilities, it tends to generate lower-quality preference signals. Our results show that this degradation in feedback quality leads to less accurate labeling and significantly reduces the overall performance of MisoDICE in Phase 2.

We have also conducted an additional ablation study to analyze the impact of noisy preference feedback in Phase 1 on the overall performance of MisoDICE. In this experiment, we started with preference data generated by GPT-4o and randomly flipped a portion of the preference labels at varying levels (5%, 10%, 20%, etc.). The results, reported in the table below, clearly illustrate that increasing the amount of noise in the preference data leads to a noticeable decline in MisoDICE's performance, highlighting the framework’s sensitivity to label quality in the preference-based learning phase.

The author claims to have solved the computational complexity of the large-scale joint state-action space, but lacks relevant theoretical or experimental proof.

We thank the reviewer for the insightful question. Our design choices—including value factorization and local network construction—are specifically aimed at addressing the computational complexity inherent in multi-agent settings, particularly when dealing with large-scale joint state-action spaces. This approach has been adopted in several recent state-of-the-art MARL methods and has been shown to significantly improve computational scalability and efficiency. We will clarify this point more explicitly in the revised version of the paper.

Proposition 5.5 assumes that behavioral strategies can be decomposed into independent local strategies, but the behaviors of agents in cooperative tasks are usually correlated. Please demonstrate the rationality of this assumption.

We thank the reviewer for the insightful question. This is actually a common assumption in the existing MARL literature. To clarify, we do not assume that the global policy is decomposable into separate independent local policies (The local policies are not necessarily independent.). Rather, our framework relies on the assumption of consistency between global and local policies—specifically, that the global policy can be expressed as the product of local policies. This formulation captures a synergistic relationship, where coordinated local behaviors collectively give rise to the global strategy, rather than treating each agent’s policy in isolation.

---

We hope the above responses satisfactorily address your concerns. We will update the paper to include the additional experiments and discussion described above. If you have any further comments or questions, we would be happy to address them

We thank the reviewer for the insightful comments. Below, we provide our detailed responses to each of your questions and concerns.

MisoDICE performance heavily rely on the quality of the external preference model...

We acknowledge that our method relies on a learned preference model guided by LLM-generated feedback to identify expert-like demonstrations. This design represents a significant departure from prior work, which typically assumes access to costly, high-quality expert-labeled trajectories. By leveraging LLMs as a scalable and cost-efficient supervision source, our framework enables IL from fully unlabeled datasets. This not only reduces reliance on human annotation but also broadens the applicability of imitation learning to more practical, real-world settings where labeled expert data is unavailable or difficult to collect.

While we agree that performance heavily depends on the quality of feedback provided by the LLM, this is a common and reasonable phenomenon in the community. In fact, all existing IL algorithms similarly rely on the quality of expert demonstrations in the dataset.

The two stage pipeline requires multiple model components which introduces additional implementation complexity.

We thank the reviewer for the insightful comment. We would like to highlight that, as demonstrated through extensive ablation studies in the appendix, our two-stage framework works effectively in tandem to deliver robust training outcomes across a range of settings. Moreover, while the stages are designed to complement each other, each stage also holds independent value. Specifically, the first stage can be broadly applied to extract expert-like demonstrations from unlabeled data and is compatible with a variety of existing imitation learning algorithms. The second stage, in turn, can operate with any dataset that includes both expert and suboptimal demonstrations, regardless of how the data is labeled. Therefore, whether used jointly or independently, both stages of our framework offer meaningful contributions to the imitation learning literature.

Though linear mixing provide convexity, the model might benefit from more expressive mixing strategies.

We agree that more expressive mixing strategies—such as two-layer nonlinear mixing networks—can, in principle, enhance representational capacity. Our algorithm is designed to be general and can indeed accommodate such architectures. That said, we intentionally adopt a linear mixing network for two key reasons. First, as shown in Proposition 5.2, the linear form guarantees convexity of the training objective, which promotes more stable and reliable optimization. Second, our design is grounded in empirical evidence from recent work in MARL and multi-agent imitation learning, which consistently finds that nonlinear mixing networks are prone to overfitting and often degrade performance in offline regimes. Thus, our choice reflects a balance between expressiveness and stability, tailored for the offline learning setting

Could preference noise lead to misleading labels that propagate into Phase 2? Have you tested how the performance of MisoDICE degrades when preference labels are noisy or contradictory?

Yes, noisy preference signals can lead to inaccurate trajectory labeling, which in turn can degrade the performance of the imitation learning algorithm in Phase 2. To investigate the sensitivity of our framework to the quality of preference feedback, we conducted an ablation study presented in Section C.4 of the appendix. In this experiment, we compared two versions of the preference labeling pipeline: one using GPT-4o and the other using GPT-4o-mini. While both models are capable of generating high-level semantic judgments, GPT-4o-mini—being a lighter variant—is significantly more limited in its reasoning and contextual understanding, especially in the complex, high-dimensional environments we consider.

Our results, reported in Table 10, reveal a clear performance gap between the two. Specifically, MisoDICE trained with GPT-4o-based labels consistently outperformed the version using GPT-4o-mini, both in terms of average returns and learning stability. This supports the hypothesis that high-quality, consistent preference feedback is critical for effective trajectory labeling and downstream policy learning. These findings also underscore the importance of choosing capable LLMs when deploying preference-based learning pipelines in practical applications.

To further address your question, we conducted an additional ablation study in which we explicitly introduced noise into the preference labeling process in Phase 1. Specifically, we obtained preference labels from GPT-4o and randomly flipped the preference outcomes for a certain percentage of trajectory pairs—at 5%, 10%, 20%, and 30% of the samples. The results, reported below, clearly show a gradual degradation in MisoDICE’s performance as the level of noise increases,

---

We hope the above responses satisfactorily address your concerns. We will update the paper to include the additional experiments and discussion described above. If you have any further comments or questions, we would be happy to address them.

We thank the reviewer for the insightful comments. Below, we provide our detailed responses to each of your questions and concerns.

Though the paper uses techniques from prior work (like O-MAPL); they are not mentioned in the Background section…

We will revise the Background section to better highlight and clarify the importance of preference-based methods (such as O-MAPL) in our framework.

The first phase of this work is summarized briefly, without providing sufficient details.

We thank the reviewer for the comment. We agree that the details of Phase 1 could be more thoroughly explained in the main paper. However, due to space constraints, we chose to move some of the descriptions to the appendix, in order to allocate more space in the main text to Phase 2, which involves more technical depth.

If LLM provides noisy preference data, I am not sure how O-MAPL learns a soft Q-function? …

When the feedback is noisy or less accurate, the ability of O-MAPL to correctly distinguish expert-like trajectories deteriorates, which in turn negatively affects the downstream performance in Phase 2. To empirically validate this, we already conducted an ablation study in Section C.4 comparing GPT-4o and GPT-4o-mini for preference labeling. Due to its limited reasoning ability, GPT-4o-mini produced noisier feedback, leading to lower labeling accuracy and degraded Phase 2 performance.

Additionally, we ran an additional ablation where we introduced synthetic noise by randomly flipping GPT-4o-generated preferences at varying rates. The results, shown below, demonstrate a clear decline in MisoDICE’s performance as noise increases.

There are 3 different local networks per agent and 2 different mixing networks making the overall framework extremely complicated and intensive…

We thank the reviewer for the insightful comment. In fact, our use of local networks with mixing architectures under the CTDE framework is explicitly designed to enhance scalability and efficiency in multi-agent learning—a strategy also adopted by recent SOTA MARL and IL methods. Our experiments confirm that MisoDICE scales well, even on the most challenging SMAC_v2 scenarios.

To further support our argument regarding the computational efficiency, we will include the following table summarizing the total training time for each benchmark environment. The results clearly demonstrate that MisoDICE requires a reasonable amount of training time.

I do not understand the intuition behind discriminator based classification problem (looks like cross-entropy loss) as in Eqn 9? …

We thank the reviewer for the question. The purpose of the discriminator in Eq. 9 is to estimate the occupancy ratio, which is a critical component of our learning objective. This classifier-based approach can be shown to recover the occupancy ratio between expert and unlabeled datasets. Similar techniques have been widely adopted in prior IL algorithms, such as GAIL, where the discriminator serves a comparable role in distinguishing between expert and non-expert trajectories.

There is a trade off between the amount of gain achieved in terms of performance versus the computational overhead… VDN also learns pretty well in this setting.

We thank the reviewer for the comment. We agree that there is an inherent trade-off between performance gains and computational efficiency, which is a common phenomenon in ML. In our context, the VDN variant serves as a simplified version of MisoDICE. As expected, its performance is slightly lower than MisoDICE, but better than other baselines.

What is the intuition behind R~Q-\gamma V in section 4?...

This approach is adapted from the O-MAPL paper on preference-based learning, where the expression $Q−\gamma V$ is interpreted as a reward function defined in the Q-value space. By learning this reward function implicitly through the Q and V functions, we enable a more consistent and integrated way of learning both the reward and policy.

Does the rankings of the trajectory done in Phase-1 align with ground truth?...

We thank the reviewer for the insightful question. In response, we conducted an additional experiment in which we sampled 2,000 trajectory pairs per task and generated preference rewards using LLM feedback and O-MAPL. These were then compared to ground-truth preferences derived from environment rewards. The results, reported below, indicate an overall accuracy of 90.5%, demonstrating the high reliability of the LLM feedback in Phase 1. This highlights the effectiveness of our approach in Phase 1 in filtering expert trajectories that closely align with the ground-truth preferences.

How often are the 2 mixing networks trained in this framework?….

We thank the reviewer for the question. The mixing network and local networks are trained simultaneously via gradient backpropagation. The main training objective for jointly updating both components is provided in Eq.6. A detailed description of the training procedure can be found in Section B.5 of the appendix, along with a visual illustration in Figure 4.

I am curious as to why MisoDICE is almost similar in training performance in smaller configurations (Fig 1; 5v5; 10Vs 10) than bigger ones?

We thank the reviewer for the question. On small-scale tasks, MisoDICE and other baselines show similar performance due to low task complexity. While training plots may appear noisy, closer inspection shows that MisoDICE consistently outperforms the baselines.

I could not find MisoDICE-VDN as mentioned in Baselines in the Experimental plots.

MisoDICE-VDN is actually VDN reported in our main experiments. We included it as a lightweight variant to highlight the modularity of our approach. We will clarify this in the revised version of the paper.

I would have liked to see an ablation study without phase-1; …

We thank the reviewer for the suggestion. In response, we ran an ablation where all unlabeled data were treated as optimal for Phase 2. Results show that MisoDICE without labeling performs worse than the full version but slightly better than BC, highlighting the importance of Phase 1's labeling strategy for learning from noisy, unlabeled data.

---

We hope the above responses satisfactorily address your concerns. We will update the paper to include the additional experiments and discussion described above. If you have any further comments or questions, we would be happy to address.