No Experts, No Problem: Avoidance Learning from Bad Demonstrations

We thank the reviewer for the insightful comments. Below, we provide our detailed responses to each of your questions and concerns.

Baselines: Building on the above point, DWBC should solve this problem quite easily. DWBC originally learns a discriminator...

We thank the reviewer for the insightful comment. While we had previously adapted DWBC to better fit our setting, the approach you suggested provides a valuable alternative perspective. To address this, we conducted an additional experiment using the version you suggested, and we report the results below. Overall, the results indicate that our adapted version of DWBC generally achieves better performance.

The likelihood estimation is done by training a binary classifier. This will lead to issues since the the negative class (unlabeled set) contains the positive class (undesirable set).

We thank the reviewer for the insightful comments. In theory, the discriminator is trained to estimate an occupancy ratio that allows us to leverage unlabeled samples while remaining consistent with the original learning objective. In practice, the discriminator tends to assign higher values to τ(s,a) when the state-action pair appears more frequently in the undesirable dataset than in the unlabeled one, and lower values otherwise. This estimated ratio is directly used in our learning objective (Eq. 6) to guide the policy away from undesirable behaviors and toward better policies. Therefore, the presence of undesirable samples within the unlabeled dataset does not pose a problem for our method.

The paper is slightly incremental, since it is basically flips the objective of IQ-Learn.

We thank the reviewer for the comment. While our approach builds on the structure of inverse Q-learning and, more broadly, the maximum entropy (MaxEnt) framework, it introduces a novel perspective by explicitly focusing on learning from undesirable behaviors—a setting that differs significantly from traditional imitation learning methods, which rely on expert or near-expert demonstrations. Although our objective mirrors the MaxEnt framework in form, it is reconstructed in the opposite direction, and many theoretical properties from MaxEnt no longer apply. This shift requires new analysis and empirical validation. For these reasons, we believe our work offers a meaningful contribution that advances the field in a new and underexplored direction.

Have you tried more diverse data compositions for experiments? For example, some medium behaviors in undesirable set, and random+expert in unlabeled set.

We thank the reviewer for the question. To clarify, our datasets are already diverse: the undesirable data includes both medium and random demonstrations, while the unlabeled dataset contains a mix of random, medium and expert demonstrations. In the experiments presented in both the main paper and the appendix, we include several ablation studies—specifically, Section E.1 shows results when more random and medium demonstrations are added to the undesirable set, and Sections E.2 and E.3 explore the effect of adding more random and expert demonstrations to the unlabeled dataset. We hope these experiments address your concern.

What is the empirical impact of the Extreme Q-learning in Section 4.3? Do you have ablations?

We thank the reviewer for the constructive comment. To evaluate the impact of our extreme Q-update, we conducted additional experiments comparing different Q-function update strategies, including Implicit Q-Learning (IQL) and Sparse Q-Learning (SQL). The results are reported below, showing that Extreme Q-learning (XQL) achieves the best performance as.

We hope the above responses satisfactorily address your concerns. We will update the paper to include the additional experiments and discussion described above. If you have any further comments or questions, we would be happy to address them.

We thank the reviewer for the response and raising important points for further discussion.

For DWBC, do you have any insights as to why this performs worse - is it because of the proportion parameter $\eta$ in the PU classifier not being representative of the data mixture?

We appreciate the reviewer's active engagement and insightful follow-up question. While we don't have a definitive explanation for why the suggested DWBC variant underperforms our adapted version, we can offer the following interpretation, focusing on the core mechanisms and their suitability for different learning scenarios.

The Problem with Aggressive Weight Separation

Both DWBC variants attempt to assign diverse weights to samples from bad and unlabeled datasets to train a policy using a behavior cloning (BC) objective. The goal is for the policy to follow unlabeled data while avoiding bad demonstrations.

However, when the unlabeled dataset is of low quality—which is often the case in our setting—aggressively assigning a "high" weight to unlabeled samples and a "low" weight to bad samples can be problematic.

• The suggested DWBC variant tries to assign diverse weights to distinguish between the two datasets as much as possible. This aggressive separation works well in a learning-from-expert setting, where the difference between expert and unlabeled data is distinct, making it effective to heavily favor expert samples.
• In contrast, the way PU learning is used in our adapted DWBC helps mitigate the divergence between the weights assigned to the two datasets, leading to more stable and reliable outcomes, especially when the quality of bad and unlabeled data is similar.

Our UNIQ Framework vs. DWBC

This reasoning also highlights the key difference between our UNIQ framework and the DWBC variants.

• DWBC relies on a weighting scheme to differentiate between bad and unlabeled data, a mechanism that can fail when the datasets are of similar quality.
• UNIQ avoids this issue entirely by learning to directly avoid bad samples while only using unlabeled data to improve sample efficiency. This approach is more robust because it doesn't depend on a potentially unreliable weighting scheme.

Regarding the parameter $\eta$, we used a default value of $\eta=0.5$ across all experiments. We also tried with higher values of $\eta$. The performance gap stems from the core mechanism of the original DWBC, which is not well-suited for the learning-from-bad-demonstrations setting, rather than from a mismatch in the proportion parameter.

For the data compositions, I was implying towards more discrepancy between unlabeled and undesired set, rather than scaling: some medium behaviors in undesirable set, and random+expert in unlabeled set. The aim is to investigate whether the method works even when all "types" of undesired data is not enumerated.

We thank the reviewer for the additional feedback, which gave us another valuable opportunity to address your concern. In response to your suggestion, we conducted an experiment using your proposed setting: medium behaviors placed in the undesirable dataset and a mix of random+expert trajectories in the unlabeled set.

The results, shown below, indicate that UNIQ—as well as other DWBC-based methods—struggle to learn effective policies in this configuration. We believe the main reason is that avoiding medium demonstrations in the undesirable dataset may unintentionally steer the policy away not only from suboptimal behavior but also towards random behavior present in the unlabeled set. This misdirection undermines the learning process and leads to degraded performance.

We hope this discussion and the accompanying experiment address your remaining concerns. We will incorporate these results and clarifications into the final version of the paper. Please feel free to let us know if you have any further questions or suggestions.

We thank the reviewer for the insightful comments. Below, we provide our detailed responses to each of your questions and concerns.

I'm not sure what point the toy MDP setting is supposed to make: wouldn't minimizing KL against a mixing baseline like SafeDICE (which down-weights p1) still upweight p2 and p3?

The purpose of the toy MDP is not to compare against other baselines, but to illustrate that our learning objective can return near-optimal policies when learning from undesirable demonstrations. We believe that methods like SafeDICE could also do the same in this toy example.

Most of the baselines seem incapable of learning in the unconstrained setting. Do you have an explanation for why this happens and what their limiting factor is?

Among all the baselines, only SafeDICE is explicitly designed to learn from undesirable demonstrations, and it primarily focuses on constrained settings. As discussed in our paper, SafeDICE ‘s performance is highly sensitive to the quality of the unlabeled dataset. In contrast, our approach is theoretically designed to be less dependent on the quality of the unlabeled data, offering greater robustness in challenging settings.

Just to clarify: for Proposition 4.1 and Section 4.3, is the optimal V being relative to the behavior policy a property of the offline setting?

Yes, this is a common property of the offline setting, where data is assumed to be sampled from a behavior policy. This assumption is widely adopted in both offline RL and IL.

Do you have a sense of whether the cooperative nature of the setting allow you to do better than the expert?

In our setting, outperforming the expert cannot be guaranteed and is inherently challenging, as we do not have access to expert demonstrations for guidance. Instead, the cooperative nature of our approach contributes to a more stable training process and helps steer learning toward a good policy.

Figure 3 seems to demonstrate that UNIQ scales better with expert demos vs other methods. Do you have an explanation for this behavior?

We thank the reviewer for the comment. Compared to other baselines, our method can better leverage expert demonstrations within the unlabeled data. As shown in our learning objective (Eq. 6), when more expert demonstrations are present, the algorithm assigns lower τ(s,a) values to those expert-like pairs, encouraging higher rewards for them and ultimately leading to a better policy.

Are there results for the data scaling experiments (expert and undesirable demos) with larger amounts of data? Right now, the ablation only spans 1 OOM.

We thank the reviewer for the comment. In our current ablation study (Section E.3), we vary the number of expert demonstrations from 50 to 300, which we believe reflects a realistic and practical range in scenarios where expert data is typically limited.

This is out of scope of the paper, but do you have any thoughts on how UNIQ scales in the online setting?

We thank the reviewer for the question. While the current framework could be extended to the online setting, making it effective would require substantial effort, including adapting the learning objective and regularization, as well as benchmarking against a new set of baselines. We consider this a valuable and promising direction for future work.

Re. weakness 1: I think the setting is novel, but the scope is pretty narrow to the offline, data-constrained setting,...

We thank the reviewer for the constructive feedback and the helpful suggestion. Our work indeed focuses on the offline, data-constrained setting, which we agree is narrower in scope compared to more general-purpose IL frameworks. However, we believe this setting is both practically relevant and novel, particularly in safety-critical applications (e.g., robotics, autonomous driving, or healthcare), where undesirable behaviors—such as collisions or failures—are easier to collect or identify than expert demonstrations. In such cases, it is often significantly more feasible to label or log what not to do, rather than gather consistent expert behavior.

Empirical evidence that UNIQ scales better with undesirable demos than baselines in this setting. For example, Figure 4 (Point-Goal) ...

We thank the reviewer for raising this valuable point. To address it, we conducted additional experiments using a significantly larger number of undesirable demonstrations to enable further analysis and comparison. The tables below report the performance of UNIQ and baseline methods on the Car-Goal and Point-Goal tasks, with up to 1000 undesirable demonstrations. Due to NeurIPS rebuttal format restrictions, we are unable to include figures, so we report only the mean and std of the final scores. Overall, the results show that UNIQ consistently achieves lower costs—indicating safer policies—while maintaining comparable returns to the baselines. This supports the robustness and effectiveness of UNIQ even in heavily imbalanced settings dominated by suboptimal data.

Point-Goal

Car-Goal

Re. weakness 2: I have some concerns about estimating the occupancy measure using a discriminator. Although the idea is pretty classic, ... I believe this would be a valuable contribution for understanding the empirical properties of algorithmic primitives for estimating occupancy ratios.

We thank the reviewer for the valuable and insightful comment. Regarding the accuracy of occupancy ratio estimation, we fully agree that it can be inaccurate when datasets are small in size—an issue common in practice. However, as shown in our experiments, our algorithm remains robust and effective even with limited undesirable data. The key reason is that precise occupancy ratio estimation is not strictly required for good policy learning. In our main training objective (Equation 6), the discriminator encourages higher τ(s,a) values for state-action pairs that appear more frequently in the undesirable dataset, and lower values otherwise. This naturally drives the learned Q-function and policy to assign lower rewards r^Q(s,a) to undesirable behaviors, guiding the agent toward better actions. This behavior aligns with findings from prior work such as GAIL and DICE-based methods, which also demonstrate robustness under noisy ratio estimates.

I think you could do this in the safety constrained setting where the oracle occupancy ratio is known since you have likelihoods of the PPO-Lagrangian and PPO policies.

We thank the reviewer for the insightful suggestion. However, we are currently unsure how to compute the oracle occupancy ratio using PPO or PPO-Lagrangian policies. Estimating this ratio would typically require access to both undesirable and unlabeled datasets, along with accurate sampling from the target policy's occupancy distribution. In our case, we are not sure how to use PPO policies to get a reliable estimate of the ratio. If the reviewer could clarify the intended procedure, we would be happy to explore it further and include the results.

The main experiment would then show that NCE is sufficient to learn a "good estimator"...

To further address your concern, we conducted an additional experiment to evaluate the estimation of the occupancy ratio τ(s,a) across varying dataset sizes. Due to NeurIPS rebuttal policy restrictions, we are unable to include figures, so we report only the mean and standard deviation of the estimated ratios across the (s,a) space. The results generally indicate that the estimation of the occupancy ratio becomes stable as the dataset size grows. In the revised version of the paper, we will include detailed distribution plots and provide further analysis to strengthen the discussion around this estimation.

We hope the above responses satisfactorily address your concerns. We will update the paper to include the additional experiments and discussion described above. If you have any further comments or questions, we would be happy to address them.

We thank the reviewer for maintaining a positive view of our work and for the valuable feedback, particularly regarding the estimation of the occupancy ratio, which has helped us significantly improve the paper. We will definitely incorporate all the discussions and additional experiments into the final version.

We thank the reviewer for the insightful comments. Below, we provide our detailed responses to each of your questions and concerns.

This approach seems to simply add the setting of undesirable demonstrations to other methods.

We thank the reviewer for the feedback. We would like to emphasize that our work introduces a novel perspective by explicitly focusing on learning from undesirable behaviors—a setting that diverges significantly from conventional IL, which typically relies on expert or near-expert demonstrations. This shift presents a fundamentally new problem, requiring new training objectives and theoretical insights. Notably, one of our key and novel findings is that imitation learning can still succeed even when no expert demonstrations are explicitly identified in the dataset. We believe this opens an important and underexplored direction in the field, and represents a meaningful contribution to advancing the boundaries of IL research.

As mentioned in the paper in Limitations, 'each undesirable trajectory may not be undesirable in its entirety', which may affect the final effect when used in practice.

We thank the reviewer for the insightful comment. Extending our approach to handle partially undesirable trajectories is indeed valuable, but it introduces significant challenges. Our method assumes each trajectory comes from a clearly defined policy—either desirable or undesirable—an assumption that breaks down when trajectories are mixed. Moreover, current benchmarks like D4RL lack segment-level labels needed to distinguish good and bad parts within a trajectory. Handling this would require additional supervision and methodological changes, which we consider a promising direction for future work beyond the scope of this paper.

If the probability of the demonstration appearing in undesirable demonstrations is less than the probability of appearing in unlabeled data, then is the optimization direction of Equation 4 reversed, and have you done any relevant experiments? For example, you can simply duplicate two copies of the undesirable demonstrations or add more to the unlabeled data.

We thank the reviewer for the helpful comment. To clarify, our learning objective in Equation 4 does not involve the unlabeled dataset directly, so the optimization direction is theoretically unaffected even if the probability of a demonstration appearing in the undesirable dataset is lower than in the unlabeled dataset.

Your point regarding adding more undesirable samples to the unlabeled data is well taken. To further investigate this, we conducted an additional experiment (see below) where we substantially increased the proportion of bad demonstrations in the unlabeled dataset to analyze how performance changes as its quality degrades.

Experimental description: In this experiment, we evaluate two Mujoco tasks. We keep the same undesirable dataset $\mathcal{D}^{UN}$ as used in the main comparison in Section 5.2, consisting of 5 random and 5 medium trajectories. For the unlabeled dataset $\mathcal{D}^{MIX}$, we also retain the same 100 expert trajectories from Section 5.2, but vary the number of random and medium trajectories. Specifically, the unlabeled dataset has the format X:100, where X is the total number of random and medium trajectories combined (with X/2 random and X/2 medium), together with the 100 expert trajectories.

Cheetah

Ant

At present, the number of experts in the setup is relatively small. Is there an experiment with a larger number of experts, and will it be better than other methods?

We thank the reviewer for the comment. To clarify, expert demonstrations are included only in the unlabeled dataset. As detailed in Section E.3 of the Appendix, we conducted an ablation study evaluating the performance of UNIQ and key baselines under varying amounts of expert data (from 50 to 300 demonstrations, with 300 considered a relatively large amount). The results show that UNIQ consistently outperforms the baselines across all settings.

Have you tried adding optimal demonstrations and undesirable demonstrations to train together, and will the results of UNIQ be better

We thank the reviewer for the insightful comments. Our work is primarily focused on the setting where expert (or optimal) demonstrations are not available, and the algorithm is explicitly designed with this assumption in mind. Incorporating expert demonstrations directly into our learning objective (Equation 4) would substantially alter its structure and potentially lead to instability, as the learning signals from expert and undesirable data may conflict and drive the optimization in opposite directions. As such, we believe the current form of our algorithm is not well-suited for simultaneously learning from both expert and undesirable demonstrations. Addressing this scenario would likely require a fundamentally new algorithmic framework.

That said, in settings where high-quality expert demonstrations are available, existing methods designed for learning from both expert and sub-optimal data—such as DemoDICE or DWBC—may be more appropriate.

The cost metric of the Car-Button in table 1 is much lower than the UNIQ on the LS-IQ method, but it shows that the UNIQ is the best, is it a recording error?

Thank you for the comment. In this experiment, the LS-IQ policy achieves low cost but at the expense of an unacceptably low reward. In fact, LS-IQ policy in fact does nothing, so the cost is low, but it also fail to gain meaningful rewards. Therefore, it is reasonable not to consider LS-IQ’s performance as best in this case. We will clarify this point in the updated version of the paper.

What are the advantages of the method proposed in this paper compared to confidence based imitation learning methods, such as 2IWIL, CAIL, WGAIL, PN-GAIL [1-4].

We thank the reviewer for the comment. The key difference between our method and confidence-based approaches like 2IWIL, CAIL, WGAIL, and PN-GAIL is that those methods rely on per-sample confidence estimation and typically assume access to expert or near-expert demonstrations. In contrast, UNIQ is specifically designed for the more challenging setting where expert data is unavailable and only undesirable and unlabeled demonstrations are provided. This makes UNIQ more practical in scenarios where collecting high-quality expert data is infeasible.

Has the time spent in each method been recorded?

We thank the reviewer for the comment. To clarify this, we compare the training speed of UNIQ to DWBC, SafeDICE, IPL, and LSIQ on a single task (Cheetah), using the same RTX 3090 GPU. The training speeds have been recorded as below:

It seems that the model is effective by avoiding undesirable demonstrations, how to ensure the comprehensiveness of undesirable demonstrations?

We thank the reviewer for the thoughtful comment. While achieving full coverage of undesirable demonstrations is indeed challenging, our work is motivated by the practical observation that collecting undesirable data is often easier and less costly than collecting expert demonstrations. In real-world scenarios, such data can be obtained from logged failures or suboptimal policies, which typically span a sufficiently diverse set of poor actions. Based on this, we assume that the undesirable dataset provides adequate coverage of common failure modes. Moreover, our experiments—including those presented in the appendix—demonstrate that UNIQ remains robust across a variety of settings. That said, we also observe a decline in performance when the number of undesirable demonstrations is very low, which is expected and consistent with our assumptions.

We hope the above responses satisfactorily address your concerns. We will update the paper to include the additional experiments and discussion described above. If you have any further comments or questions, we would be happy to address them.

We thank the reviewer for the insightful comments. Below, we provide our detailed responses to each of your questions and concerns.

The discriminator estimating occupancy ratios plays a central role. How robust is this estimation under class imbalance or covariate shift between DUN and DMIX? Would the method degrade gracefully or fail catastrophically? If the robustness of occupancy ratio estimation is shown to fail under modest noise, that would weaken the empirical claims.

We thank the reviewer for the insightful comment. It is indeed true that the estimation of occupancy ratios can be inaccurate when datasets are small or imbalanced. However, as shown in our experiments (especially those reported in appendix), despite the UN (undesirable) dataset typically having a low sample size, our UNIQ algorithm remains robust and effective.

The key reason is that precise estimation of the occupancy ratio is not strictly necessary for learning a good policy. Specifically, in our main training objective (Equation 6), the discriminator is designed to assign a higher value to τ(s,a) when the state-action pair (s,a) appears more frequently in the UN dataset, and lower values otherwise. As a result, the objective encourages the learned Q-function and policy to assign lower rewards r^Q(s,a) to undesirable state-action pairs, thereby guiding the policy away from undesirable behaviors and toward more desirable ones.

This phenomenon is consistent with prior findings in related works, such as GAIL and DICE-based algorithms, where learning remains effective even when occupancy ratios are estimated from low-sample datasets.

The authors note partial undesirability in demonstrations as a limitation. Could the method be extended to segment trajectories and learn from sub-trajectory labeling, possibly via importance weighting or causal inference?

We thank the reviewer for the insightful comments. As mentioned in our paper, extending our approach to handle partially undesirable trajectories would introduce several new challenges:

• Violation of policy-level assumptions: Our method relies on estimating occupancy measures for undesirable and mixed policies, assuming that each dataset is generated by a well-defined policy—either desirable or undesirable. When trajectories are only partially undesirable, this assumption no longer holds, as the generating policy cannot be clearly classified as good or bad. This ambiguity makes our current formulation unsuitable for such cases.
• Lack of segment-level labels: Identifying which parts of a trajectory are desirable or undesirable is itself a challenging task. Existing benchmark datasets such as D4RL do not provide this level of granularity—they typically consist of full trajectories generated by either expert or non-expert policies, without annotations indicating which segments are good or bad. Addressing this would require additional supervision or expert feedback, as well as new methodological tools and thorough empirical validation.

Given the complexity and scope of these challenges, we believe this extension falls outside the focus of our current work, but we agree it is a promising direction for future research.

While the paper claims minimal hyperparameter tuning, can the authors provide more details on how sensitive UNIQ is to choices like β, the regularization coefficient, or the discriminator architecture?

We thank the reviewer for the comment. To address this, we conducted additional ablation studies to analyze the impact of the entropy regularization parameter β on the performance of UNIQ. The results are presented below. We note that β is a standard hyperparameter in maximum entropy frameworks, and its effects have been extensively studied in prior work. Our findings align with those results, showing that performance remains stable across a reasonable range of values.

For other standard hyperparameters—such as the regularization strength and the discriminator architecture—we adopted settings from prior work. These choices have been shown to be effective and well-validated, representing strong defaults for our setting.

Have the authors considered applying this to high-dimensional observation spaces (e.g., images) or domains with long-horizon credit assignment challenges?

We thank the reviewer for the thoughtful comments. In this work, we have focused our experiments on widely used benchmark tasks from the D4RL suite, which are standard in the RL and IL communities. These tasks provide a solid foundation for evaluating and comparing the performance of our method against state-of-the-art baselines.

We acknowledge that applying our approach to more complex settings—such as high-dimensional observation spaces (e.g., vision-based tasks) or domains with long-horizon credit assignment challenges—would be a valuable extension. However, these settings often require significant additional engineering effort (e.g., integrating convolutional architectures or temporal abstraction mechanisms) and computational resources for robust experimentation and validation. We consider this an important and promising direction for future work.

Although occupancy correction is intended to remove bias, is there any empirical observation on how performance changes with extreme noise or very low quality unlabeled data?

We thank the reviewer for the insightful comment. To further address this point, we conducted additional experiments to evaluate the performance of our algorithm under conditions where the unlabeled dataset is of extremely low quality. Specifically, we varied the number of suboptimal (bad) demonstrations within the unlabeled set, thereby progressively degrading its overall quality. As expected, we observed a noticeable performance degradation as more bad demonstrations were added to the unlabeled dataset. When the overall quality of the unlabeled data became extremely poor, the algorithm's performance dropped significantly. This outcome is reasonable, as the learning signal becomes increasingly noisy and less informative in such scenarios.

Experimental description: In this experiment, we evaluate two Mujoco tasks. We keep the same undesirable dataset $\mathcal{D}^{UN}$ as used in the main comparison in Section 5.2, consisting of 5 random and 5 medium trajectories. For the unlabeled dataset $\mathcal{D}^{MIX}$, we also retain the same 100 expert trajectories from Section 5.2, but vary the number of random and medium trajectories. Specifically, the unlabeled dataset has the format X:100, where X is the total number of random and medium trajectories combined (with X/2 random and X/2 medium), together with the 100 expert trajectories.

Cheetah

Ant

We hope the above responses satisfactorily address your concerns. We will update the paper to include the additional experiments and discussion described above. If you have any further comments or questions, we would be happy to address them