Offline Imitation Learning without Auxiliary High-quality Behavior Data

Q1: Discussion with model-based methods

Thanks for your valuable suggestions. When conducting the experimental comparison, we mainly referred to previous offline imitation learning methods and their competing baselines [1-3]. Our proposal is also a model-free solution, so we primarily compare it with model-free methods, which are widely adopted in this topic. Comparing with model-based methods may not be fair as they require additional training on dynamics models.

[1] Demodice: Offline imitation learning with supplementary imperfect demonstrations. ICLR 2022
[2] Discriminator-weighted offline imitation learning from suboptimal demonstrations. ICML 2022
[3] Optimal transport for offline imitation learning. ICLR 2023

It is true that model-based methods also utilize transition information, (Reviewer Xzkb also pointed out it). Therefore, we add the discussion with the model-based methods in Appendix B to clarify the difference between our proposal and the model-based methods. Overall, our approach is fundamentally different from existing model-based methods in terms of utilizing transition information. Existing model-based methods [4-7] may still struggle to handle the problem we propose.

1. On the one hand, since the data does not have reward labels, the trajectories sampled by model-based methods are still unlabeled. We still need to consider how to utilize these low-quality behavior data that have been sampled. In other words, the problem we presented in this paper is still unsolved.
2. On the other hand, these methods [4,7] still perform distribution matching between the generated trajectories and the state-action distribution of expert behavior ($p(s, a)$) to annotate reward labels, which is consistent with our summary of existing work in equations 2 and 3. As we discussed in sections 2.2 and 2.3, they cannot provide optimization objectives on expert-unobserved states because this similarity can only rely on the model's function approximation ability to distinguish the quality of unlabeled trajectories and utilize them. This makes it difficult for them to effectively utilize offline data, which may contain many expert-unobserved states.

On the contrary, when utilizing transition information, we consider the subsequent expert-state distribution maximization. This approach does not expect the sampled trajectories to necessarily contain samples with a similar state-action distribution to expert data. Instead, it provides guidance to the agent towards expert-observed states through the transition information between states, thereby improving the lower bound of long-horizon return.

[4] Discriminator-Guided Model-Based Offline Imitation Learning. CoRL, 2022.
[5] Model-Based Imitation Learning for Urban Driving. NeurIPS, 2022.
[6] MobILE: Model-Based Imitation Learning From Observation Alone. NeurIPS, 2021.
[7] Mitigating Covariate Shift in Imitation Learning via Offline Data with Partial Coverage. NeurIPS’21

Q2: Update of Q function

The delayed update is designed in the TD3 algorithm, aiming to decrease the error per update. In simple terms, the delayed update allows the critic to stabilize before updating the actor, avoiding the critic's changing too much and causing the actor's optimization direction to become unstable. For detailed information and further discussion, please refer to the TD3 paper [8].

[8] Addressing Function Approximation Error in Actor-Critic Methods. ICML 2018

Thanks for your positive comments and valuable suggestions. Below are our responses to the concerns.

W1: The reward design seems to be too strict

We obtained this reward design by maximizing the expert-state distribution: samples in the expert dataset have a reward of 1, and samples in the auxiliary dataset have a reward of 0. This design may be conservative because we mainly focus on the utilization of low-quality data, thus only utilizing transition information, which is independent of behavior quality. When the data contains some expert-similar trajectories, we can also replace the current identical reward with the discriminator used in prior work to make use of it. Here we provide the corresponding additional experimental results. Specifically, we use DWBC, and OTIL to annotate the reward labels for offline auxiliary data, and then apply the BCDP method with optimized rewards.

The results of BCDP with optimized rewards demonstrate its potential when combined with previous reward annotation methods. Especially when the auxiliary data contains some relatively high-quality data, the performance of the BCDP framework can be further improved. It clearly supports the advantages of our proposal on scalability. We have added them to Appendix C.6 as additional supplementary material.

W2: The detailed hyperparameters and some auxiliary plots

We have provided the detailed hyper-parameters and the learning curves in Appendix C.4 and C.9 respectively.

W3: Limitation of the current work.

In this work, our main focus is to remove the traditional assumption of behavior quality in auxiliary data and leverage the transition information to assist in imitation learning. It is worth noting that our approach may be conservative, particularly when high-quality data is available in the dataset. However, as you suggested, we can further enhance the performance of BCDP by combining previous reward annotation methods. We acknowledge that our exploration in this area is currently limited, we will combine these two different approaches of utilizing auxiliary data in future research to achieve a better balance between performance and robustness. Moreover, this study currently only considers the model-free solution. Exploring how to integrate existing model-based techniques to guide the agent toward expert-observed states is also a promising direction that we will attempt in future work.

W4: Typos and notation issues

Thanks for your suggestions, we have revised them in the new version.

W1 and Q1: Motivation and the details of Figure 1.

Thanks for your comments. However, it looks like you have some misconceptions about motivation. We clarify our motivation in the following:

The most recent successes in offline imitation learning heavily rely on high-quality behavior auxiliary data. For example, previous studies either manually integrate expert data into sub-optimal data or introduce noise into expert data to construct the auxiliary unlabeled dataset [1-4]. Unfortunately, the high-quality behavior data assumption neither holds in real-world applications nor in the benchmark datasets (e.g., D4RL). Facing low-quality behavior data, existing methods suffer severe performance degradation (as shown in Figure 1). How to exploit the easily obtained low-quality behavior data is a realistic yet unaddressed problem. We are the first ones to attempt to remove the high-quality behavior auxiliary data assumption for offline imitation learning and make a successful step. We believe this sheds new light on the study of offline imitation learning.

[1] Behavioral cloning from noisy demonstrations. ICLR 2021
[2] Demodice: Offline imitation learning with supplementary imperfect demonstrations. ICLR 2022
[3] Discriminator-weighted offline imitation learning from suboptimal demonstrations. ICML 2022
[4] Imitation learning from imperfection: theoretical justifications and algorithms. NeurIPS 2023

In Figure 1, we conducted a motivation evaluation on the maze2d task to observe the performance curve of the algorithm when we reduce the number of expert trajectories in the auxiliary data or even when it only contains low-quality behavior data (x-axis = 0). As a result, we found the performance of DWBC (a SOTA method) degrades significantly as the number of expert trajectories in the unlabeled data decreases. These findings further support the viewpoint that the previous methods may not effectively utilize low-quality data. The experimental details of Figure 1 have been added to Appendix C.2.

W2 and Q2: Comparison between BCDP and TD3+BC

In this work, we formalize the utilization of low-quality data in offline imitation learning as an expert-state distribution maximization problem and it can be instantiated using different dynamic programming methods. We chose TD3 because it is concise and stable. This does not mean that BCDP is a special case of TD3+BC. The difference arises from two completely different purposes. The behavioral cloning in the original TD3+BC is for conservatism, to keep the learned policy close to the offline policy and eliminate the estimation bias of TD3. Our goal with behavior cloning is to follow the imitation learning of expert data. Our TD3 (DP part) on the offline dataset ($D^O \cup D^E$) is to enable the agent to transition from expert-unobserved states to expert-observed states. Compared to previous imitation learning methods, this provides a new direction for the safe utilization of low-quality data, that is, using their transition information to help imitation learning.

We actually compared it with the TD3+BC in all settings. As we claimed in the raw paper: “UDS (Yu et al., 2022) labels all rewards from the unlabeled datasets with 0 and utilizes offline reinforcement learning algorithms to train the agent on the merged dataset. We have selected TD3+BC as our most similar offline RL algorithm, which allows it to be considered as an ablation study of our approach.” In all experiments, the UDS represents the raw TD3+BC in the offline imitation learning setting, as an ablation. In the offline imitation learning setting, where all data is unlabeled, applying offline RL algorithms on the union dataset is equivalent to UDS (Unlabeled Data Sharing), where expert data has reward label 1 and auxiliary data has reward label 0. As shown in the original paper's results and corresponding discussions, TD3+BC performs unsatisfactorily when the behavior quality of offline data is low.

W3 and Q3: Comparision with more offline RL methods

Thank you for your interest in comparing BCDP with more offline RL methods. We provide here a comparison with some popular offline RL methods, including CQL, IQL, and BEAR. Furthermore, we can also implement different (offline) RL methods in the BCDP framework, as an alternative to TD3.

From the results, we could find that our BCDP has scalability for different offline reinforcement learning methods. Compared to directly applying offline reinforcement learning methods (UDS), BCDP brings consistent performance improvement. These results further demonstrate the scalability of our BCDP. We have added these results to Appendix C.7.

W4 and Q4: OOD Evaluation

We answered Q3 (Q3 in the main paper) in two parts: 1) Whether the learned agent tends to move towards expert-observed states when it is in expert-unobserved states, along with the optimization objectives we provided. 2) Whether BCDP improves the agent's long-term returns on expert-unobserved states. We addressed them in Figure 4(a) and (b).

(1) DRG serves as a characterization of an agent's transition from one state $s$ to another state $s’$, with respect to the 1-NN distance from expert observed states, as defined by its definition. Specifically, $min_{s_1\in D^E} \lVert s-s_1 \rVert_2$ represents the 1-NN distance from state s to the expert dataset $D^E$ and $min_{s_2\in D^E} \lVert s’-s_2 \rVert_2$ is similar, where $s’$ represents the expected next state given policy $\pi$. The evaluation is performed on navigation in the maze2d. Therefore, the Euclidean distance can serve as an approximate measure for state distances. If DRG(s) > 0, it means that the agent will transition from state $s$ to a state $s'$ that is closer to the expert-observed states. Figure 4(a) illustrates that the BCDP actually has a positive expected DRG and tends to the expert-observed states, particularly in states with larger out-of-distribution (OOD) distances (indicated by a bigger value on the x-axis).

(2) As you pointed out, DRG does not directly quantify the performance of policy in unobserved states. Therefore, we also presented the long-term return in Figure 4(b). The Long-Term-Return ($s$) represents the sum of the agent's subsequent 400 steps starting from state $s$, calculated through 1000 random repetitions. We report the Long-Term-Return($s$) in Figure 4(b). Please note that Figure 4(b) is aligned with Figure 4(a), indicating that the positive DRG actually brings higher long-term returns for our agent. These results support that we have implemented our proposal, which is to leverage transition information from low-quality behavior data and improve the long-term return on expert-unobserved states, aligning with the theoretical motivation.

W5: Writing and Presentation
Thanks for your kind suggestions. We have revised them in the new version.

Thank you for your reply and for giving us the opportunity to clarify some misconceptions.

Response to motivation

In offline imitation learning with auxiliary data, there are expert dataset $D^E$ and auxiliary offline dataset $D^O$. Indeed, for general offline imitation learning methods, the return decreases if the number of expert trajectories in $D^O$ is reduced. However, a notable fact in Figure 1 is that the performance of DWBC (which works on $D^E \cup D^O$) degrades to being weaker than BC on $D^E$. This contradicts the goal in this field of utilizing auxiliary data as an enhancement to $D^E$.

Thanks for your interest in our BCDP. We have provided the corresponding results under the same settings as shown below.

From the results, we can find that BCDP can efficiently utilize auxiliary data $D^O$ from low-quality behavioral policy and consistently outperform BC, achieving the goal of using an offline auxiliary dataset to assist limited expert data $D^E$. Additionally, we can observe that BCDP achieved a performance of 72.4 in $D^O$ even without expert trajectories, surpassing the performance of DWBC with a mixture of expert trajectories in auxiliary data. This highlights the potential of low-quality behavior data, which was overlooked in previous OIL research, and may pave the way for future advancements in the field (as reviewer Yds8 said).

Response to Practical Implementation

Regarding the comment mentioned, "Authors mention that BCDP abstracts imitation from dynamic programming for conservatism and in-distribution transitions respectively," we would like to clarify that this statement is not accurate.
As presented in section 3, we follow the classic imitation learning approach by using BC to imitate the expert data $D^E$. When BC guarantees the return on expert-observed states (as claimed in our lemma 1), we further point out that the transition information in the auxiliary data is independent of the behavior quality, even if it comes from random exploration. Therefore, we propose the objective of maximizing expert-state distribution and implementing it through DP / TD algorithms. This is a completely different purpose from offline RL, such as TD3+BC, where BC is used to keep the learned policy close to the offline policy and eliminate the estimation bias of TD3.

Let's consider a specific scenario as an example (Figure 2). In the navigation task, we have an expert trajectory ($D^E$) that can reach the goal and a large amount of random exploration on the map as auxiliary data ($D^O$), which means uniformly random actions on many states. When applying TD3+BC on the union dataset, its BC term limits the agent to learn a uniformly random action and then fail. In contrast, our BC only applies to expert data $D^E$, ensuring the ability in expert-observed states. TD3 maximizes expert-state distribution and enables the agent to transition from expert-unobserved states to expert-observed states (as shown in Figure 2).

We have already pointed out this in Section 3.2, which makes TD3+BC difficult to adapt to low-quality auxiliary data. Extending to other Offline RL algorithms, their conservative regularization limits their effectiveness in offline imitation learning problems. The comprehensive experimental results also supported this point. The corresponding explanations are provided in Section 4.1 and Appendix C.7 and C.8.

Overall, our work bridges the offline IL and offline RL, providing a guiding framework for introducing existing mature RL algorithms in offline IL. More importantly, it brings a new perspective to the existing offline IL, which mainly focuses on the utilization of expert-similar samples, showing that low-quality behavioral data could also help with imitation learning.

Thanks for your positive comments and suggestions.

Weaknesses: missing citations

The discussion of these two works has already been added to the revised version.
The discussion about BC-SAC [2] has been placed in section 3.3. The discussion about MILO [1] has been placed in Appendix B, where we provide a review of previous model-based methods and discuss our differences from them.

[1] Mitigating Covariate Shift in Imitation Learning via Offline Data with Partial Coverage. NeurIPS’21
[2] Imitation is not Enough: Robustifying Imitation Learning with Reinforcement Learning for Challenging Driving Scenarios. IROS’23

Thanks for your positive comments and valuable suggestions. Below are our responses to the concerns.

Q1: Details of data quality.

We introduced the data acquisition in Section 4.1, most of which comes directly from the D4RL benchmark. In addition to low-quality random exploration, the experiments also include multi-task data (umaze-v1, medium-v1, large-v1), early-stopped policy (-medium-v2), and human demonstrations (human-v1). We also provided quantitative comparisons in Appendix C.1.

Q2: Explanations on non-random and human datasets.

In this paper, we mainly focus on whether it is possible to remove previous assumptions on high-quality behavioral data and attempt to take advantage of the benefits of low-quality data. When the data contains some high-quality behavioral data, our method only uses the transition information in the data. This may be conservative, so its performance improvement is not as significant as in the setting with purely low-quality data. Fortunately, as suggested by reviewer uGnS, in this case, we can also combine previous research, such as DWBC or OTIL, to identify potential expert-similar data and further enhance performance. The corresponding experimental results are provided in Appendix C.6.

In human data, the number of auxiliary data is limited (25 trajectories per task), which makes the transition information they can provide relatively limited. Therefore, performance improvement brought by BCDP is not very significant.

Weakness: One minor issue: Table 1: the second-best result is not underlined.

We have revised it in the new version.