Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration

Thanks for your detailed review and insightful comments. We especially appreciate the comments on the motivation. For your concern on the novelty of our method, we believe there might be a misunderstanding of our method – we believe that what we were describing is actually one of our baselines, “Trajectory Skills”. In our response, we explain in detail that we are not just a naive extension of OPAL combined with UCB (that is the baseline “Trajectory Skills”) and the differentiating step in our method makes a huge difference in performance from this naive extension as demonstrated in our experiments.

"The proposed method can be considered as an extension of OPAL (but online version), combined with a UCB approach. In other words, from a technical perspective, it seems equivalent to applying a UCB-style reward to OPAL and training it in an online setting."

We believe there is a misunderstanding. Our method is not equivalent to an extension of OPAL combined with a UCB approach. Rather, our “Trajectory Skills” baseline is equivalent to an online version of OPAL combined with a UCB approach from a technical perspective. Our “Trajectory Skills” baseline pretrains skills the same way as OPAL and learns a high-level online RL agent with a UCB approach to encourage exploration. The only, yet very important difference that distinguishes SUPE from this baseline is the pseudo-relabeling of the offline data that allows it to be used as additional off-policy data during online learning. Without the relabeling, we would not be able to leverage the offline data as additional off-policy data for online learning. We showed in our paper that using the offline data this way online results in a consistent/large performance boost over the “Trajectory Skills” baseline (Figure 2 and 3).

"It remains unclear how common in the real world the special case of offline data being un-labeled but online interactions being rewarded is."

In robotics, having access to task specific data can be expensive as it often requires human demonstration (e.g., through teleoperation or carefully scripted policy). It is often much easier to have access to a diverse dataset that is not directly related to the downstream task and use unsupervised RL algorithms to pretrain on this diverse dataset, and then fine-tune on a downstream task. This paradigm also started to gain popularity in RL in recent years [1, 2, 3, 4, 5]. In particular, O2O [1], FB [5] and ExPLORe [4] all study the setting where the offline data is unlabeled but the reward information is available in the online phase. O2O and FB assume the reward function is available right from the beginning of the online phase, whereas ExPLORe and our paper assumes the reward signal can only be obtained from online interactions. While the relevance of this problem setting merits discussion (which we will add to our paper), we believe that the number of prior works on this in just the last few years suggest that the topic is of interest to the ML community.

[1] "Unsupervised-to-Online Reinforcement Learning"

[2] "Fast Imitation via Behavior Foundation Models"

[3] "Reinforcement learning from passive data via latent intentions"

[4] "Accelerating exploration with unlabeled prior data"

[5] "Learning one representation to optimize all rewards"

“Why do the HILP-based methods achieve a non-zero return right from the start?”

The data distribution for D4RL Kitchen is high quality, and BC methods can achieve some non-zero return [1]. Thus, since the HILP skills are effective at mimicking dataset behavior, sampling HILP skills can potentially achieve nonzero return prior to online exploration.

[1] D4RL: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.

"What are the pros and cons of SUPE into HILP and the vanilla version, and what insights can you offer on which one should be chosen depending on the environment?"

From current empirical observation, SUPE appears to be more stable during offline learning, and excels at long-horizon locomotion tasks (ie. $`visual-antmaze`$, $`antmaze`$, $`humanoidmaze`$). SUPE (HILP) is competitive for some manipulation tasks ($`antsoccer`$, $`scene`$), and achieves higher initial performance in some environments.

We would like to thank you again for your constructive feedback and detailed reviews. Please let us know if you have any other concerns or questions. If we have successfully addressed all your concerns, could you kindly raise your rating?

Thanks for your detailed review and insightful comments. We especially appreciate the additional references you point out and the clarifying question on pseudo-labeling. For your question on how policy operates conditioned on different latent skills, we provide an additional detailed analysis that visualizes the skill latent and show how leveraging skills effectively reduces the exploration horizon, resulting in learning speedup online.

"I suggest adding visualizations, such as those in Figure 4 of BeCL [Yang et al., 2023], to illustrate how the policy operates conditioned on different latent skills"

We conducted additional analysis of the skill latent to demonstrate how skill discovery helps online exploration in one of our hardest domains, $`humanoidmaze`$. In particular, we randomly sample 16 latent vectors from our skill latent and roll-out our pre-trained low-level skill policy (corresponding to each of the sampled latent vectors) for an entire episode and visualize the x-y position throughout each of the 16 trajectories. For comparison, we also plotted the agent’s trajectories if actions were completely random. As shown in the figures here, our unsupervised offline pretrained skills were able to navigate quite far from the initial positions even before any online learning. Such structured navigating behaviors allow the high-level policy to effectively operate at a reduced exploration horizon. Instead of training a low-level agent to predict $H$ correct actions in a row, the high-level policy only needs to predict 1 correct action to achieve a similar effect. As the exploration horizon is effectively reduced by a factor of $H$, we are able to train the high-level policy in SUPE to explore more efficiently and consequently solve the task much more quickly.

"Can you explain why skill extraction via pseudo-labeling provides a significant advantage over directly using offline data for training?"

The pseudo-labeling enables us to use offline data as additional off-policy data for the online training of the high-level agent. The high-level agent operates at a longer time-scale – taking only one high-level action ($z$) per 4 environment steps. Pseudo-labeling skills allows us to convert the low-level trajectories in the offline data $(s_t, a_t, s_{t+1}, a_{t+1}, \cdots)$ into high-level trajectories $(s_t, z_t, s_{t+H}, z_{t+H}, \cdots)$. Without pseudo-labeling, we would not be able to directly use the offline data to update the high-level RL agent online, which means that we could no longer use the offline data twice.

"Table 1 shows that the KL coefficient and GRU layers vary across environments rather than being consistent. Can you explain the rationale for these differences?"

The OGBench ($`humanoidmaze`$, $`antsoccer`$, $`scene`$, $`cube-single`$, $`cube-double`$) tasks are generally more difficult, so we use larger networks than for the simpler D4RL tasks. We select our network sizes for these OGBench environments according to the reference policy size used in the original OGBench paper [1]. For the manipulation tasks $`cube-*`$ and $`scene`$, the data distribution is narrower, and we found that a higher KL coefficient helps account for the difference in dataset diversity.

[1] Park, Seohong, et al. "OGBench: Benchmarking offline goal-conditioned RL." arXiv preprint arXiv:2410.20092 (2024).

Related work references

Thanks for the additional references on unsupervised RL. We will incorporate them in our discussion in the paper and update in the camera ready version.

We would like to thank you again for your constructive feedback and detailed reviews especially on the additional related work references. Please let us know if you have any other concerns or questions. If we have successfully addressed all your concerns, could you kindly raise your rating?

Thanks for your detailed review and insightful comments. We especially appreciate the additional references you point out and the clarifying question on UCB. For your concern on the dependence on quality of the offline data, we had ablation studies in our appendix that demonstrate the robustness of our approach on offline datasets with different levels of quality. For your question on how skill discovery helps, we provide an additional detailed analysis that visualizes the skill latent and show how leveraging skills effectively reduces the exploration horizon, resulting in a learning speedup online.

Analysis on offline datasets with different coverage and failure modes

We had ablation studies on datasets with different levels of quality in the Appendix J and showed our approach is robust against different dataset corruptions. A notable failure mode is the $`explore`$ dataset where the dataset contains completely random actions, and where the skill-pretraining methods we use in SUPE cannot learn meaningful behaviors. Aside from this failure case, SUPE is able to learn efficiently from offline datasets with limited coverage (e.g., missing transitions around the goal location, Figure 19, right), limited scale (e.g., 95% of trajectories removed, Figure 19, left), and suboptimal data (e.g., the $`stitch`$ dataset, Figure 18). We will refer to these ablation studies and discuss the failure modes of our approach in the main paper in the camera ready version.

Detailed analysis on skill discovery

We conducted additional analysis of the skill latent to demonstrate how skill discovery helps online exploration in one of our hardest domains, $`humanoidmaze`$. In particular, we randomly sample 16 latent vectors from our skill latent and roll-out the corresponding low-level skill policy for an entire episode and visualize the x-y position throughout each of the 16 trajectories. For comparison, we also plotted the agent’s trajectories if actions were completely random. The graphs can be viewed here. We can see that fixed skill latents are able to navigate quite far from the starting point. Such structured navigating behaviors allow the high-level policy to effectively operate at a reduced exploration horizon. Instead of training a low-level agent to predict $H$ correct actions in a row, the high-level policy only needs to predict 1 correct action to achieve a similar effect. As the exploration horizon is effectively reduced by a factor of $H$, we are able to train the high-level policy in SUPE to explore more efficiently and solve the task much faster.

Additional references

Thanks for the additional references. In our paper, we actually do have experiments using ICVF (Ghosh et al., 2023) on the high-dimensional image domain in Appendix E. ICVF is used complementary to our proposed method and our baselines. In our experiments (Figure 6), we found that our method works well without ICVF, achieving a much better performance than our baseline “Trajectory Skills” with or without the ICVF pre-trained representations. We also found that using ICVF representations further improves our method but only marginally. Even with ICVF, many of our baselines still failed to achieve significant return (e.g., ExPLORe as shown in Figure 6, right).

Conceptually, both VIP (Ma et al., 2022) and ICVF are very related but different from our work in two ways: 1) they focus on pre-training on observation-only offline data whereas we assume access to actions, 2) they focus on extracting good image representations from the offline pre-training, whereas we pretrain low-level skills and retain the offline data to be used as additional off-policy transitions for online learning.

Both (Sharma et al., 2022) and (Song et al., 2024) are also relevant and we will cite and discuss them (along with the two above) in the camera ready version of our paper.

Question about UCB

In the RL literature, UCB is typically used on the optimal value function with a confidence interval (e.g., [1]). We take the same definition of the term UCB in our paper and to describe the upper-confidence bound $r_{\mathrm{UCB}}$ of a reward function – with high probability the reward value is at most $r_{\mathrm{UCB}}$. As what we stated in our method section, we followed prior work (Li et al., 2024) to instantiate this estimate of the bound using an RND network and a reward model. The reward model predicts the mean of the reward estimate and the RND provides an estimate on how wide the confidence interval is.

[1] "Minimax regret bounds for reinforcement learning."

We would like to thank you again for your constructive feedback and detailed reviews especially on the related work references. Please let us know if you have any other concerns or questions. If we have successfully addressed all your concerns, could you kindly raise your rating?

Thanks for your detailed review and insightful comments. We especially appreciate your clarifying question on the setting of our work. Regarding your concern on not comparing to “diversity-encouraging” objectives, we conducted additional experiments with METRA and showed that METRA finds a reward signal online much slower than our method. In addition, our experiments already involved a comparison to a variant of our method that leverages “diversity-encouraging” objectives in the offline pre-training phase (SUPE (HILP)). For your concern about the dependence on offline data quality, our ablation studies in the appendix of our original submission demonstrated the robustness of our approach to offline datasets with different levels of quality, quantity, and coverage.

Experiments with “diversity-encouraging” methods

We conducted additional experiments to evaluate how fast a diversity-encourage objective (METRA) is able to find the reward signal (reach the desired goal) online. In the table below, we report the # of env steps that METRA took before the learned skills could reach the desired goal from the initial state (6 seeds). Our method took significantly fewer environment steps (on the order of 1/100x) compared to METRA. These results suggest that the $`antmaze`$ domain poses a significant exploration challenge, and naively using a diversity-encouraging objective (METRA) for online exploration can be extremely sample inefficient (100x worse than our method). The table can be viewed here.

These results may not be too surprising, since METRA does not leverage offline data. One may wonder what might happen if we adapt METRA to use offline data. This is actually quite similar to SUPE (HILP), a variant of our method which uses a “diversity-encouraging” objective for offline pretraining instead of a behavioral cloning objective. Similar to METRA, HILP learns a metric space of states that preserves temporal distance. SUPE (HILP) is otherwise identical to SUPE (e.g., uses the offline data for both skill pretraining and high-level agent learning online) with the only difference being that the offline skill pretraining objective is different. SUPE works better across six of the eight domains we experimented on, and is competitive with SUPE (HILP) on the other two.

Our method is not tied to a specific choice of skill pre-training method or online exploration bonus. We used VAE behavioral cloning skill pretraining and an RND exploration bonus because they are simple and effective in the benchmark tasks considered.

Does SUPE only work on very high quality data?

In Appendix J of our submission, we had some ablation studies on datasets with different levels of quality. In Figure 18&19, we show that our approach is reasonably robust against various dataset corruptions (e.g., insufficient coverage around the goal location, noisy experts with short trajectory lengths). In addition, our approach can be easily adapted to work in conjunction with different skill pre-training methods. On datasets that are diverse and “high-coverage”, we can leverage pre-trained skills that are more suitable for these properties (e.g., HILP) [1]. As mentioned in the previous section, one variant of our approach (SUPE (HILP)) leverages such skills.

[1] "Foundation policies with hilbert representations."

“What's the difference between this setting and (Kim et al., 2024)? Could you motivate in what real problems this is useful?”

Your understanding is correct. In the O2O paper, the reward function is available at the start of the online phase. Our setting is harder since we don’t have access to the reward function and the reward signals must be obtained through environment interactions.

In robotics, manually specifying a good reward function is challenging and it often suffers from misspecification issues (when the optimal policy of the specified reward misaligns with the expected optimal policy). For example, if you would like to specify a reward function for a home robot to clean a house, there are a lot of things to consider: should the robot clean the kitchen? What if the robot bumps into a wall? How do we define if cleanliness is acceptable? A much more appealing interaction with the robot would be that the robot first does some exploratory cleaning behavior (informed by its prior experience, possibly from its prior cleaning experience in another house) and then the user gives feedback (sparse reward signal) to the robot on whether the cleaning is done properly in a safe manner, and the robot improves upon the signal provided. We believe our work is an important first step towards making this part of our reality in the future.

Thank you again for your constructive feedback and detailed reviews. Please let us know if you have any other concerns or questions. If we have successfully addressed all your concerns, could you kindly raise your rating?