Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration

Can expert data lead to better skills?

We conducted an additional set of experiments that use offline data with different qualities and reported the results in Figure 16. Among these datasets, the first one is an expert dataset collected by rolling out an expert policy. We find that our method does not have advantage over non-skill-based methods (e.g., ExPLORe) and even learns worse than the setting where a more diverse dataset is provided. We hypothesize that it is because completely expert data is very narrow and can lead to a very narrow set of pre-trained skills that may not behave well in states that are out of the distribution of the offline data. This in turn can harm online learning as the high-level policy can have trouble finding low-level skills that can correct the agent from these out-of-distribution states.

Hope these address all of your questions and concerns. Thank you again for your time to review and if you have any remaining questions or concerns, please let us know!

Tasks are simplistic and monotonous.

We show additional results on four new domains (23 new tasks in total!) with two locomotion domains and two manipulation domains in Figure 3. Each of these domains contains 2-5 tasks. We selected these environments from OGBench [1], since they provided challenging, long-horizon tasks which also require exploration to solve, making them well-suited for testing the use of offline data to accelerate online exploration. The two manipulation domains (Cube and Scene) contain different tasks that are long-horizon and require composition of multiple skills by design. For example, one of the tasks in Scene requires the robotic arm to 1) press a button to unlock a drawer, 2) open the unlocked drawer, and 3) pick an object and place the object in the drawer, and 4) close the drawer. The two additional locomotion domains are HumanoidMaze and AntSoccer. HumanoidMaze is more difficult than AntMaze, since it involves controlling a 21-DoF humanoid agent. The tasks have a significantly longer time horizon, with the giant maze requiring up to 4000 environment steps. AntSoccer is also much harder than AntMaze, since the ant needs to navigate to a soccer ball and then dribble it to the goal location.

In the challenging HumanoidMaze domain, our method is often the only method that achieves non-zero success rate on the four most difficult mazes. On manipulation tasks, our method consistently outperforms all prior methods on all domains with the only exception on Scene where one of the baselines (Offline w/ HILP) performs better. It is worth noting that Offline w/ HILP is a baseline that we introduced to also leverage offline data twice, both during offline and online learning with the only difference being that the unsupervised skill pre-training algorithm is HILP (instead of using trajectory VAE). This further demonstrates that the principle of leveraging offline data for both skill pre-training and online learning is effective. The effectiveness of our method across seven domains further highlights the importance of a careful combination of skill pre-training and effective online learning that utilizes the offline data.

We would be happy to add additional experiments in the final version if there are specific benchmark tasks that the reviewer could suggest that they believe to be a better test for the method.

[1] Park, Seohong, et al. "OGBench: Benchmarking Offline Goal-Conditioned RL." arXiv preprint arXiv:2410.20092 (2024).

Why isn’t SPiRL used as a baseline for comparison?

We already compare our method to an improved version of SPiRL. Our baseline “Online w/ Traj. Skill” in Figure 3 is an implementation of SPiRL but with additional improvements such as the addition of RND reward bonus, as well as replacing the KL constraint on the prior with entropy regularization in SAC. These improvements are crucial for online learning to work well in the challenging domains that we experiment with. This can be seen in our ablations for each of these improvements on AntMaze Large (Figure 6: KL, Figure 4, left: RND). We have made additional clarifications in our paper to reflect this.

How does trajectory segment length affect performance?

We included a set of new sensitivity analysis results in our paper on the AntMaze-Large (Top-right goal). Figure 4, right shows how the performance of our method changes as we increase and decrease the trajectory segment length (skill horizon length), H. When we decreased the length to 2, we found that our method can actually achieve an even higher final performance at the cost of slower initial learning. This is likely due to the fact that having shorter skills in the beginning makes exploration less structured, slowing down learning. Shorter skills allow the high-level policy to stitch them more optimally, improving the final performance. When we increased the skill horizon length to 8, we found that our method can still solve the task, but much more slowly. We used a constant skill length of 4 for all our experiments and we found it to work well across all the domains we tested.

Thanks for the detailed feedback and insightful comments! For your concern on the novelty of our method, we would like to highlight that important, careful design decisions in our current method enable significant performance gains over baselines whereas the naive combination of prior works falls short. For your concern on the current domains being too monotonous and simplistic, we additionally evaluated our method on four additional domains that are much more challenging and diverse than the three domains in our initial submission. We showed that our method exhibits similar performance gains on most domains. In the HumanoidMaze domain, our method is the only method that can solve all of the tasks.

Novelty.

We would like to emphasize several key design decisions in our current method that are different from prior methods, which contribute to the performance gains over baselines.

First of all, all prior work on online learning with skills extracted from offline data simply discards the offline data when learning the high-level policy (e.g., Pertsch et al. (2021), Ajay et al. (2020)). In our experiments, the baseline “Online w/ Traj. Skill” does exactly this (learning trajectory skills from offline data, then learning a high level policy purely from online samples), and is consistently worse than our method that utilizes offline data during online learning (especially on more challenging tasks like the Large and Ultra AntMaze environments in Figure 8 and on all HumanoidMaze tasks in Figure 13).

In addition, our method is not a naive combination of SPiRL (Pertsch et al. (2021)) and ExPLORe (Li et al. (2024)). Pertsch et al. (2021) use a KL constraint between the high-level policy and a state-dependent prior obtained from offline pretraining. We found this design can actually hurt the online performance. We show that a simpler design without the KL constraint works much better. In Figure 6, we compare (Ours (KL)) with our method (Ours) and demonstrate that the final performance and the sample efficiency of the naive combination is much worse. As described in Appendix E, we borrow the policy parameterization from Haarnoja et al. (2018) and adopt a tanh policy parameterization with entropy regularization on the squashed space. Such design ensures that the online high-level policy is not explicitly constrained to a pre-trained prior, allowing the online policy to learn skill distributions that are more suitable for the task online. It also allows the addition of entropy regularization to the high level policy, which helps exploration.

These careful designs are what make our method extremely stable, sample efficient, and scalable to more complex tasks.

How does your method compare to using offline-to-online RL methods which have access to reward labels?

In Figure 7, we show a comparison between our method and two state-of-the-art offline-to-online methods (CalQL [1] and IDQL [2]) on AntMaze-large (top-right goal). We also include a version of our method (Ours (Ground Truth)) where we assume access to the ground truth reward similar to all the offline-to-online RL methods. While Ours performs slightly worse than CalQL since we do not assume access to the offline reward, Ours (Ground Truth) performs better than CalQL with a much faster initial learning thanks to structured exploration using pre-trained skills.

[1] Nakamoto, Mitsuhiko, et al. "Cal-QL: Calibrated offline RL pre-training for efficient online fine-tuning." Advances in Neural Information Processing Systems 36 (2024).

[2] Hansen-Estruch, Philippe, et al. "IDQL: Implicit Q-learning as an actor-critic method with diffusion policies." arXiv preprint arXiv:2304.10573 (2023)

Questions

What should I take from the coverage results?

Since we DO NOT have reward labels in the dataset and the reward information is NOT given to the agent in the beginning of the online phase, on AntMaze, the agent must find the goal first before it can learn to reach it consistently. The coverage metric provides information on how quickly the agent is able to explore the maze. However, we do agree that the coverage plot adds little in addition to Table 1 (which already provides insights on how quickly each method can find the goal). We have moved it to the Appendix in the interest of space.

What do the bold numbers represent in Table 1?

The table values are bolded if their confidence intervals overlap with the confidence interval of the best method (determined by the mean value) in each row. We included additional clarification in our paper.

Typos and minor comments

Thanks for the detailed comments! We have fixed all the typos mentioned. We also moved the last paragraph of Section 5.3 to the beginning of Section 5.4. We also added a clarification in the paper about what the solid line indicates. It represents the mean value across seeds. We also improved all the figures such that the colors of the legend text match the line colors.

Hope these address all of your concerns. Thank you again for your time to review and if you have any remaining questions or concerns, please let us know!

Thanks for the detailed feedback and insightful comments! We addressed your concern on sensitivity of the hyperparameter by adding a new section in our experimental results that is dedicated to sensitivity analysis of various hyperparameters and how they were chosen (RND coefficient, skill horizon length). We also provided additional results on offline data with different qualities (expert/exploratory, noisy/non-noisy) with more insights on when our method is expected to work or not work depending on the dataset. Finally, we demonstrated that our method, when given access to ground truth offline reward, can outperform the state-of-the-art offline-to-online RL methods, showcasing the effectiveness of our method at leveraging structured exploration with skills.

How was the RND coefficient $\alpha$ set?

We did not find the performance of our approach to be sensitive to the RND coefficient. We ran a small hyperparameter tuning sweep on AntMaze-Large (top-right goal) and displayed the results in Figure 4, left. The performance remains almost the same for $\alpha \in \{2, 4, 8, 16\}$. We picked one of these values ($\alpha=8$) and used the SAME value for all other tasks. All the non-skill-based methods use a RND coefficient that is $4\times$ smaller to keep the reward scale consistent with the skill-based methods. This is because the transitions in skill-based methods have a reward value that is equal to the sum of the rewards in 4 low-level transitions (H=4 is the skill horizon length).

Additional datasets for AntMaze

We include the results for the play datasets for the medium maze layout and the large maze layout (Figure 12). The results are consistent with our results on the diverse datasets.

In addition, we experimented on a narrow, expert dataset and three other datasets used in an offline goal-conditioned RL benchmark (OGBench [1]), shown in Figure 16. We picked datasets from this benchmark as it features a more diverse set of offline data distributions and hoped that it would provide more insights on when our method works or fails. As a summary, we find that our method does not need a dataset that is collected by unsupervised RL agents. Completely exploratory dataset can actually break our method due to the lack of behavioral structure that can be extracted as skills. Our method excels at learning from datasets that contain segments of meaningful (e.g., navigating around the maze) behaviors. We discuss the results in detail below.

The four datasets we consider are ordered in decreasing difficulty:

• Expert: collected by a non-noisy expert policy that we train ourselves.
• Navigate: collected by a noisy expert policy that randomly navigates the maze (from OGBench).
• Stitch: collected by the same noisy expert policy but with much shorter trajectory length (from OGBench)
• Explore: collected by moving the ant in random directions, where the direction is re-sampled every 10 environment steps. A large amount of action noise is also added (from OGBench).

As expected, the baseline ExPLORe shows a gradual performance degradation from Expert to Navigate, to Stitch, and to Explore. All skill-based methods (including our method) fail completely on Explore. This is to be expected because the Explore dataset contains too much noise and the skills extracted from the dataset are likely very poor and meaningless. The high-level policy then would have trouble composing these bad skills to perform well in the environment. On Navigate and Stitch, our method outperforms other baselines, especially on the more challenging Stitch dataset where it is essential to stitch shorter trajectory segments together. On Expert, all methods perform similarly with ExPLORe doing slightly better. We hypothesize that this is because with the expert data, online learning does not require as much exploration, and skill-based methods are mostly beneficial when there is a need for structured exploratory behaviors.

To further test our method’s ability to handle different offline data quality, in our original submission, we included ablation studies where the offline data is corrupted. We tested a dataset without transitions near the goal location (Insufficient Coverage), and a low-data setting where 95% of the trajectories are removed from the dataset (5% Data). While we see performance degradation from such data erasure, our method is still the most robust, consistently outperforming the baselines (Figure 17).

We hope that these additional experiments provide more insights on when we expect our skill-based method to work or not work.

[1] Park, Seohong, et al. "OGBench: Benchmarking Offline Goal-Conditioned RL." arXiv preprint arXiv:2410.20092 (2024).

Importance of Reusing Prior Data Twice

We would like to clarify that the baseline Online w/ Traj. Skill only uses the offline data for unsupervised skill pretraining and the baseline ExPLORe only uses the offline data during online learning and does not have a skill pre-training phase. In Figure 3, we show that none of these baseline methods that only use the offline data once can achieve good performance. Our method consistently beats these two baselines, demonstrating that reusing the prior data twice is crucial.

In addition, one of the baselines that we introduce in this work, Online w/ HILP Skills, also only uses the offline data for unsupervised skill pretraining. We apply the same principle of reusing the prior data twice to this baseline which leads to the HILP w/ Offline Data baseline. Across domains, HILP w/ Offline Data consistently outperforms Online w/ HILP Skills, further highlighting the benefits of reusing the prior data during online learning.

Trajectory-Segment VAE

Trajectory-segment VAE is a common design choice for extracting a latent space of skill policies offline adopted by a range of prior works and has shown effectiveness in accelerating RL [1-2]. While such a design is certainly not the only approach that can extract useful skills from offline data, it is the simplest formulation that we found to be effective. In addition, the trajectory encoder in the VAE allows to conveniently transform the offline data into high-level off-policy data such that they can be readily used by the actor-critic RL agent online as additional off-policy data, allowing us to use the offline data twice. In addition, the idea of “reusing the prior data twice” can be applied to potentially other unsupervised skill pre-training algorithms. In our work, we present one alternative where we use HILP [3], a recently proposed offline unsupervised skill pre-raining method and implement two baselines. The first baseline “Online w/ HILP Skill” is the naive version that does not use the prior data twice (the online learning does not use the offline data as additional off-policy data). The second baseline “HILP w/ Offline Data” is the version that does use the prior data twice and we observe that the second baseline (that uses the data twice) performs consistently better than the first baseline (that only uses the data once) across all the domains (Figure 3).

[1] Pertsch, Karl, Youngwoon Lee, and Joseph Lim. "Accelerating reinforcement learning with learned skill priors." Conference on robot learning. PMLR, 2021.

[2] Ajay, Anurag, et al. "Opal: Offline primitive discovery for accelerating offline reinforcement learning." arXiv preprint arXiv:2010.13611 (2020).

[3] Park, Seohong, Tobias Kreiman, and Sergey Levine. "Foundation policies with hilbert representations." arXiv preprint arXiv:2402.15567 (2024).

Hope these address all of your questions and concerns. Thank you again for your time to review and if you have any remaining questions or concerns, please let us know!

Thanks for the detailed feedback and insightful comments! For your concern on the novelty of our method, we would like to highlight that important, careful design decisions in our current method enable significant performance gains over baselines whereas the naive combination of prior works falls short. We additionally evaluated our method on four additional domains and our method exhibits similar performance gains. We provided clarification on how each of our baselines use prior data and showcase the effectiveness of the principle of “reusing prior data twice” on the original domains and the new four domains that we added in this rebuttal.

Novelty

We would like to emphasize several key design decisions in our current method that are different from prior methods, which contribute to the performance gains over baselines.

First of all, all prior work on online learning with skills extracted from offline data simply discards the offline data when learning the high-level policy (e.g., Pertsch et al. (2021), Ajay et al. (2020)). In our experiments, the baseline “Online w/ Traj. Skill” does exactly this (learning trajectory skills from offline data, then learning a high level policy purely from online samples), and is consistently worse than our method that utilizes offline data during online learning (especially on more challenging tasks like the Large and Ultra AntMaze environments in Figure 11 and on all HumanoidMaze tasks in Figure 16).

In addition, our method is not a naive combination of SPiRL (Pertsch et al. (2021)) and ExPLORe (Li et al. (2024)). Pertsch et al. (2021) use a KL constraint between the high-level policy and a state-dependent prior obtained from offline pretraining. We found this design can actually hurt the online performance. We show that a simpler design without the KL constraint works much better. In Figure 7, we compare (Ours (KL)) with our method (Ours) and demonstrate that the final performance and the sample efficiency of the naive combination is much worse. As described in Appendix E, we borrow the policy parameterization from Haarnoja et al. (2018) and adopt a tanh policy parameterization with entropy regularization on the squashed space. Such design ensures that the online high-level policy is not explicitly constrained to a pre-trained prior, allowing the online policy to learn skill distributions that are more suitable for the task online. It also allows the addition of entropy regularization to the high-level policy, which helps exploration.

These careful designs are what make our method extremely stable, sample efficient, and scalable to more complex tasks. We show additional results on four new domains with two locomotion domains and two manipulation domains in Figure 3. We selected these environments from OGBench [1], since they provided challenging, long-horizon tasks which also require exploration to solve, making them well-suited for testing the use of offline data to accelerate online exploration. In the challenging HumanoidMaze domain, our method is often the only method that achieves non-zero success rate on the four most difficult mazes. On manipulation tasks, our method consistently outperforms all prior methods on all domains with the only exception on Scene where one of the baselines (Offline w/ HILP) performs better. It is worth noting that Offline w/ HILP is a novel baseline that we introduced to also leverage offline data twice, both during offline and online learning with the only difference being that the unsupervised skill pre-training algorithm is HILP (instead of using trajectory VAE). This further demonstrates that the principle of leveraging offline data for both skill pre-training and online learning is effective. The effectiveness of our method across seven domains further highlights the importance of a careful combination of skill pre-training and effective online learning that utilizes the offline data.

[1] Park, Seohong, et al. "OGBench: Benchmarking Offline Goal-Conditioned RL." arXiv preprint arXiv:2410.20092 (2024).

Thanks for the detailed feedback and insightful comments! To address your questions on our design choices, we provided additional sensitivity analysis on design choices in our algorithm. We also provided more discussions on the contribution of the two uses of offline data. Hope these help provide more insights on how our method works.

Is the algorithm robust to different design choices? How important is the optimistic labelling?

We include additional sensitivity analysis on the skill horizon (H) and the RND coefficient (the amount of optimism added in optimistic labeling) using the AntMaze-Large (top-right goal) task.

Optimistic labeling is important for our method to successfully solve the task. In Figure 5, left, our method with no optimistic labeling ($\alpha=0$) completely fails to solve the task. However, the performance of our method is robust to the value of the RND coefficient as long as it is non-zero – the performance remains almost the same for $\alpha \in \\{2, 8, 16\\}$.

For the skill horizon (H), Figure 5, right shows how the performance of our method changes as we increase and decrease the trajectory segment length (skill horizon length), H. We find that while there is some variability across individual tasks (Appendix H, Figure 10), a skill horizon length of 4 generally performs the best. We used a constant skill length of 4 for all our experiments and we found it to work well across all the domains we tested.

Why can we use the offline data trajectories twice?

The use of offline data during offline pre-training and online learning are along two axes that complement each other. Offline pre-training leverages the short horizon behavioral structure in offline dataset whereas online learning leverages the more high-level dynamics information of the environment (e.g., how a state at the current time step $s_t$ may be transformed to a state $H$ steps in the future $s_{t+H}$ via high-level skills). The behavioral structure helps construct the skills and the high-level dynamics information helps stitching/composing these skills together at a higher-level. Since our high-level policy is an off-policy RL agent, it can consume off-policy high-level transitions directly from the offline data to help it stitch/compose the low-level skills. This can be further justified by observing that on the domains where stitching is needed less (e.g., Single Cube tasks and Kitchen-complete where the demonstration of completing the full task is directly available in the offline data), the gap between our method and the Online w/ Traj. Skills baseline (that does not use the offline data trajectories during online learning) is lower.

Hope these address all of your questions and concerns. Thank you again for your time to review and if you have any remaining questions or concerns, please let us know!

Dependence of Data Quality

To gain more insights on when we expect our method to work and the data quality dependency, we experimented on a narrow, expert dataset and three other datasets used in an offline goal-conditioned RL benchmark (OGBench [1]), shown in Figure 19. We picked datasets from this benchmark as it features a more diverse set of offline data distributions and hoped that it would provide more insights on when our method works or fails. As a summary, we find that our method does not need the dataset to be very diverse (e.g., collected by unsupervised RL agents). Completely exploratory dataset can actually break our method due to the lack of behavioral structure that can be extracted as skills. Our method excels at learning from datasets that contain segments of meaningful (e.g., navigating around the maze) behaviors. We discuss the results in detail below.

The four datasets we consider are ordered in decreasing difficulty:

• Expert: collected by a non-noisy expert policy that we train ourselves.
• Navigate: collected by a noisy expert policy that randomly navigates the maze (from OGBench).
• Stitch: collected by the same noisy expert policy but with much shorter trajectory length (from OGBench)
• Explore: collected by moving the ant in random directions, where the direction is re-sampled every 10 environment steps. A large amount of action noise is also added (from OGBench).

As expected, the baseline ExPLORe shows a gradual performance degradation from Expert to Navigate, to Stitch, and to Explore. All skill-based methods (including our method) fail completely on Explore. This is to be expected because the Explore dataset contains too much noise and the skills extracted from the dataset are likely very poor and meaningless. The high-level policy then would have trouble composing these bad skills to perform well in the environment. On Navigate and Stitch, our method outperforms other baselines, especially on the more challenging Stitch dataset where it is essential to stitch shorter trajectory segments together. On Expert, all methods perform similarly with ExPLORe doing slightly better. We hypothesize that this is because with the expert data, online learning does not require as much exploration, and skill-based methods are mostly beneficial when there is a need for structured exploratory behaviors.

To further test our method’s ability to handle different offline data quality, in our original submission, we included ablation studies where the offline data is corrupted. We tested a dataset without transitions near the goal location (Insufficient Coverage), and a low-data setting where 95% of the trajectories are removed from the dataset (5% Data). While we see performance degradation from such data erasure, our method is still the most robust, consistently outperforming the baselines (Figure 20).

We hope that these additional experiments provide more insights on when we expect our skill-based method to work or not work.

[1] Park, Seohong, et al. "OGBench: Benchmarking Offline Goal-Conditioned RL." arXiv preprint arXiv:2410.20092 (2024).

Training stability in HRL

Many instability problems in HRL methods stem from the fact that both the low-level policy and the high-level policy are learning simultaneously at the same time. We circumvent such issues by pre-training low-level skill policies offline using a static dataset and then keep them fixed during online learning. In addition, our method utilizes the offline data as additional off-policy data for the high-level actor-critic RL agent such that the high-level RL agent can sample high-level transitions from offline data to perform TD updates. This effectively increases the amount of data that the high-level RL agent has access to right from the beginning of online learning, further stabilizing high-level policy learning.

Hope these address all of your questions and concerns. Thank you again for your time to review and if you have any remaining questions or concerns, please let us know!

Limited discussion on HRL

Thanks for pointing out these related works that are really relevant to our work. We have added two new sections in the related works to discuss the relationship between our work and prior works in hierarchical RL and options framework (please see Section 2, “Hierarchical reinforcement learning” and “Options framework”). While some prior HRL methods simultaneously learn the low-level skill policies and the high-level policy online, others opt for a simpler formulation where the low-level skills are pre-trained offline and kept frozen during online learning. None of the prior HRL methods simultaneously leverage offline skill pre-training and offline data as additional off-policy data for high-level policy learning online. As we show in our experiments, both of them are crucial in enabling extremely sample efficient learning on challenging sparse-reward tasks, sometimes even solving tasks that all prior methods cannot (e.g., Figure 16 on HumanoidMaze).

It is also worth noting that we have already discussed OPAL [1] in the unsupervised skill discovery section in the related work in our initial submission. Our unsupervised skill pre-training implementation closely follows the implementation of OPAL [1] and SPiRL [2] where the latent skills are extracted using a VAE. In addition, [3] does not use reinforcement learning and only focuses on extracting primitives, so we integrated this reference into our “unsupervised skill discovery” paragraph in the related work instead.

[1] Ajay, Anurag, et al. "Opal: Offline primitive discovery for accelerating offline reinforcement learning." arXiv preprint arXiv:2010.13611 (2020).

[2] Pertsch, Karl, Youngwoon Lee, and Joseph Lim. "Accelerating reinforcement learning with learned skill priors." Conference on robot learning. PMLR, 2021.

[3] Paraschos, Alexandros, et al. "Probabilistic movement primitives." Advances in neural information processing systems 26 (2013).

High-level policy that is updated every 𝐻 timesteps and keeps the pre-trained skill and trajectory encoder fixed during the online phase. This limits the adaptability of the method.

While we acknowledge that this is indeed a limitation of our approach and using a more adaptive skill framework (e.g., options framework) can address this limitation (we add a new paragraph to discuss it in our related work section), our design (where the high-level policy outputs a skill at a regular interval) is a common design that appears in many prior methods [1-9] and many of them keep the skills fixed during online learning ([1-4]), and find it to be effective. In practice, we also find such design to be sufficient for a wide range of tasks (now with four new challenging domains in addition to the ones we tested in our initial submission).

[1] Dalal, Murtaza, Deepak Pathak, and Russ R. Salakhutdinov. "Accelerating robotic reinforcement learning via parameterized action primitives." Advances in Neural Information Processing Systems 34 (2021): 21847-21859.

[2] Gehring, Jonas, et al. "Hierarchical skills for efficient exploration." Advances in Neural Information Processing Systems 34 (2021): 11553-11564.

[3] Pertsch, Karl, Youngwoon Lee, and Joseph Lim. "Accelerating reinforcement learning with learned skill priors." Conference on robot learning. PMLR, 2021.

[4] Ajay, Anurag, et al. "Opal: Offline primitive discovery for accelerating offline reinforcement learning." arXiv preprint arXiv:2010.13611 (2020).

[5] Xie, Kevin, et al. "Latent skill planning for exploration and transfer." arXiv preprint arXiv:2011.13897 (2020).

[6] Gupta, Abhishek, et al. "Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning." arXiv preprint arXiv:1910.11956 (2019).

[7] Fang, Kuan, et al. "Dynamics learning with cascaded variational inference for multi-step manipulation." arXiv preprint arXiv:1910.13395 (2019).

[8] Merel, Josh, et al. "Neural probabilistic motor primitives for humanoid control." arXiv preprint arXiv:1811.11711 (2018).

[9] Nachum, Ofir, et al. "Data-efficient hierarchical reinforcement learning." Advances in neural information processing systems 31 (2018).

Thanks for the detailed feedback and insightful comments. For your concern on the novelty of our method, we would like to highlight that important, careful design decisions in our current method enable significant performance gains over baselines whereas the naive combination of prior works falls short. We additionally evaluated our method on four additional domains and our method exhibits similar performance gains. We also addressed your concern on the lack of discussion on HRL by adding a new paragraph in the related work to discuss prior works in HRL and provided justification of our choice of the skill formulation.

Novelty

We would like to emphasize several key design decisions in our current method that are different from prior methods, which contribute to performance gains over baselines.

First of all, all prior work on online learning with skills extracted from offline data simply discards the offline data when learning the high-level policy (e.g., Pertsch et al. (2021), Ajay et al. (2020)). In our experiments, the baseline “Online w/ Traj. Skill” does exactly this (learning trajectory skills from offline data, then learning a high level policy purely from online samples), and is consistently worse than our method that utilizes offline data during online learning (especially on more challenging tasks like the Large and Ultra AntMaze environments in Figure 11 and on all HumanoidMaze tasks in Figure 16).

In addition, our method is not a naive combination of SPiRL (Pertsch et al. (2021)) and ExPLORe (Li et al. (2024)). Pertsch et al. (2021) use a KL constraint between the high-level policy and a state-dependent prior obtained from offline pretraining. We found this design can actually hurt the online performance. We show that a simpler design without the KL constraint works much better. In Figure 7, we compare (Ours (KL)) with our method (Ours) and demonstrate that the final performance and the sample efficiency of the naive combination is much worse. As described in Appendix E, we borrow the policy parameterization from Haarnoja et al. (2018) and adopt a tanh policy parameterization with entropy regularization on the squashed space. Such design ensures that the online high-level policy is not explicitly constrained to a pre-trained prior, allowing the online policy to learn skill distributions that are more suitable for the task online. It also allows the addition of entropy regularization to the high level policy, which helps exploration.

These careful designs are what make our method extremely stable, sample efficient, and scalable to more complex tasks. We show additional results on four new domains with two locomotion domains and two manipulation domains in Figure 3. We selected these environments from OGBench [1], since they provided challenging, long-horizon tasks which also require exploration to solve, making them well-suited for testing the use of offline data to accelerate online exploration. In the challenging HumanoidMaze domain, our method is often the only method that achieves non-zero success rate on the four most difficult mazes. On manipulation tasks, our method consistently outperforms all prior methods on all domains with the only exception on Scene where one of the baselines (Offline w/ HILP) performs better. It is worth noting that Offline w/ HILP is a novel baseline that we introduced to also leverage offline data twice, both during offline and online learning with the only difference being that the unsupervised skill pre-training algorithm is HILP (instead of using trajectory VAE). This further demonstrates that the principle of leveraging offline data for both skill pre-training and online learning is effective. The effectiveness of our method across seven domains further highlights the importance of a careful combination of skill pre-training and effective online learning that utilizes the offline data.

[1] Park, Seohong, et al. "OGBench: Benchmarking Offline Goal-Conditioned RL." arXiv preprint arXiv:2410.20092 (2024).

Thanks for the quick reply. I appreciate the new experiments. Regarding the novelty, you mentioned the key insights are mainly i) using offline data twice and ii) removing the KL. For i) using the offline data twice, from the concept level, ExPLORe introduces using the offline data with optimistic rewards for better online learning. Built upon ExPLORe, the proposed method learns skills from offline data and shows better results, while the effectiveness of extracting skills is already demonstrated in many previous papers in both online and offline settings. For ii) removing the KL, it's more like an empirical observation (you included this in the practical implementation details section) rather than new insights. To make it a new insight for the community, you need to e.g., reveal the reason why removing KL is necessary and when (under which conditions) should we remove the KL, etc, which requires thorough analysis.

Thanks you again for your time to review our paper and read over our response. Could you be more specific about your concerns on the paper's novelty and how we did not sufficiently address your concerns in our response? We have showed with our experiments that our method outperforms prior approaches across seven domains and careful ablation to show that our key algorithm designs (e.g., using the offline data twice, removing the KL) are crucial in achieving the performance gain. We have also included a new sensitivity analysis experiments (Figure 5) that provides more insights to our algorithm. If there are any new experiments or specific questions that could help improve the paper, please let us know!