Unsupervised-to-Online Reinforcement Learning

Thank you for the detailed review. We appreciate the reviewer’s questions about the skill selection strategy as well as alternative ways to adapt unsupervised RL policies. Please find our response below. The revisions made are marked with “$\text{\color{red}red}$” in the revised paper.

Q1. More advanced skill identification processes can be developed for encapsulating long-horizon information

A1. Thanks for the question. While we found in our experiments that fixing a single $z$ was sufficient for the various tasks we used, we agree that U2O may benefit from a more sophisticated skill identification strategy for even longer-horizon and more complex tasks. We leave this extension for an interesting future research direction.

Q2. "Unsupervised pretraining can facilitate various types of fine-tuning, including offline RL fine-tuning, imitation learning, and others. Limiting the study to only online RL fine-tuning could be expanded for a more comprehensive evaluation."

A2. Thanks for the suggestion! In Q4 of Section 5, we tested various strategies to utilize a pre-trained unsupervised RL policy — i.e., (1) online RL fine-tuning, (2) zero-shot RL, (3) hierarchical RL, and (4) PEX, and found online RL fine-tuning to be the best among the four on our tasks. However, as the reviewer pointed out, there might be even more diverse ways and different problem settings (e.g., imitation learning) to adapt an unsupervised RL policy. We leave investigating such a possibility as future work.

We thank the reviewer again for providing valuable comments and feedback. We hope that our response has addressed the questions raised in the review, and please feel free to let us know if there are any additional concerns or questions.

Thanks for the detailed responses.

+ Although most reviewers believe this paper is a naive combination of existing methods, I think the contribution of proposing the U2O framework is significant, since it holds the potential to utilize a broader range of task-agnostic data to faciliate efficient task-specific training, which is especially crucial when deploying RL in many applications where task-specific data is limited.

- I'm curious about the impact on online RL fine-tuning if offline RL pretraining incorporates regularizations like DR3 to prevent feature collapse. It seems the key advantage of U2O over O2O doesn't stem from this factor.

- I agree with other reviewers that the paper may overstate some claims.

Considering the potential advantages of leveraging more task-agnositc data, I still maintain a positive score. But, due to the remaining unresolved concerns, I will maintain my score to a boardline accept.

Thank you for the detailed review. We especially appreciate the reviewer’s questions and feedback about some potential overclaims in the paper. Following the suggestion, we have conducted additional experiments with the $`kitchen-complete`$ dataset, and have revised the paper to avoid any overclaims. Please find our response below. The revisions made are marked with “$\text{\color{red}red}$” in the revised paper.

Q1. The claim made in the abstract "Do you agree that your method does not outperform O2O methods when using expert data sets?"

A1. Thanks for the feedback. While we have shown that U2O often matches or outperforms O2O across many different tasks, indeed it is not always the case, as pointed out by the reviewer and as mentioned in the conclusion section. We agree that the sentence in the abstract is not entirely accurate in its current form, and have toned it down to more accurately reflect our experimental results. Thanks again for pointing out this issue.

Regarding the question, yes, as we have previously mentioned in Section 7, we would agree that U2O would be most effective when the underlying dataset is diverse (so that the agent can learn diverse skills and rich representations), and it might not be as effective when the dataset is monolithically optimal.

Q2. "I want to know how U2O compares to O2O in Kitchen when using the "complete" data set." / "Why have you only included one experiment using exert data sets, put it in the appendix, and not explained the results in the main text?"

Thanks for the suggestion! While we have originally mentioned this point as a limitation in the main text (L531-L539), the experiments were limited to two tasks, $`antmaze-large`$ and $`antmaze-ultra`$. Hence, following the suggestion, we conducted additional experiments on $`kitchen-complete`$.

Normalize return in Kitchen-complete (4 seeds) \begin{array}{l|cccccc} \hline \text{Environment Steps} & 0.0 \times 10^5 & 1.0 \times 10^5 & 2.0 \times 10^5 & 3.0 \times 10^5 & 4.0 \times 10^5 & 5.0 \times 10^5 \newline \hline \text{U2O RL (Ours)} & 46.37 \pm 17.60 & 52.75 \pm 17.20 & 52.94 \pm 17.47 & 64.19 \pm 6.49 & 58.56 \pm 11.56 & 59.69 \pm 9.79 \newline \text{O2O RL} & 35.63 \pm 21.68 & 48.23 \pm 18.96 & 51.56 \pm 19.73 & 54.69 \pm 11.74 & 48.06 \pm 15.11 & 50.50 \pm 14.87 \newline \hline \end{array}

We found that, perhaps a bit unexpectedly, U2O slightly outperforms O2O on $`kitchen-complete`$. Honestly, we don't think this indicates that U2O can necessarily be better than O2O even on some monolithic datasets. Instead, we would interpret this result (and the results in Appendix A.3) as follows: even on monolithic datasets, U2O is not necessarily significantly worse than O2O (though U2O can be a bit slower at the beginning on expert-only datasets for some $`antmaze`$ tasks). We find this to be rather promising because, unlike traditional O2O, U2O does not use any task information during pre-training, and thus a single pre-trained model can be applied to many different reward functions. Hence, we believe U2O should generally be preferred to O2O if they achieve similar performances.

We thank the reviewer again for providing valuable comments and raising important points about U2O RL. We hope that our revisions have removed any potential overclaims in the revised version of the paper. We also believe the additional results with the $`kitchen-complete`$ dataset have significantly improved the paper. Please let us know if we have addressed your concerns and if so, we would be grateful if the reviewer could consider adjusting the score.

Thank you for the detailed review. We especially appreciate the reviewer’s question about the contribution of our work. Following the suggestion, we have conducted additional experiments with manipulation tasks. Please find our response below. The revisions made are marked with “$\text{\color{red}red}$” in the revised paper.

Q1. "While this paper proposes the U2O RL framework, it does not introduce any novel methods. Both the unsupervised offline RL pre-training and the online fine-tuning stages rely on existing algorithms."

A1. Thanks for raising the question about our contributions. As the reviewer pointed out, individual components of U2O RL are not necessarily completely novel. However, to our knowledge, we are the first to show that (skill-based) unsupervised offline RL can often be even better than supervised offline RL for online fine-tuning even for the same task (L087). We believe this observation is novel (at least it was quite surprising to us — how can not using task information during pre-training possibly be better than using it?). In the paper, we experimentally identified the potential cause behind this phenomenon (i.e., representation quality issue) via our analyses, and developed a practical method (U2O RL) to substantiate this insight. We believe our insight and practical recipe constitute a solid contribution to the community.

However, we agree with the reviewer that calling it an entirely novel framework might potentially be a bit misleading. We have thus revised the manuscript to call it a "recipe" instead of a "framework", and toned down several claims accordingly.

Q2. “The paper should conduct a broader set of experiments.”

A2. Thanks for the feedback. To further strengthen the results, we additionally conducted experiments with new manipulation environments ($`cube-single`$ and $`cube-double`$) from OGBench [Park et al., 2024], bringing the total to $\mathbf{11}$ environments. The goal of these cube tasks is to perform pick-and-place manipulation of multiple cubes from an unlabeled dataset that consists of diverse, random pick-and-place behaviors.

Success rate in OGBench-cube-single (4 seeds) \begin{array}{l|cccccc} \hline \text{Environment Steps} & 0.0 \times 10^5 & 2.0 \times 10^5 & 4.0 \times 10^5 & 6.0 \times 10^5 & 8.0 \times 10^5 & 10.0 \times 10^5 \newline \hline \text{U2O RL (Ours)} & \bf{72.50} \pm 12.79 & \bf{90.00} \pm 9.38 & \bf{97.00} \pm 1.00 & \bf{99.00} \pm{1.15} & \bf{100.00} \pm 0.00 & \bf{99.50} \pm 1.00 \newline \text{O2O RL} & 36.50 \pm 9.43 & 79.00 \pm 7.75 & 71.00 \pm 4.76 & 78.00 \pm 9.38 & 91.50 \pm 5.74 & 94.50 \pm 4.43 \newline \hline \end{array}

Success rate in OGBench-cube-double (4 seeds) \begin{array}{l|cccccc} \hline \text{Environment Steps} & 0.0 \times 10^5 & 2.0 \times 10^5 & 4.0 \times 10^5 & 6.0 \times 10^5 & 8.0 \times 10^5 & 10.0 \times 10^5 \newline \hline \text{U2O RL (Ours)} & \bf{2.50} \pm 5.00 & \bf{21.00} \pm 14.09 & \bf{19.00} \pm 13.61 & \bf{33.50} \pm 19.82 & \bf{38.00} \pm 21.97 & \bf{44.00} \pm 27.52 \newline \text{O2O RL} & 0.50 \pm 1.00 & 0.50 \pm 1.00 & 0.00 \pm 0.00 & 0.00 \pm 0.00 & 0.50 \pm 1.00 & 0.00 \pm 0.00 \newline \hline \end{array}

The tables above show that U2O outperforms O2O in these complex tasks as well. We have added these new results to the revised draft.

Q3. "the paper needs to provide more substantial evidence on why this U2O RL framework is more effective than the traditional O2O approach"

A3. Through our experiments, we showed that U2O often matches or outperforms O2O across a variety of domains and that it often learns representations of high quality. Unfortunately, U2O does not always outperform O2O; indeed, it sometimes just achieves a matching performance to O2O, as the reviewer mentioned. However, we would like to highlight one important advantage of U2O: it does not use any task information during pre-training, and thus a single pre-trained model can be applied to many different reward functions (unlike U2O), as we have demonstrated in Figure 8. Hence, we believe U2O should generally be preferred to O2O if they achieve similar performances.

Q4. "Why does the feature dot product in the O2O framework for Walker Run and Cheetah Run (Fig. 4) diverge, indicating much poorer representation learning compared to U2O, yet the performance of O2O in these environments is similar or even superior to U2O?"

A4. Thanks for this valid question! As briefly noted in the paper (L460) (and as shown in previous work [Fu et al., 2022]), we believe the quality of representations alone does not fully explain the performance, although there exists a correlation to some degree [Kumar et al., 2022]. In Figure 4, we mainly wanted to highlight that the representation quality of U2O is generally better than that of O2O, which we believe can partially explain the performance of U2O in some environments.

Thanks for the detailed responses.

Although I believe that research on pretraining-finetuning in RL is quite important, I think this paper does not adequately explain when the proposed U2O is more applicable, because clearly U2O does not show advantages in all environments. Additionally, the data available for pretraining might be limited to that environment or related environments, which is quite different from LLM pretraining where data from a wide range of unrelated tasks can be utilized. I believe that U2O still faces significant challenges in terms of broader applicability. Furthermore, I suggest that the authors provide some recommendations on the types of environments or tasks where U2O can perform better, and where it might not.

In summary, I regret that I cannot increase my score.

Thank you for the detailed review. We especially appreciate the reviewer’s questions and feedback about the significance of our empirical results and the potential overclaim of the scope. Following the suggestion, we have conducted additional experiments with the DR3 regularizer, and have revised the paper to avoid any potential overclaims. Please find our response below. The revisions made are marked with “$\text{\color{red}red}$” in the revised paper.

Q1. "Insufficient empirical evidence to claim U20>O2O: In figure 3, it seems the results are not significant in 10/14 environments. How can we claim the U2O is a better strategy?"

A1. Thanks for asking this question! Although U2O outperforms O2O in several tasks, as the reviewer pointed out, U2O indeed does not always outperform O2O. However, we would like to highlight one important advantage of U2O: it does not use any task information during pre-training, and thus a single pre-trained model can be applied to many different reward functions (unlike U2O), as we have demonstrated in Figure 8. Hence, we believe U2O should generally be preferred to O2O if they achieve similar performances.

Q2. Network sizes and discount factor for U2O and O2O

A2. Thanks for asking for the details about hyperparameters. We would first like to note that, in Figure 3, we ensured a fair comparison between U2O and O2O with the same amount of hyperparameter tuning (e.g., search over {256, 512} for the hidden dimension and {2, 3} for the number of layers, for both methods). In Table 1, we directly took the results from the previous work (Cal-QL), and thus it does not necessarily make a complete apples-to-apples comparison (due to the potential difference in network sizes; though we used the same discount factor of 0.99 for all methods on D4RL tasks). That said, we believe this table still helps contextualize our performance in previously reported numbers. Moreover, for the strongest baseline in Table 1 (Cal-QL), we additionally tuned and ran it ourselves to obtain the results in the most complex environment ($`antmaze-ultra`$), and found that U2O outperforms Cal-QL by a significant margin.

Q3. Can DR3 be used to prevent feature collapse in O2O?

A3. Thanks for the question! Following the reviewer's suggestion, we conducted additional experiments to compare U2O and O2O+DR3 on AntMaze-ultra-diverse.

Normalize return in AntMaze-ultra-diverse (4 seeds) \begin{array}{l|cccccc} \hline \text{Environment Steps} & 0.0 \times 10^5 & 2.0 \times 10^5 & 4.0 \times 10^5 & 6.0 \times 10^5 & 8.0 \times 10^5 & 10.0 \times 10^5 \newline \hline \text{U2O RL (Ours)} & \bf{22.00} \pm 9.97 & \bf{36.25} \pm 13.24 & \bf{43.00} \pm 10.09 & \bf{47.00} \pm 24.26 & \bf{46.50} \pm 12.91 & \bf{54.00} \pm 14.24 \newline \text{O2O RL + DR3} & 4.67 \pm 5.03 & 18.67 \pm 6.11 & 29.33 \pm 12.22 & 27.33 \pm 10.07 & 30.00 \pm 5.29 & 32.00 \pm 12.49 \newline \text{O2O RL} & 13.25 \pm 6.67 & 9.25 \pm 6.67 & 14.25 \pm 11.73 & 23.00 \pm 9.44 & 29.75 \pm 20.27 & 29.00 \pm 13.18 \newline \hline \end{array}

The result above shows that simply adding the DR3 regularizer is not as effective as U2O RL, and we believe this is likely because the full unsupervised RL procedure can lead to much richer representations than simply adding a regularizer. We have added this result to Appendix A.7 of the revised draft.

Q4. "To be calling this a framework might be an overstatement as they consider very related unsupervised approaches based on a single bucket of successor features."

A4. Thanks for the feedback. As the reviewer pointed out, and as we have mentioned throughout the paper, U2O RL assumes a skill-based offline unsupervised RL method (e.g., successor feature learning, offline goal-conditioned RL, etc.), and is not directly compatible with other types of unsupervised RL methods in the current form. Following the suggestion, and to avoid any overclaiming, we have revised the paper to replace "framework" with "recipe", and have clarified that we assume a skill-based unsupervised offline RL algorithm throughout the paper.