A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning

Many thanks to Reviewer MwS2 for providing the detailed and valuable review, and we respond to the weaknesses and questions as below.

Q1: Definition of uncertainty.

In this work, we define the uncertainty as the approximate negative log likelihood of the state-action density as shown in Page 4: $\mathcal{U}(s,a)\overset{\rm{def}}{=}-\log p(s,a)\approx\mathcal{L}_{\mathrm{ELBO}}(s,a;\psi,\varphi).$

Q2: Advantage of VAE over ensemble-based technique for uncertainty quantification.

We detail the reasons at the beginning of Section 4.1 as below.

Many prior works typically utilize Q ensembles to qualify uncertainty. However, the ensemble technique may significantly increase computational costs. Thus, in this work, we utilize a simple yet effective approach by adopting VAE as a state-action visitation density estimator for uncertainty quantification.

Furthermore, we also derive some initial empirical results on the ensemble setting in Table 9 in the appendix. From Table 9, we can observe that uncertainty quantification with VAE can attain comparable performance to using the variance of 10 Q functions, where the latter would significantly increase computational costs.

Q3: Enhance data support of the offline dataset through online interaction.

Yes, the collected online data can enhanced the data support of the offline dataset. However, we aim at 100k environment step setting in this work, which requires extremely high sample efficiency. Therefore, although the policy can automatically recover the extrapolation error brought by OOD state-actions via trial-and-error, it leads to inferior sample efficiency. Thus, we introduce the adaptive exploitation method with OOD sample identification to overcome this issue.

Q4: Explanations on exploration-exploitation issue in offline-to-online RL.

We observe that aggressive exploration may exacerbate the distribution shift, while conservative exploitation can hinder agents from efficient online finetuning. Specifically, as stated at the end of Section 4.2, to solve the constrained exploratory behavior issue, we introduce optimistic exploration strategy. However, this may also bring negative effects by further increasing state-action distribution shift, as a result of the principle of optimism in the face of uncertainty. Thus, simultaneously and efficiently addressing both challenges brings additional improvement.

Q5: "Forgetting" issue.

We want to clarify that the forgetting issue is one outcome of failing to handle state-action distribution shift. As stated in Introduction,

Accordingly, the state-action distribution shift between offline and online data occurs, leading to the well-known extrapolation error during exploitation, which may wipe out the good initialization obtained from the pretraining stage.

Thus, we solve this issue by proposing the adaptive exploitation method with OOD sample identification.

Q6: Evaluation of "generic" solution for offline-to-online RL.

We want to clarify that the genericness of SUNG means that SUNG can be seamlessly combined with multiple existing offline RL methods. We verify this by combing SUNG with four backbone offline RL methods, including TD3+BC, CQL, CQL-10, and SPOT.

Q7: Hyper-parameter Tuning.

We want to clarify that we only tune these two hyper-parameters of SUNG in this paper, i.e., finalist action set size k and OOD sample percent p. Actually, SUNG does not require extensive hyper-parameter tuning; instead, it only demands minimal effort in that regard. As demonstrated in Figure 3, SUNG consistently outperforms the best baseline for any choice of k. Similarly, as illustrated in Figure 4, SUNG consistently outperforms the best baseline when p is appropriately set within the range of [5, 10]. Thus, SUNG achieves robust performance without the need for intensive hyper-parameter fine-tuning, which aligns with our initial motivation to develop a simple-yet-effective offline-to-online RL algorithm.

In terms of other hyper-parameters, we set them with an appropriate value and never tune them out of simplicity. For example, we set N=100 for TD3+BC and CQL, and set them to 20 for CQL-10 to decrease the computational costs. We set alpha=1.0, which is the exactly the default setting without temperature. We set lambda=1.0, which is exactly the default multi-task loss setting. Thus, we also highlight that carefully tuning these hyper-parameters may bring further improvement of SUNG.

Q8: Relationship between finetuning method and pretraining algorithm.

Yes, you are right. In this work, we utilize the corresponding online RL objectives for online finetuning to keep consistent.

We hope that our response could address your concerns and that you would consider increasing your score.

Dear reviewer MwS2, as the deadline for author-reviewer discussions is approaching, could you kindly confirm if your concerns have been adequately addressed? We are more than willing to continue the discussion if there are any lingering issues. Thank you sincerely for your efforts!

(Part 1/2)

We extend our sincere gratitude to Reviewer YGBs for valuable and constructive feedback. In response to the identified weaknesses and questions, we provide the following clarifications and improvements

Q1: Novelty Concern.

We want to highlight our novelty from the following aspects.

Firstly, we mark that we identify the key question in offline-to-online RL as simultaneously addressing both challenges of constrained exploratory behavior and state-action distribution shift while eliminating the inherent conflict between exploration and exploitation. While prior research has attempted to address each challenge independently, tackling both challenges simultaneously is not a trivial task. For instance, some previous works (e.g., O3F) proposes efficient exploration strategy to mitigate the constrained exploratory behavior. However, we observe their divergence and sub-optimal performance when finetuning CQL agents, precisely due to this inherent conflict. Please refer to Appendix B for detailed analyses. In contrast, SUNG naturally unifies the solutions to both challenges with the tool of uncertainty, achieving the desired trade-off between exploration and exploitation. As a result, when combined with either TD3+BC or CQL, SUNG significantly surpasses previous SOTAs.

Secondly, we point out that we develop novel techniques, such as bi-level action selection mechanism and adaptive exploitation with OOD sample identification, enabling SUNG to present a simple-yet-effective manner. Different from previous optimistic exploration techniques, we propose a bi-level action selection mechanism to mitigate the challenges of computation complexity and intensive task-specific hyper-parameter tuning, while achieving efficient exploration. In contrast to previous adaptive exploitation methods, SUNG builds on the insight that offline RL objectives can be typically interpreted as an online RL objective and a regularization term. As such, we develop a novel adaptive exploitation method with OOD sample identification to achieve the desired trade-off between performance and stability. Significantly, SUNG demonstrates robust superiority over previous SOTAs across a diverse set of hyper-parameters, as shown in Figure 3 and 4, which further enhances its practical applicability.

Thirdly, we emphasize that we perform systematic empirical evaluations for SUNG. We combine SUNG with 4 backbone offline RL algorithms across 15 settings spanning MuJoCo and AntMaze domains, and we utilize 13 baselines for comparison. Experimental results demonstrate that SUNG significantly outperforms previous SOTAs. Furthermore, we conduct extensive ablation studies on each component to thoroughly verify the effectiveness of SUNG. Additionally, we also showcase SUNG's seamless integration with other techniques, such as ensemble Q learning and high update-to-data ratios, leading to superior finetuning performance.

Finally, we also contribute a series of fresh insights to the offline-to-online setting, e.g., (1) Exploration with high Q-value actions leads to divergence when finetuning CQL, as elaborated in Appendix B. SUNG can stably finetune CQL agents by balancing the trade-off between uncertainty and Q value. (2) Fully online RL objectives fail in challenging sparse-reward tasks, as detailed in Appendix D.1. SUNG can achieve flexible trade-off between offline and online RL objectives, which enables stiching sub-optimal sub-trajectories and overcoming the sparse reward issue.

Overall, in this paper, we aim to develop a simple, generic, sample-efficient offline-to-online RL framework as a good starting point for deriving strong empirical performance.

(Part 2/2)

Q2: Concern on theoretical analysis.

We would like to clarify that the scope of this paper is to develop a generic offline-to-online RL framework that can be applied to different kinds of offline RL algorithms. Despite the practical theoretical results achieved by the offline RL community, to the best of my knowledge, there still exists a gap in establishing an initial theoretical foundation for offline-to-online RL. Moreover, given the diversity of theoretical foundations of offline RL algorithms, proposing a single framework considering different theoretical guarantees presents challenges. Therefore, we follow previous works [1-5] to focus on developing empirically effective and robust offline-to-online RL algorithms. From our systematic empirical evaluation on MuJoCo and AntMaze domains, SUNG significantly outperforms the previous SOTAs and does not require intensive hyper-parameter tuning. Moreover, extensive ablation studies on each component also demonstrate the effectiveness of SUNG.

[1] Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble, CoRL'22.

[2] Online Decision Transformer, ICML'22.

[3] Policy Expansion for Bridging Offline-to-Online Reinforcement Learning, ICLR'23.

[4] Adaptive Policy Learning for Offline-to-Online Reinforcement Learning, AAAI'23.

[5] Actor-Critic Alignment for Offline-to-Online Reinforcement Learning, ICML'23.

We hope that our response could address your concerns and that you would consider increasing your score.

Dear reviewer YGBs, as the deadline for author-reviewer discussions is approaching, could you kindly confirm if your concerns have been adequately addressed? We are more than willing to continue the discussion if there are any lingering issues. Thank you sincerely for your efforts!

We would like to sincerely thank Reviewer K3Su for providing the constructive review, and we respond to the weaknesses and questions as below.

Q1: Contradiction between high-uncertainty and near-on-policy actions.

As shown in Eq. (7), SUNG involves selecting actions with both high Q values and high uncertainty to guide exploration. Specifically, we restrict our exploration to near-on-policy actions to prevent significant deviations. This restriction offers two key advantages:

(1) Ensuring finetuning stability: By exploring near-on-policy actions during finetuning, we maintain stability in the learning process. The policy improvement relies on experiences collected during exploration. Deviating too far from the current policy could lead to abrupt and undesirable changes, rendering the learning process unstable and less sample-efficient.

(2) Avoiding inaccurate Q value and uncertainty estimation: Sampling actions far from the on-policy actions may result in less reliable estimates of Q values and uncertainty. This unreliability has the potential to throw the exploration direction out of control, hindering the overall effectiveness and efficiency of exploration.

Q2: The absolute uncertainty value at which a sample could be labeled as OOD can vary.

We fully agree with your perspectives. SUNG detects OOD samples via percentage-based filtering strategy out of simplicity. This approach is intentionally designed to minimize the complexity associated with hyper-parameter tuning. However, we want to highlight that empirical results demonstrate the effectiveness of SUNG, as evidenced by the SOTA finetuning performance achieved by SUNG through the proposed adaptive exploitation. We acknowledge that there is room for future exploration in refining adaptive and precise identification of OOD samples.

Q3: Why introducing additional OOD detection in the adaptive exploitation.

Although many offline RL methods integrate OOD detection in the adaptive exploitation, they still necessitate heavy conservatism in the objective to avoid the extrapolation error. In this paper, SUNG introduces OOD sample identification to adaptively relax such conservatism for higher sample efficiency during online finetuning. It is noteworthy that certain offline RL algorithms, including TD3+BC, CQL, and SPOT, lack explicit OOD detection. Hence, the proposed adaptive exploitation holds significant importance for these methods to attain desirable fine-tuning performance.

Q4: Explanation on trade-off between value and uncertainty.

In the suggested optimistic exploration strategy, choosing the ranking criteria as "value first" implies that a low value of k indicates a preference for high value, whereas a high value of k indicates a preference for high uncertainty. Conversely, if the ranking criteria are set as "uncertainty first", the scenario is precisely the opposite. As such, by altering ranking criteria and k, the algorithm allows for the customization of preferences between uncertainty and Q value to varying extents. This flexibility empowers users to finely adjust the trade-off according to the specific requirements of their realistic scenarios.

Q5: How to select the ranking criteria?

We detail this problem in Appendix A.2 and A.3. Specifically, we choose uncertainty first for value regularization based offline RL methods, whereas Q first for other offline RL methods. This choice is underpinned by the following considerations. Firstly, we want to highlight the empirical finding that value regularization based offline RL methods (e.g., CQL) are only compatible with preference for high uncertainty, whereas other offline RL methods are compatible with preference for both high uncertainty and high value. We give a detailed analysis for the reason of this finding in Appendix B. As articulated in Q4, irrespective of the chosen ranking criteria, we possess the flexibility to adeptly adjust k to attain the desired preference. Consequently, we select the aforementioned criteria to fix the tuning range within a small value of k, only aiming for consistency.

Q6: Other evaluation metrics.

Since this paper focuses on the improving the sample efficiency of offline-to-online RL, we choose the final average performance of 100k environment steps as the evaluation metric in the main body of this paper for clear comparison. Note that we also provide the full training curve for 30 settings in Appendix D.4 to further demonstrate the effectiveness of SUNG.

Q7: Some related works.

Thanks for providing these related works! They are all included for discussion in the revised version.

Q8: Label of "Optimistic Exploration without Uncertainty"

Here, the optimism refers to exploration with optimism in the face of Q value. We give detailed experiment settings of the ablation study in Appendix D.5.

We hope that our response could address your concerns and that you would consider increasing your score.

Dear reviewer K3Su, as the deadline for author-reviewer discussions is approaching, could you kindly confirm if your concerns have been adequately addressed? We are more than willing to continue the discussion if there are any lingering issues. Thank you sincerely for your efforts!

(Part 2/2)

Q3: Claim of ranking criteria.

The choice is underpinned by the following considerations:

Firstly, we want to highlight the empirical finding that value regularization based offline RL methods (e.g., CQL) are only compatible with preference for high uncertainty, whereas other offline RL methods are compatible with preference for both high uncertainty and high value. We give a detailed analysis for the reason of this finding in Appendix B.

Secondly, choosing the ranking criteria as "value first" implies that a low value of k indicates a preference for high value, whereas a high value of k indicates a preference for high uncertainty. Conversely, if the ranking criteria are set as "uncertainty first", the scenario is precisely the opposite. As such, by altering ranking criteria and k, the algorithm allows for the customization of preferences between uncertainty and Q value to varying extents.

Thus, irrespective of the chosen ranking criteria, we possess the flexibility to adeptly adjust k to attain the desired preference. Consequently, we select the aforementioned criteria to fix the tuning range within a small value of k, only aiming for consistency.

Q4: Concern on hyper-parameter tuning.

We want to clarify that SUNG does not require extensive hyper-parameter tuning; instead, it only demands minimal effort in that regard. As demonstrated in Figure 3, SUNG consistently outperforms the best baseline for any choice of k, with the optimal D4RL score achieved at k=10/20. Similarly, as illustrated in Figure 4, SUNG consistently outperforms the best baseline when p is appropriately set within the range of [5, 10], with the highest D4RL score obtained at p=5. Thus, SUNG achieves robust performance without the need for intensive hyper-parameter fine-tuning, which aligns with our initial motivation to develop a simple-yet-effective offline-to-online RL algorithm. We only tune these two hyper-parameters of SUNG in this paper, i.e., finalist action set size k and OOD sample percent p.

Q5: Claim for state distribution shift.

Thanks for pointing out this ambiguity. You are right. Here, offline RL methods refer to model-free ones. We make this statement clearer in the revised version.

We hope that our response could address your concerns and that you would consider increasing your score.

(Part 1/2)

We express our heartfelt appreciation to Reviewer WjUY for their detailed and valuable review. In response to the weaknesses and questions, we provide the following clarifications and improvements.

Q1: Incremental contribution.

We want to highlight our contributions from the following aspects.

Firstly, we mark that we identify the key question in offline-to-online RL as simultaneously addressing both challenges of constrained exploratory behavior and state-action distribution shift while eliminating the inherent conflict between exploration and exploitation. While prior research has attempted to address each challenge independently, tackling both challenges simultaneously is not a trivial task. For instance, some previous works (e.g., O3F) proposes efficient exploration strategy to mitigate the constrained exploratory behavior. However, we observe their divergence and sub-optimal performance when finetuning CQL agents, precisely due to this inherent conflict. Please refer to Appendix B for detailed analyses. In contrast, SUNG naturally unifies the solutions to both challenges with the tool of uncertainty, achieving the desired trade-off between exploration and exploitation. As a result, when combined with either TD3+BC or CQL, SUNG significantly surpasses previous SOTAs.

Secondly, we point out that we develop novel techniques, such as bi-level action selection mechanism and adaptive exploitation with OOD sample identification, enabling SUNG to present a simple-yet-effective manner. Different from previous optimistic exploration techniques, we propose a bi-level action selection mechanism to mitigate the challenges of computation complexity and intensive task-specific hyper-parameter tuning, while achieving efficient exploration. In contrast to previous adaptive exploitation methods, SUNG builds on the insight that offline RL objectives can be typically interpreted as an online RL objective and a regularization term. As such, we develop a novel adaptive exploitation method with OOD sample identification to achieve the desired trade-off between performance and stability. Significantly, SUNG demonstrates robust superiority over previous SOTAs across a diverse set of hyper-parameters, as shown in Figure 3 and 4, which further enhances its practical applicability.

Thirdly, we emphasize that we perform systematic empirical evaluations for SUNG. We combine SUNG with 4 backbone offline RL algorithms across 15 settings spanning MuJoCo and AntMaze domains, and we utilize 13 baselines for comparison. Experimental results demonstrate that SUNG significantly outperforms previous SOTAs. Furthermore, we conduct extensive ablation studies on each component to thoroughly verify the effectiveness of SUNG. Additionally, we also showcase SUNG's seamless integration with other techniques, such as ensemble Q learning and high update-to-data ratios, leading to superior finetuning performance.

Finally, we also contribute a series of fresh insights to the offline-to-online setting, e.g., (1) Exploration with high Q-value actions leads to divergence when finetuning CQL, as elaborated in Appendix B. SUNG can stably finetune CQL agents by balancing the trade-off between uncertainty and Q value. (2) Fully online RL objectives fail in challenging sparse-reward tasks, as detailed in Appendix D.1. SUNG can achieve flexible trade-off between offline and online RL objectives, which enables stiching sub-optimal sub-trajectories and overcoming the sparse reward issue.

Overall, in this paper, we aim to develop a simple, generic, sample-efficient offline-to-online RL framework as a good starting point for deriving strong empirical performance.

Q2: Theoretical analysis/insight on bi-level action selection.

We want to highlight that the bi-level action selection mechanism is a practical solution for Eq. (7). It overcomes two challenges of solving Eq.(7): the difficulty in finding an optimal solution and the necessity for task-specific hyper-parameter tuning. Although we do not include theoretical insights of it, we empirically demonstrate the effectiveness of the proposed optimistic exploration strategy.

Dear reviewer WjUY, as the deadline for author-reviewer discussions is approaching, could you kindly confirm if your concerns have been adequately addressed? We are more than willing to continue the discussion if there are any lingering issues. Thank you sincerely for your efforts!