Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization

Thanks Reviewer jjSi for the valuable review of our paper. We appreciate the questions you raised and are committed to delivering a comprehensive response to address the issues.

Q1.1: What do these variants (Conservatism, Constraint, Off-policy) mean in (a)?

A1.1: Here we use offline -> online method to specify the offline-to-online mehtod. In the previous version of Fig. 1(a), the conservatism, constraint, and off-policy represent the offline-to-online methods CQL->CQL, IQL->IQL, and CQL->SAC. We have revised the legend to make it more clear in the updated manuscript.

Q1.2: How does the Q value compare with the V value in (b)? Moreover, from (b), it seems that CQL->SAC shows faster improvement than On-policy (V). How does this conclude that Q values of SAC exhibit slow improvement?

A1.2: In the motivating example, we aim to compare the naïve off-policy (SAC), conservative method (CQL), and naïve on-policy (ours). During the offline-to-online transition, naïve off-policy methods often encounter distribution shifts when the policy explores out-of-distribution regions. This can result in evaluation errors and a subsequent drop in performance. A simple method to address this challenge is to inherit the conservatism employed during the offline phase. However, relying solely on this conservative strategy can introduce suboptimality during the online phase, resulting in data inefficiency. Therefore, we aim to investigate whether a naïve on-policy RL algorithm can effectively tackle this challenge. Uni-O4 initialized the policy and $V$-function from the offline phase for a standard online PPO.

Both the $V$ and $Q$ functions can evaluate the performance of the policy. Therefore, we consider it acceptable to compare the $V$-value and $Q$-value across these methods to investigate how the value function guides performance improvement. In the case of the naïve off-policy method, the $Q$-value exhibits a significant drop and unstable improvement during online training, leading to a decline in policy performance. The conservative method, on the other hand, shows slow improvement in the $Q$ value, resulting in suboptimality. In contrast, the on-policy method demonstrates stable improvement in the $V$ value, leading to consistent and efficient fine-tuning.

Q1.3: How does this conclude that Q values of SAC exhibit slow improvement?

A1.3: The $Q$-value of the off-policy method improves from 27 to 41, while the $V$-value of the on-policy method improves from approximately 36.5 to 47. It is true that the improvement in the $Q$-value is slightly faster than its counterpart. However, two points should be considered: 1) The on-policy method (ours) has a significantly higher initial performance from the offline phase compared to the off-policy method, resulting in higher initial scores for the value function. Therefore, a slightly slower convergence speed is reasonable. 2) The off-policy method experiences a noticeable drop and unstable improvement in the value, whereas this phenomenon is absent in the on-policy method.

Q1.4: Furthermore, CQL->CQL and CQL->SAC are naïve solutions for offline-to-online RL. What about advanced offline-to-online RL algorithms, such as off2on?

A1.4: Thank you for your insightful suggestion. It is interesting to investigate how the ensemble value function guides policy improvement. Thus, we have added the off2on as a comparison. Our findings reveal that the $Q$ value of the off2on method demonstrates faster improvement and can eventually converge to the same level as the $V$ value of Uni-O4. However, it is worth noting that the off2on approach experiences performance drops and unstable training during the initial stages, primarily due to the unstable improvement of the $Q$ value.

Thanks Reviewer bAiM for the valuable review of our paper. We appreciate the questions you raised and are committed to delivering a comprehensive response to address the issues.

Q1: The behavior cloning method proposed in section 3.1 requires training multiple policy networks, which incurs significant computational overhead.

A1: Thank you for your comment. We have thoroughly examined the computational overhead and performance implications associated with different ensemble sizes of policies in Section 5.3 and Appendix A.11. Our findings indicate that an ensemble size of 4 strikes a balance between performance and computational overhead. Specifically, the total training time for the offline phase is approximately 265 minutes (with an ensemble size of 4), compared to 200 minutes (with a single policy) using our PyTorch implementation. Despite the slight increase in training time, this trade-off is acceptable considering the significant performance improvement achieved through the ensemble approach.

Q2: Definition error, the definition of f used in formulas 6 and 7 is incorrect. Taking the maximum value of multiple distributions cannot guarantee a single distribution (the sum cannot be guaranteed to be 1), and analysis based on this definition is also meaningless. If the code is truly implemented based on this definition, I am skeptical about the final performance of the algorithm.

A2: Thank you for pointing this out. We have taken measures to address this concern. Firstly, we have normalized the distribution of the combined policies, as outlined in Proposition 1. Additionally, we have derived a lower bound over the defined distance, as described in Theorem 1. By optimizing this lower bound, we are able to enhance the diversity among behavior policies. Notably, the derived lower bound corresponds to the penalty term mentioned in Equation 7 of the previous version. Therefore, the implementation can be guaranteed based on these derived results.

Dear reviewer bAiM:

We appreciate again for your constructive comments and helpful suggestions. Since the Reviewer-Author discussion phase is coming to an end, we would like to post a follow-up discussion.

In our previous response, we clarified the raised questions and made corresponding improvements in the updated manuscript. We hope to further discuss with you whether your concerns have been addressed. We are always looking forward to your further comments or suggestions.

Thank you, Reviewer rY3g, for your valuable review of our paper. We sincerely appreciate the questions you have raised and we are fully dedicated to providing a comprehensive response to address all of the concerns.

Q1: The complexity of the method, especially regarding ensemble behavior cloning and disagreement-based regularization, may present a steep learning curve for practitioners.

A1: We have added a more intuitive explanation and algorithm description, aiming to provide a clearer understanding for practitioners. Uni-O4 adopts a straightforward approach by combining online and offline RL learning without introducing additional regularization during their transfer.

In detail, the online RL algorithm employed is a standard PPO. While the offline stage involves multiple components such as value functions, transition models, and ensemble BC, it is worth noting that these components are designed to be implementation-friendly, straightforward, and efficient. For the ensemble BC, we train each policy using a standard loss, incorporating an approximate KL-divergence penalty with a combined policy. This penalty term is implemented by sampling data from the offline dataset. These components remain fixed during the multi-step policy optimization stage. During this stage, AM-Q consists of the well-trained transition model, and the $Q$-function is used to evaluate the performance of the target policy when replacing the behavior policy. By permitting the replacement of the current behavior policy with the target policy, it achieves multi-step optimization. It differs from iteratively updating the behavior policy which leads to accumulated errors that disrupt the desired monotonicity. In contrast, the multi-step method can guarantee monotonicity due to the AM-Q error is bounded.

We'll also fully open-source our code with detailed documentation to help implement our method, including in a new real-world robot scenario.

Q2: What are the computational overheads associated with the ensemble policies, and how do they impact the method's scalability?

A2: Thank you for this suggestion. We've conducted the computational overheads and the performance associated with the ensemble size of policies in Section 5.3 and Appendix A.11. We found that the ensemble size $4$ is a trade-off between performance and the computational overheads. Specifically, the whole training time of the offline phase is around 265 minutes (ensemble size $4$) vs. 200 minutes (single policy) based on Pytorch implementation. It's acceptable due to the significant performance improvement. For offline training, the primary runtime of the offline phase is attributed to the supervised learning stage. To evaluate the runtime, we conducted 2 million steps for $Q, V$ training, 0.5 million steps for ensemble policy training, and 1 million steps for transition model training. However, in practical scenarios, these training steps can be halved, resulting in reduced runtime.

Q3: Why don't use the ensemble approach to mitigate mismatches instead of other methods for handling the diverse behaviors in the datasets? For example, Diffusion-QL [1] demonstrates that the Diffusion model can be used to learn multimodal policy.

A3: Thank you for the insightful comment. Diffusion-QL [1] is a representative work that uses diffusion policies to handle the multi-modal dataset. It uses a diffusion model as the policy to predict actions and trains the policy with a behavior cloning loss and an extra state-action value term. It is a value-based training paradigm. Indeed, the diffusion policy has the capability to replace ensemble BC during the supervised stage. However, it cannot directly be optimized by the PPO objective during the multi-step policy optimization stage. This kind of policy-based algorithm needs to model the policy's distribution explicitly and rely on differentiating through the policy distribution to compute gradients and perform updates. It is not suitable for optimizing this kind of generative model directly.

Additionally, we have added diffusion-QL as a baseline for comparison. The performance of Uni-O4 significantly outperforms the Diffusion-QL over several domain tasks. Moreover, Uni-O4 is more computationally efficient than Diffusion-QL, consuming 12.6 hours (Diffusion-QL) vs. 4.4 hours (Uni-O4).

[1] Wang Z, Hunt J J, Zhou M. Diffusion policies as an expressive policy class for offline reinforcement learning[J]. arXiv preprint arXiv:2208.06193, 2022.

Dear reviewer rY3g:

We would like to thank you again for your constructive comments and helpful suggestions. Since we are nearly at the end of the discussion phase, we'd like to know whether our response has addressed your concerns and we are always looking forward to your further comments or suggestions.

Q3: In Sec 3.1: "BPPO leads to a mismatch ... due to the presence of diverse behavior policies in the dataset D", could authors explain further why the diversity is blamed for the mismatch?

A3: One example that can illustrate this motivation is the presence of multi-modality within a diverse dataset. In other words, the dataset exhibits multiple modes, with the main mode primarily composed of low-return actions. Conversely, the subdominant mode consists of low-density but high-return actions. In such a scenario, standard behavior cloning (BC) is susceptible to imitating the high-density but low-return actions, resulting in a bias towards fitting the main mode. However, during the offline multi-step optimization stage, the policy optimization is constrained by the clip function, making it difficult for the policy to escape this mode. Consequently, this can lead to a sub-optimal policy as it becomes unable to explore the high-return action region.

In contrast, our ensemble BC approach learns diverse behavior policies that are more likely to cover all modes present in the dataset. This facilitates exploration of the high-return region, enabling the discovery of the optimal policy. To support this claim, we have included a motivating example and conducted additional ablation studies in Appendix A.5. These findings demonstrate that the utilization of multiple diverse behavior policies allows for effective exploration of the high-return action region, particularly in the case of the "medium-replay" dataset, which contains more sub-optimal data than others.

Q4: Lack of theoretical analysis (to support the motivation of technique details), but it has sufficient experiments thus this point is acceptable I think.

A4: Thank you for this suggestion. We have reevaluated the approximation employed throughout the entire pipeline and included a thorough analysis to address this concern. During the multi-step policy optimization stage, the decision for behavior policy replacement is based on the proposed AM-Q metric, which incorporates the true transition model $T$ and optimal value function $Q^*$. In the offline setting, however, the agent does not have access to the true model and $Q^*$. But $Q_{\tau}$ approaches $Q^*$ based on the dataset support constraints. We fit $\hat{T}$ and $\widehat{Q_{\tau}}$ by gradient-based optimization. In this way, the practical AM-Q can be expressed as $\widehat{J_{\tau}}(\pi)$. Thus, the OPE bias is mainly coming from the transition model approximation. We have analyzed and derived the bound of the offline policy evaluation, described by Theorem 2 in Section 3.2, please refer to check out the detail.

BPPO [2] have derived the offline monotonical improvement bound (their Theorem 3) between the target policy and behavior policy, i.e., $\pi_{k+1}$ and $\pi_k$. However, since the behavior policy is updated iteratively, this bound can result in cumulative errors that disrupt the desired monotonicity. Consequently, this bound cannot be directly utilized in the offline setting. This limitation is the reason why BPPO requires online evaluation to ensure performance improvement when replacing the behavior policy with the target policy. Given the OPE bound, we can replace the online evaluation with AM-Q to guarantee monotonicity.

Additionally, we compute the advantage function by $(\widehat{Q_{\tau}}-\widehat{V_{\tau}})$, which provides an approximation of the optimal $Q^*$ and $V^*$ based on the dataset constraint assumption. In Appendix A.8, we conduct experiments to demonstrate this replacement is a superior choice compared to iteratively fitting the value function of the target policy in the offline setting. The results from these experiments prove that iteratively updating the value function will lead to overestimation which causes unstable training performance and potential crashes.

Q5: Suggest to add legends for Fig. 3 or bringing the legend in Fig. 4 forward.

A5: Thank you for pointing this out. We have addressed this issue in the updated manuscript.

[1] Kostrikov, I., Nair, A., & Levine, S. (2021). Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06169.

[2] Zhuang Z, Lei K, Liu J, et al. Behavior proximal policy optimization[J]. arXiv preprint arXiv:2302.11312, 2023.

Thanks Reviewer CZhd for the valuable review of our paper. We appreciate the questions you raised and are committed to delivering a comprehensive response to address the issues.

Q1: Notions are confusing in this paper, especially after the overloading in Equ. (8).

A1: Thank you for pointing this out. We have rewritten Section 3.2 "Multi-step policy ensemble optimization" to enhance clarity in our presentation. In Equ. 8 (7 in the new version), specifically, we overload the notion of behavior policy for introducing the iteration number. It is necessary for showcasing how Uni-O4 performs multi-step optimization by adding the iteration number as subscript $k$ in behavior policies $\pi^i_k$ where $i$ specifies the policy index in the ensemble. Furthermore, we have revised the notion of the $Q$ and $V$ -function to be consistent with IQL as $\widehat{Q_{\tau}}$ and $\widehat{V_{\tau}}$ from $Q_{\pi_{\beta}}$ and $V_{\pi_{\beta}}$. Because we found that the previous version ($Q_{\pi_{\beta}}$ and $V_{\pi_{\beta}}$) will cause ambiguity. On the other hand, IQL has the capability to reconstruct the optimal value function, i.e., $\lim_{\tau \rightarrow 1} Q_{\tau} (s, a) = Q^{\ast} (s, a)$ based on dataset support constraints [1]. In this work, we exploit the desirable property to facilitate multi-step policy optimization and recover the optimal policy. $Q_{\tau}$ and $V_{\tau}$ are the optimal solution. We use $\widehat{Q_{\tau}}$ and $\widehat{V_{\tau}}$ to denote the value functions obtained through gradient-based optimization.

Q2: In Fig.2, It is hard to capture the Offline Multi-Step Optimization process, i.e. the sequence relationship of each step.

A2: Based on this comment, we have updated Fig. 2 by incorporating sequence numbers and a detailed description for each step. During the offline multi-step optimization stage, the policy optimization begins with the learned behavior policies. Specifically, each policy undergoes optimization using the PPO loss for a designated number of gradient steps outlined in step 1 of Fig. 2. Subsequently, both the target policy and behavior policy are assessed using AM-Q to determine whether the behavior policy should be replaced by the target policy, as specified in step 2. If the evaluation results meet the OPE conditions, the behavior policy is substituted with its target policy, denoted as step 3. These three steps are iterated until the predetermined number of gradient steps is reached. Also, we sincerely invite the reviewer to check the GIF of Fig. 2 on our website for a clearer presentation.