Decoupled Offline to Online finetuning via Dynamics Model

Thank you for the valuable comments. We provide clarification on your concerns as follows. We hope these new theoretical and experimental results can address your concerns, and we look forward to your further discussion.

$~$

W1: In Section 3.1, the authors assume minimal expected distance within the distribution of the offline dataset, overlooking a key limitation: even if specific state-action pairs are present in the offline dataset, the small sample size of some pairs can lead to significant uncertainty between the learned and true dynamics. I recommend that the authors address this limitation directly, discussing its implications for the reliability of their method in practice, particularly under conditions of sparsity in the offline dataset.

• This issue is underscored in prior work on model-based offline RL, as noted in Theorem 1 of [1].
• Figure 1 (the distribution of total variation distance on the WK-m dataset) further illustrates this, where the orange distribution (offline dataset) shows substantial overlap with the blue distribution (OOD actions).

A1: In Line 202~203, we state that "the expected distance within the distribution of the offline dataset (the first term a)) should be minimal". What we intend to convey is that, the expectation of the distance on offline dataset $(s_{\mathcal{D}}, a_{\mathcal{D}})$ is largely smaller than the expected distance on OOD state-action pairs, $(s_{\mathcal{D}}, a_{\neg \mathcal{D}})$ or $(s_{\neg \mathcal{D}}, a)$. Besides, this statement can also be supported by the Theorem 1 of [1], which has been included in Discussion of Theorem 3.1 and 3.2 in new paper version。

The reviewer's concern is that the expected distance on the offline dataset $(s_{\mathcal{D}}, a_{\mathcal{D}})$ be larger due to the infrequent occurrence of certain $(s,a)$. This situation can indeed arise, as illustrated in Figure 1, but it does not conflict with our statement.

$~$

W2: The theoretical results presented indeed appear to rely substantially on prior work by Yu et al. (2020) and Luo et al. (2018), without providing additional theoretical guarantees, such as a formal proof that the policy trained by the DOOF algorithm converges to the optimal policy. This raises questions about the potential advantages of DOOF over existing algorithms. To strengthen their theoretical contribution, the authors should explore convergence guarantees or derive performance bounds for their decoupled approach, highlighting how it compares with established methods in the field.

A2: On the one hand, we acknowledge that the theoretical aspect of this paper is indeed somewhat limited. With the assistance of reference [1], we have supplemented our discussion with guarantees regarding online dynamics finetuning and offline policy finetuning. We are grateful for the valuable feedback and crucial references provided by the reviewers. We have added the following theoretical results to verify the effectiveness of our proposed DOOF:

• Online dynamics finetuning can indeed eliminate distribution shift by learning a more accurate dynamics model, refer to Discussion of Theorems 3.1 and 3.2 for more details.

• Offline policy finetuning can lead to performance improvements compared to the offline pre-trained policy $\pi_{\text{off}}$, refer to Discussion of Theorems 3.3 for more details.

On the other hand, our contribution lies in analyzing the offline-to-online problem from a decoupled perspective within the model-based framework. Although we are not the first to study the offline-to-online in a model-based context, we are the first to introduce a decoupled perspective, fully unleashing the potential of model-based RL in addressing this issue.

$~$

W3: The uncertainty u(s,a) indeed relies on a strong assumption, as seen in Yu et al. (2020). More recent works ([1] and [2]) have addressed this by redefining u(s,a) based on state-action pair frequency and Bellman estimation inconsistency, potentially offering a more precise measure. How about comparing their uncertainty measure with these newer approaches and discussing the benefits or limitations of incorporating these alternative interpretations within their framework?

A3: Our theoretical framework is not built upon the specific form of $u(s,a)$, which makes the proposed decoupled framework universally applicable. This decoupled framework is not only suitable for MOPO but also for any other uncertainty-based methods, including Count-MORL [1] and MOBILE [2]. We have chosen the simplest and most general algorithm to validate the rationality of this framework.

Q1: Observing Equation (6), modifying the reward to encourage optimistic exploration appears very similar to UCB methods, which are commonly used in online RL.

A4: In the paper [3], the conservative term in offline reinforcement learning (RL) is linked to the concept of anti-exploration. From this perspective, the conservative reward $\tilde{r} = r - u$ in MOPO can also be viewed as a form of anti-exploration. Furthermore, the uncertainty policy in DOOF, trained under the guidance of $r^u = r + u$, can be seen as encouraging exploration.

[3] Rezaeifar S, Dadashi R, Vieillard N, et al. Offline reinforcement learning as anti-exploration[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2022, 36(7): 8106-8114.

$~$

Q2: In Table 3, the distance differences seem almost identical regardless of the value of λon. Does this imply that simply collecting synthetic data with optimistic rewards suffices?

A5: In fact, different values of $\lambda_{on}$ in Table 3 lead to varying distances. Some values of $\lambda_{on}$ may paradoxically result in an increased distance after online dynamics finetuning.

Merely interacting with the offline dynamics to collect synthetic data with optimistic rewards is insufficient. Optimistic rewards can cause the collected synthetic data to be out-of-distribution (OOD) compared to the offline dataset, which cannot ensure accuracy.

$~$

Q3: λon is also a hyperparameter; what is the optimal way to select it?

A6: $\lambda_{\text{on}}$ is a hyperparameter that needs finetuning and is closely related to the penalty coefficient $\lambda_{\text{off}}$ of offline training. We use this ratio $\lambda_{\text{on}}:\lambda_{\text{off}} = ratio$ to indirectly choose the hyperparameter $\lambda_{\text{on}}$.

$~$

Q4: In Appendix C.3, the offline penalty coefficient differs from the MOPO official coefficient. Did they tune it separately?

A7: we do not use the official implementation of MOPO. Instead, we use the offlineRL-kit repository(https://github.com/yihaosun1124/OfflineRL-Kit) which the implementation shows better performance than official MOPO repository. We also implement the experiments based on the offlineRL-kit, rather than MOPO official implementation.

Thank you for the valuable comments. We provide clarification on your concerns as follows. We hope these new theoretical and experimental results can address your concerns, and we look forward to your further discussion.

$~$

W1: The idea of the article is relatively intuitive, but its effectiveness lacks theoretical proof. Additionally, there may be an error in Equation (5), as each of the three terms on the right side is missing a constant coefficient.

A1: The most significant innovation of our paper lies in successfully decoupling the offline-to-online transition within the model-based framework into two distinct challenges: a) the elimination of distributional shift and b) policy improvement. Subsequently, we leverage online dynamics fine-tuning to address the distributional shift. We have added the following theoretical results to verify the effectiveness of our proposed DOOF:

• Online dynamics fine-tuning can indeed eliminate distribution shift by learning a more accurate dynamics model, refer to Discussion of Theorems 3.1 and 3.2 for more details.

• Offline policy finetuning can lead to performance improvements compared to the offline pre-trained policy $\pi_{\text{off}}$, refer to Discussion of Theorems 3.3 for more details.

The Equation (5) in original paper is "$=$", while it should be "$\leq$". Thanks again for your careful review. In new version of the paper, this issue has been addressed in Equation (6).

$~$

W2: There is no formal definition of the three types of state-action pairs: ID state-action pairs, ID state but OOD action pairs, and totally OOD state-action pairs.

A2: We apologize for the omission of a formal definition, which we have now included in the new paper version. The definitions are as follows:

1. ID state-action pairs $(s_{\mathcal{D}}, a_{\mathcal{D}})$ indicate that both the state and action belong to the offline dataset $\mathcal{D}$, i.e., $s \in \mathcal{D}$ and $a \in \mathcal{D}$.
2. ID state but OOD action pairs $(s_{\mathcal{D}}, a_{\neg \mathcal{D}})$ indicate that the state belongs to the offline dataset $\mathcal{D}$ while the action does not, i.e., $s \in \mathcal{D}$ and $a \not\in \mathcal{D}$.
3. Totally OOD state-action pairs $(s_{\neg \mathcal{D}}, a)$ indicate that the state does not belong to the offline dataset $\mathcal{D}$, and the action is arbitrary, i.e., $s \not\in \mathcal{D}$ and $a \in \mathcal{A}$. In this case, regardless of whether the action belongs to the offline dataset, the distribution is already significantly distant from the offline dataset, hence the term totally OOD state-action pairs.

$~$

W3: Experiments were only conducted on a subset of the datasets in D4RL (Halfcheetah, Hopper, Walker2D, Pen). Firstly, the computational cost of conducting experiments in D4RL is not high, and there are other environments in Mujoco like Ant, as well as environments in Adroit like Hammer, Door, and Relocate. D4RL also includes Maze-like environments and kitchen-like environments, among others. Secondly, most datasets used in this paper were collected by RL agents, and the feature distribution of these datasets differs significantly from that of data collected by human experts. Therefore, it is unknown whether the method is applicable to human data.

A3: Offline model-based algorithms have performed poorly in other datasets, such as AntMaze, Adroit, and Kitchen. This poor performance hinders the ability to further enhance the algorithms through fine-tuning, preventing us from validating the performance of DOOF across a more diverse set of datasets.

Besides, Pen-human (P-h) is the dataset that contains expert demonstration from human.

$~$

W4: DOOF is a model-based offline RL method, but most of the experimental results are compared with model-free offline RL methods, as shown in Table 4. (According to the introduction in Section 4, only FOWM is a model-based method.)

A4: In the context of model-based RL, the offline to online finetuning algorithms include: MOORe, and FOWM [1].

• MOORe: Model-based Offline-to-Online Reinforcement Learning. MOORe directly adopts the balanced buffer from another model-free offline to online finetuning algorithm [2]. MOORe finetunes both the dynamics model and the policy through online interaction. The code is not public.
• FOWM: Finetuning Offline World Models in the Real World. FOWM is based on TDMPC2 [3, 4] and regularizes the planner at test-time by balancing estimated returns and epistemic model uncertainty. During online finetuning, FOWM finetunes the dynamics model and the policy simultaneously. The official code has been open-sourced.

MOORe simply transplants model-free approaches into model-based scenarios without any specific design considerations. Therefore, we have not included it in our baselines, focusing solely on the comparison with FOWM.

W5: The rationality of the evaluation indicators needs to be questioned. For example, in Section 5.2, TV is calculated on the data in the fake buffer, which satisfies a special distribution. Whether the TV calculated on this distribution is representative is unknown. Similarly, the results in Table 3 can only prove that DOOF can reduce TV on the data distribution represented by the fake buffer.

Q2: Is it reasonable to calculate TV on the data in the fake buffer? Why not randomly sample states and actions in the environment and calculate TV?

A5: W5 and Q2 essentially are the same issue, hence we address them collectively here.

The fake buffer's states are derived from the offline dataset $\mathcal{D}$, while actions are sampled from the policy. Although the policy is conservative, there is still a certain probability of selecting out-of-distribution (OOD) actions, making the fake buffer a combination of $(s_{\mathcal{D}}, a_{\mathcal{D}})$ and $(s_{\mathcal{D}}, a_{\neg \mathcal{D}})$. Such a fake buffer indeed has its limitations.

Therefore, following your suggestion, we have assessed this distance within a uniformly sampled dataset. The results are presented in Table 3 of the revised manuscript.

$~$

W6: The experimental scenarios are not consistent. Table 4 provides experimental results for the four environments Halfcheetah, Hopper, Walker2D, and Pen, while Table 3 does not include experimental results for Pen. Similarly, the experimental results in Fig.1-4 also only include results from some of the environments.

A6: Figure 1 and 2 are used to illustrate the motivation. Besides, we have added results on other datasets in Appendix D. The conclusion derived from these datasets is similar.The new Figure 3 has contained all the datasets in new paper version.

$~$

Q1.1: Regarding Equation (3), by splitting the left side into the sum of the two terms on the right side, can it be demonstrated that the first term will decrease when fine-tuning M^?

Q1.1: When optimizing πoff, will the second term decrease without causing the first term to increase?

A7.1: First and foremost, we have supplemented Theorem 3.2, which, in conjunction with the previous Theorem 3.1, allows us to ensure that the first term in equation (3) can be addressed through online fine-tuning of the dynamic model.

A7.2: Strictly speaking, we are not certain whether the first term increases or decreases. However, we can demonstrate that the policy $\pi^*_{\text{off}}$, obtained by optimizing $\pi^*_{\text{off}}$, is closer to the optimal policy. We have supplemented Definition 3.3, which leads us to this conclusion.

$~$

Q3.1: If online fine-tuning of the world model is possible, why not use a random policy to sample data in the environment and learn the world model entirely based on this data;

Q3.2: or only use reward based on uncertainty and no longer use the original reward from the environment?

A8.1: It is unnecessary and wasteful to learn the dynamic model from scratch, discarding the well-learned offline dynamics. Correcting the errors in the offline dynamic model is a more efficient approach.

A8.2: Discarding the original reward is not reasonable. The online policy should collect data from regions with high rewards and high uncertainty (i.e., areas where the dynamics are not well learned). Using only the uncertainty reward would result in collecting data that includes a large number of low-reward areas, which would be of little help for downstream learning.

$~$

Q4: Fig.2 presents experimental results obtained on the specific task of Walker2D. Whether the insights gained from this result are universal and sufficient to support the motivation for designing the algorithm is questionable.

A9: We use this example to illustrate the exact phenomenon, while similar occurrences can be observed in other datasets as well. To this end, we have supplemented additional experimental results from other datasets in Appendix D.2.

$~$

Reference

[1] Feng Y, Hansen N, Xiong Z, et al. Finetuning offline world models in the real world[J]. arXiv preprint arXiv:2310.16029, 2023.

[2] Lee S, Seo Y, Lee K, et al. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble[C]//Conference on Robot Learning. PMLR, 2022: 1702-1712.

[3] Hansen N, Wang X, Su H. Temporal difference learning for model predictive control[J]. arXiv preprint arXiv:2203.04955, 2022.

[4] Hansen N, Su H, Wang X. Td-mpc2: Scalable, robust world models for continuous control[J]. arXiv preprint arXiv:2310.16828, 2023.

As the author-reviewer discussion period will end soon, we will appreciate it if you could check our response to your review comments. We are always willing to discuss with you about any further concerns or questions. If our response resolves your concerns, we hope the rating of this paper can be elevated. Thanks you again for the time and effort you have invested in our paper!

I would like to express my gratitude to the authors for their elaborate response to my concerns. During the rebuttal period, there have been changes to the manuscript. The formulaic expressions have become more lucid, and a portion of the experiments has been augmented. Nevertheless, your response only partially addresses my concerns: First, the formula reasoning needs to be further improved. For example, I believe that the first line of Equation (14) should be $\int\left[P(s_t,a_t)d_{\mathrm{TV}}\left(P_{\widehat{\mathcal{M}}}\left(s_t,a_t\right),P_{\mathcal{M}}\left(s_t,a_t\right)\right)^2\right]\mathrm{d}(s_t,a_t)$. The deduction process from Eq.(13) to Eq.(14) is problematic. The assumption of "the same reward function" in Theorem 3.1 does not match the actual implementation of your algorithm, and this gap between theory and practice can severely limit the effectiveness of the algorithm. Second, the experimental scenarios are relatively few and simple, and the existence of coding bugs in the environment should not be a reason for the lack of relevant experimental results. In addition, more appropriate comparison algorithms and evaluation indicators should be selected based on the motivation. I hope you can get some inspiration from our discussion. I am inclined to maintain my score.

Thank you for the valuable comments. We provide clarification on your concerns as follows. We hope these new theoretical and experimental results can address your concerns, and we look forward to your further discussion.

$~$

W1： The algorithmic innovations seem limited in the paper, it looks like moving the policy fine-tuning from the online stage into the offline stage with the online collected datasets used for fine-tuning the dynamics model.

A1: From the implementation perspective, our decoupled framework is indeed straightforward, as the reviewer has noted. Our innovation lies in being the first to decouple the offline to online problem into two distinct phases: online distributional shift elimination followed by offline policy performance improvement. Additionally, we have theoretically analyzed this framework, including:

• Online dynamics finetuning can indeed eliminate distributional shift by learning a more accurate dynamics model, refer to Discussion of Theorems 3.1 and 3.2 for more details;
• Offline policy finetuning can lead to performance improvements compared to the offline pre-trained policy $\pi_{\text{off}}$, refer to Discussion of Theorems 3.3 for more details.

In our experiments, we further demonstrate that this innovative framework offers exceptionally high interaction efficiency.

$~$

Q1: Since we have an online stage to finetune the dynamics model, what are the actual advantages of using offline methods to fine-tune the policy instead of using the online methods?

A2: Transitioning from offline algorithms to online algorithms often leads to a sudden drop in performance, as depicted in Figure 1 of Cal-ql [1]; this is not a desirable approach. Utilizing offline algorithms ensures consistency and avoids such a sudden performance decline.

Furthermore, we have supplemented our paper with Theorem 3.3. leveraging it, we demonstrate that the policy obtained from offline finetuning indeed shows an improvement over the offline pretrained policy $\pi_{\text{off}}$.

[1] Nakamoto M, Zhai S, Singh A, et al. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning[J]. Advances in Neural Information Processing Systems, 2024, 36.

As the author-reviewer discussion period will end soon, we will appreciate it if you could check our response to your review comments. We are always willing to discuss with you about any further concerns or questions. If our response resolves your concerns, we hope the rating of this paper can be elevated. Thanks you again for the time and effort you have invested in our paper!

The primary concern (also the weakness), mainly consists in the novelty of our proposed DOOF. Through discussions with other reviewers, we have formulated a better response:

The novelty of DOOF lies in the decoupled framework for offline to online finetuning.

Offline to online finetuning needs to achieve the policy improvement disturbed by the distributional shift. DOOF decouples these challenges. During online stage, DOOF finetunes dynamics model to address the distributional shift. Then during offline stage, DOOF finetunes policy in offline manner using a more accurate and finetuned dynamics model to achieve the policy improvement.

decoupling offers two significant advantages:

• Theoretical aspect: The decoupled framework allows us to leverage existing offline model-based RL theory to analyze and address the offline to online finetuning. Specifically, although Theorems 3.1, 3.2, and 3.3 originate from MOPO [1] or Count-MORL [2], we can derive the guarantees for both online dynamics finetuning and offline policy finetuning. The key here lies in the decoupled framework.
• Experimental aspect: This decoupled framework, equipped with an optimistic uncertainty policy, boasts exceptional interactive efficiency. The key is that the online phase focuses on eliminating distributional shift, laying the groundwork for policy improvement in the offline phase.

[1] Yu T, Thomas G, Yu L, et al. Mopo: Model-based offline policy optimization[J]. Advances in Neural Information Processing Systems, 2020, 33: 14129-14142.

[2] Kim, Byeongchan, and Min-hwan Oh. "Model-based offline reinforcement learning with count-based conservatism." International Conference on Machine Learning. PMLR, 2023.

$~$

$~$

We hope that our above responses can address your concerns. We are always ready for further discussion with you. Should our responses successfully resolve your concerns, we hope you will consider increasing the score accordingly.

Thank you once again for your efforts, and we wish you all the best.

As the author-reviewer discussion phase is coming to an end, we are eagerly awaiting your feedback on whether your previous concerns have been addressed and if there are any further questions. If all your concerns have been resolved, we hope that you can consider raising the rating accordingly.

Regardless, we wish you all the best!

Thank you for the valuable comments. We provide clarification on your concerns as follows. We hope these new theoretical and experimental results can address your concerns, and we look forward to your further discussion.

$~$

W1: The experimental design is not comprehensive enough.

Q2: The number of environments involved in the paper is limited. D4RL is a basic Offline Benchmark, and I suggest evaluating on more updated tasks.

A1: W1 and Q2 essentially are the same issue, hence we address them collectively here.

1.For model-based offline to online baselines, only two algorithms are available:

• MOORe: Model-based Offline-to-Online Reinforcement Learning. MOORe directly adopts the balanced buffer from another model-free offline to online finetuning algorithm [1]. MOORe finetunes both the dynamics model and the policy through online interaction. The code is not public.
• FOWM: Finetuning Offline World Models in the Real World. FOWM is based on TDMPC2 [2, 3] and regularizes the planner at test-time by balancing estimated returns and epistemic model uncertainty. During online finetuning, FOWM finetunes the dynamics model and the policy simultaneously. The official code has been open-sourced.

MOORe simply transplants model-free approaches into model-based scenarios without any specific design considerations. Therefore, we have not included it in our baselines, focusing solely on the comparison with FOWM.

2.For datasets, uncertainty-based offline model-based algorithms are only effective in Gym environments and Adroit's pen tasks. Their performance in other environments is extremely poor, which hinders further finetuning. As a result, the datasets that can be used to validate the DOOF are relatively limited.

[1] Lee S, Seo Y, Lee K, et al. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble[C]//Conference on Robot Learning. PMLR, 2022: 1702-1712.

[2] Hansen N, Wang X, Su H. Temporal difference learning for model predictive control[J]. arXiv preprint arXiv:2203.04955, 2022.

[3] Hansen N, Su H, Wang X. Td-mpc2: Scalable, robust world models for continuous control[J]. arXiv preprint arXiv:2310.16828, 2023.

$~$

Q1: Why can't you compare with some specialized Offline-to-Online algorithms? You can set their Online interaction steps to 10k like yours, and report their Finetuning performance. If such a comparison is possible, I suggest increasing the Online interaction step budget to compare the final asymptotic performance.

A2: The results of on(10k) are the finetuning results using only 10k online data. Since online interaction may be expensive or dangerous, we focus on small online interaction steps.

$~$

Q3: Regarding the off(300k) baseline, does it mean that the initial Offline data, plus the 10k data from online interaction, are used for the final 300k offline gradient steps?

A3: Yes, this statement is right. All the off(300k) results, including DOOF and baselines, are obtained using the offline dataset with another 10k online sampled data. During training, half data is sampled from offline dataset while the other half is sampled from the 10k online data.

Thank you very much for engaging in further discussion with us! With the help of your suggestions and advice, we have once again enhanced the quality of our paper (denoted as V2). Below is our clarification about the futher concerns:

$~$

[ Table 1 vs. Experiment Implementation (A2 follow up) ] The claim in Table 1 seems to not align with the actual experimental setup. In Table 3, the authors report offline finetuning results (i.e., OFF-300K) for other baselines, but Table 1 does not include an offline finetuning stage for these baselines. If the authors also applied Distribution Shift + Policy Improvement in the offline finetuning, Table 1 should be updated to reflect this. I suggest either moving Table 1 to the "Experiments" section for implementation details or replacing it with a textual explanation. Additionally, why is CQL considered to apply distribution shift when it is a model-free method?

A1: Strictly speaking, the claim in Table 1 corresponds to the ON-10K setting. This is because all the baselines include only the online finetuning without an offline finetuning stage. The OFF-300K results for baselines are introduced to make the comparison with DOOF more fair. DOOF finetunes the dynamics model through 10k online interactions and then further improve the policy with an additional 300k offline steps. Comparing DOOF directly with ON-10K would be unfair due to the extra 300k gradient steps.

Furthermore, the purpose of Table 1 is to highlight the innovative aspect of DOOF's framework in 'decoupled' by comparing it with other offline to online algorithms.

CQL is a classical and strong model-free offline to online baseline, although it is not designed for offline to online finetuning. All the offline to online algorithms face the challenge of distributional shift.

$~$

[ Fair Comparison and Confidence in Table 3 Results (A3 follow up) ]

a) Does the implementation technique of "average of best evaluation scores" used in Table 3 also apply to other methods?

b) The absence of variance results in Table 3 and the fluctuations in Figure 6 raise concerns about the reliability of DOOF's offline finetuning improvements. Including the variance for DOOF in Table 3 and Figure 6 would help address these concerns. Figure 6(a) suggests that DOOF only yields comparable finetuning results when 𝜆𝑜𝑛 is set to 2. This raises doubts about the robustness of DOOF, as it seems sensitive to hyperparameters. Re-plotting Figure 6 and adding variance data could help clarify this.

c) In Q2, I noted that the pretrained result for WK-m in Table 4 (original manuscript) is 77.7, which appears to correspond to Figure 6(b) at t=0. However, the authors stated that finetuned results exceed 90, which is confusing, as Figure 6(b) shows no significant increase and indicates results above 80 (>77.7) at t=0. Could the authors clarify this discrepancy?

A2-a): Yes, all the finetuning results are the average of the best results with 3 seeds.

A2-b): The variance of DOOF has been added in Table 6 of Appendix C.4 and the following table also contains this results. Besides, the Figure 6(a) and (b) have been updated with variance in new paper version.

For dataset HC-m (Figure 6(a)), $\lambda_{\text{on}}$ affect the performance heavily. While for WK-m (Figure 6(b)), the performance is relatively robust with different $\lambda_{\text{on}}$.

A2-c): For offline policy training, each epoch contains 1k steps training and one evaluation. So the result at $t=0$ is not the offline pretrained performance. Actually, the $t=0$ is finetuned result with 1k steps, larger than 77.7.

We have recognized the potential for misunderstanding with our previous presentation, and therefore, we have introduced a dashed gray line in Figures 6(a) and 6(b) to represent the offline pretrained results.

Thank you for the valuable comments. We provide clarification on your concerns as follows. We hope these new theoretical and experimental results can address your concerns, and we look forward to your further discussion.

$~$

W1: The proposed method appears to be a technical fine-tuning extension of MBPO, a model-based offline RL method. While the proposed fine-tuning concept could potentially apply to other model-based offline RL methods, this paper focuses solely on MBPO without broader discussion.

A1: Our proposed decoupled framework is general and indeed applicable to other uncertainty-based offline model-based algorithms. In addition to MOPO, our framework is compatible with Count-MORL [1] and MOBILE [2]. The choice of MOPO is motivated by its simplicity and representativeness.

We have expanded the discussion on the universality of the decoupled framework in Section 3.2 of the new paper.

[1] Kim, Byeongchan, and Min-hwan Oh. "Model-based offline reinforcement learning with count-based conservatism." International Conference on Machine Learning. PMLR, 2023.

[2] Sun, Yihao, et al. "Model-Bellman inconsistency for model-based offline reinforcement learning." International Conference on Machine Learning. PMLR, 2023.

$~$

W2: In Table 1, the authors list key differences between the proposed DOOF and other methods. However, there is the conceptual difference between the baseline methods did not attempt offline finetuning and cannot be applied to it. The authors should consider the key differences between DOOF and those methods.

A2: In Table 1, we list the distinctions between our implementation of DOOF and other algorithms. However, the key differences are highlighted in Table 2. Existing algorithms for offline-to-online finetuning, including model-based methods, attempt policy improvement while eliminating distributional shift during the online phase. In contrast, within the model-based framework, we have linked the dynamics model finetuning to the elimination of distributional shift. This has led to the decoupled framework that addresses distributional shift through the dynamics finetuning, followed by offline policy improvement.

Furthermore, we fully concur with the reviewer's perspective on Table 1. In order to draw greater attention to the key differences presented in Table 2, we have decided to remove Table 1.

W3: The experimental results lack convincing evidence. For example, only one seed is used in Table 4, despite Fig. 6(a) showing significant variance during offline fine-tuning. Additionally, there seems to be a statistical mismatch between Table 4 and Fig. 6. Please see detailed feedback in Questions.

Q2: There appears to be a statistical mismatch between Table 4 and Figure 6. For example, DOOF achieves 92.5 for HC-m in Table 4 after off(300k) finetuning, but in Figure 6, the highest results fluctuate around 80. Additionally, the pretrained offline result for WK-m in Table 4 is 77.7, yet almost all DOOF results with various coefficient factors are higher than 80. Therefore, using only one seed does not convincingly demonstrate the robustness of the improvement.

A3: W3 and Q2 essentially are the same issue, hence we address them collectively here.

Table 4 (now is the Table 3 in new paper version) actually represents the average across three seeds; due to space limitations, the standard deviations have been omitted.

In Figure 6, we indeed only present the training curve for one seed, resulting in relatively larger fluctuations. But actually, we don't have mismatch. As finetuning is to achieve the best performance as we can,so the results in Table 4 is the average of best evaluation scores of multiple policies that we saved during offline finetuning stage. It's true that some intermediate finetuning policy did indeed achieve better evaluation results at certain moments due to learning better behavior, which is consistent with the best results presented in Table 4.

As for WK-m in Figure 6 (b), the finetuning results with various coefficient factors are higher than 90, not only higher than 80.

$~$

Q1: What is the value of the coefficient parameter λon in Table 4?

A4: $\lambda_{\text{on}}$ is a hyperparameter that is closely related to the penalty coefficient $\lambda_{\text{off}}$ of offline training. We tune the value of $\lambda_{\text{on}}$ based on the offline penalty coefficient $\lambda_{\text{off}}$, with $\lambda_{\text{on}}:\lambda_{\text{off}} = 0.25,0.5,1,2$. We have listed $\lambda_{\text{on}}$ values in Table 6 of the appendix.

$~$

Q3: I appreciate that the authors demonstrate and discuss the dynamics estimation error with different quality datasets under Table 3. Similarly, the final improvements with varying dataset qualities could be discussed in depth. While data are presented in Table 4, it might be more straightforward to compare the average improvement across all tasks (HC, HP, WK) under different dataset qualities (r, m, mr) in a figure. Additionally, the authors mention “medium-expert” in the caption of Table 4 but do not include the results. It would be more comprehensive to include these results in future revisions if such datasets are available.

A5: Thank you very much for your suggestions. we have computed the average improvement across all tasks under different dataset qualities as listed in Table 7 in appendix E. As we can see, our method DOOF outperform all other baselines on Gym tasks.

As for the task on medium-expert, it has achieved near optimal scores during offline stage, leaving limited room for policy improvement. Therefore, these expert datasets are not suitable for offline to online finetuning.