Online Pre-Training for Offline-to-Online Reinforcement Learning

We appreciate the reviewer for the valuable and constructive feedback. We respond to each point in detail below and will reflect the suggested improvements in the revised manuscript.

 

R4-1. Questions

• Question about umaze dataset

According to the official D4RL[4-1] paper, the Antmaze environment consists of three types of datasets: umaze, diverse, and play. Among them, the ”play” dataset is collected by commanding the ant to navigate from various hand-picked start locations to hand-picked goals.

In contrast, the umaze dataset, which appears in the first row of Table 2, uses a single fixed start and goal location without diversity in either. Since this setup does not match the definition of a play dataset, it is not categorized as one.

We will clarify this point in the revised manuscript in Appendix B.2.

• Question about training protocol

All experiments in our paper follow a training protocol consisting of a 1M offline phase and a 300k online phase. For OPT, the first 25k steps of the online phase are used for online pre-training, while the remaining steps are allocated for online fine-tuning. This setup is also applied in Figure 3.

Since the key changes in OPT occur during the online phase, Figure 3 and Appendix H present performance curves for the online phase. This is reflected in the initially flat performance observed in the graphs, as OPT does not update the policy during online pre-training.

To avoid any confusion, we will explicitly clarify this in the caption of Figure 3 and in Appendix H.

 

R4-2. Sensitivity to $\kappa$

To provide clearer evidence for the sensitivity of our method, we conduct additional experiments in the Antmaze domain. Table F at the anonymous link (https://sites.google.com/view/icml2025opt/) highlights two key findings:

• First, the comparison with fixed $\kappa$ values demonstrates the necessity of scheduling $\kappa$ during training.
• Second, the results indicate that performance is not sensitive on a specific $\kappa$ value; rather, the crucial factor is the gradual transition from $Q^{\text{off-pt}}$ to $Q^{\text{on-pt}}$

These findings validate both the importance of $\kappa$ scheduling and the robustness of the approach to sensitivity in its exact values.

 

R4-3. Regarding Writing and Reference

We sincerely thank the reviewer for the detailed and thoughtful feedback on the writing. We appreciate the time and care taken to point out areas for improvement, as well as for providing helpful references. All suggestions will be carefully considered and reflected in the revised manuscripts, and we believe these revisions will improve the clarity and overall quality of the paper.

 

R4-4. Regarding Figure 6

We thank the reviewer for pointing out the insufficient information provided in Figure 6 regarding variability and statistical significance.

To provide further clarity, we include environment-specific results corresponding to Figure 6 in Table G in the anonymous link (https://sites.google.com/view/icml2025opt/). These results show that performance degradation is especially noticeable in the random dataset. We will include these additional results in the revised manuscript.

 

Once again, we greatly appreciate the reviewer's thoughtful comments and suggestions. The feedback has been instrumental in improving the quality and clarity of our work. We hope our responses sufficiently address the concerns raised.

 

[4-1] Fu, Justin, et al. "D4rl: Datasets for deep data-driven reinforcement learning." arXiv (2020).

We thank the reviewer for the constructive feedback, which helps improve both the clarity and completeness of our analysis. We address each point below.

 

R3-1. Regarding Statistical Significance

To reduce statistical uncertainty, we increase the number of random seeds from 5 to 10 for all baselines for Tables 1-3 and 10, and report 95% confidence intervals (CI) instead of standard deviation to better reflect the reliability of the results. Tables A-C at the anonymous link (https://sites.google.com/view/icml2025opt/) show that OPT consistently outperforms baselines, with minimal CI overlap, indicating statistical significance.

Table E further supports this trend when combined with IQL, demonstrating robustness across backbone algorithms. For clarity, we bold entries whose CIs include the highest mean value among compared methods. We will include these results in the revised manuscript.

 

R3-2. Regarding Aggregated Statistics

We appreciate the suggestion to use aggregated metrics such as the Interquartile Mean (IQM) and include corresponding comparisons. Figure D at the anonymous link (https://sites.google.com/view/icml2025opt/) shows our method consistently achieves the strongest performance.

We also provide aggregated learning curves. As shown in Figure E, our method demonstrates consistent improvement, especially strong result in the Adroit domain.

These results support the statistical significance and overall effectiveness of our approach. We will include them in the revised manuscript.

 

R3-3. OPT with Simple Alternatives

As described in Section 3.1, Online Pre-Training initializes the new value function to enable effective online fine-tuning. To achieve this, we proposed a meta-learning strategy, formalized in Equation 3.

The second term in Equation 3 updates the value function using online samples while incorporating gradients from $\mathcal{L}^{\text{off}}$. This aligns two terms, allowing $Q^{\text{on-pt}}$ to generalize across offline and online data. To assess its usefulness, we compare it with a simpler alternative that jointly trains on $B_{\text{off}}$ and $B_{\text{on}}$, as follows:

$\mathcal{L}_{Q^{\text{on-pt}}}^{\text{pretrain}}= \mathcal{L}^{\text{off}}(\psi) + \mathcal{L}^{\text{on}}(\psi)$

This experiment is shown in Table D of our anonymous link (https://sites.google.com/view/icml2025opt/). The variant Pre-trained with $B_{\text{off}}$ and $B_{\text{on}}$ shows a performance drop in harder tasks e.g. large mazes, due to conflicting learning dynamics between two terms, which the simpler method cannot resolve. In contrast, our method reconciles the two objectives, enabling effective online fine-tuning.

These results highlight the value of the meta-adapation objective under distribution shift. We consider this a key contribution and will include this discussion in the revised manuscript.

 

R3-4. Ablation Study for Equation 4

We conducted experiments in Antmaze to assess the necessity of Eq. 4.

Table F at the anonymous link (https://sites.google.com/view/icml2025opt/) compares $\kappa$ scheduling with fixed alternatives. While fixed $\kappa$ works in simple tasks, performance degrades in harder environments. In particular, $\kappa=0.5$ causes instability due to prolonged reliance on $Q^{\text{off-pt}}$.

We also observe that performance is not sensitive to a specific $\kappa$ value, suggesting the key factor is a gradual shift from $Q^{\text{off-pt}}$ to $Q^{\text{on-pt}}$. These results underscore the importance of Equation 4 for stable and effective training.

 

R3-5. Computational Cost

We compare the wall-clock time of TD3 and our method. As shown in Figure F at the anonymous link (https://sites.google.com/view/icml2025opt/), TD3 takes about 4000 seconds, while ours takes around 6000 seconds due to the added value function and Online Pre-Training ($N_\tau = 25k$, $N_{\text{pretrain}}=50k$).

Although this introduces extra overhead, we believe the performance gains reasonably justify the additional cost. We will include this analysis in the appendix.

 

R3-6. Regarding Theoretical Justification

Our work aims to empirically address challenges in offline-to-online RL. Although it does not include theoretical justification, the results support the effectiveness of our design. We recognize the value of theoretical analysis and consider it as a promising future direction.

 

R3-7. Regarding Missing Standard Deviation

We apologize for omitting standard deviations in Table 11 and Figures 4 and 6. We will include them in the revised manuscript for proper assessment.

 

We appreciate the reviewer again for the valuable feedback and hope our response sufficiently addresses the concerns. We believe our method achieves statistically significant performance and is readily applicable to a wide range of algorithms. We hope this to be considered in the reviewer’s reevaluation of our work.

We appreciate the reviewer for the detailed and insightful comments. The suggestions are highly valuable in helping us refine both the presentation and the depth of our analysis. We carefully address each point below.

 

R2-1. Regarding Asymptotic Performance of Figure 4

We appreciate the reviewer for raising this interesting point. To investigate the asymptotic performance for different initialization strategies, we provide learning curves in Figure B at the anonymous link (https://sites.google.com/view/icml2025opt/). As discussed in Section 5.1 of our paper, Random Initialization initially causes an early performance drop due to unstable value estimates, which in turn results in lower asymptotic performance. Pre-trained with $B_{\text{on}}$ achieves performance comparable to OPT in medium and medium-replay datasets but shows weak asymptotic performance on the random dataset. These results further support the effectiveness of the initialization strategy employed in our method.

 

R2-2. Regarding of $\kappa$ Scheduling

The motivation behind the design is that during online fine-tuning, $Q^{\text{off-pt}}$ becomes less reliable due to distribution shift, resulting in inaccurate value estimation. To address this, we introduce $\kappa$ scheduling strategy that gradually shifts emphasis toward $Q^{\text{on-pt}}$, enabling more effective online fine-tuning.

We empirically validate this approach through an experiment comparing $Q^{\text{off-pt}}$ and $Q^{\text{on-pt}}$ against an optimal value function during the online fine-tuning. The optimal value function is estimated by training a TD3 agent until it achieves near-optimal performance. For a fair evaluation, we use 10 fixed state-action pairs, where states are sampled from the initial state distribution, and actions are selected using the optimal policy.

As shown in Figure C at the anonymous link (https://sites.google.com/view/icml2025opt/), $Q^{\text{off-pt}}$ exhibits higher estimation bias due to its inability to adapt under distribution shift. In contrast, $Q^{\text{on-pt}}$, aided by Online Pre-Training, effectively reduces estimation bias. These results highlight that a gradual shift from $Q^{\text{off-pt}}$ to $Q^{\text{on-pt}}$ serves as an effective mechanism for mitigating estimation bias during online fine-tuning.

 

R2-3. Analysis Experiments on Other Domains

We agree that extending the analysis experiments in Section 5 to all environments can improve the comprehensiveness of our study. We are currently conducting analysis experiments, and one preliminary result is shown in Table D at the anonymous link (https://sites.google.com/view/icml2025opt/), which exhibits patterns consistent with those in Figure 4 of the main paper. We plan to include results covering all environments and analyses in the revised manuscript.

 

R2-4. Regarding Different Backbone Algorithms

The reason for using different backbone algorithms is that no single algorithm consistently performs well across all domains. For example, Off2On[2-1] performs well in MuJoCo but is not evaluated on Antmaze, and our reproduction did not perform well there. Conversely, SPOT[2-2] performs well in Antmaze but underperforms in MuJoCo. As each algorithm tends to be specialized for certain benchmarks, we choose the most suitable backbone for each to better assess the effect of our method.

This is possible because our method is designed to be flexible and readily applicable to a wide range of algorithms. We will clarify this in the revised manuscript to avoid potential confusion.

 

R2-5. Regarding RLPD

Our work focuses on methods in the offline-to-online RL framework, which includes a distinct offline phase. For this reason, the current manuscript does not include RLPD[2-3] in the main results, as it is primarily designed for online RL settings and does not involve a separate offline phase. However, to prevent confusion in baseline comparisons, we will include RLPD in Table 1 in the revised manuscript, along with a brief explanation clarifying its distinction.

 

We sincerely thank the reviewer again for the constructive comments. The feedback helped us strengthen both the empirical evidence and the clarity of our manuscript. We hope our responses have sufficiently addressed the concerns and contributed to a clearer understanding of our method.

 

[2-1] Lee, Seunghyun, et al. "Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble." CORL 2022.

[2-2] Wu, Jialong, et al. "Supported policy optimization for offline reinforcement learning." NeurIPS 2024.

[2-3] Ball, Philip J., et al. "Efficient online reinforcement learning with offline data." ICML 2023.

We sincerely thank the reviewer for the thoughtful and constructive feedback. The comments help improve the clarity and completeness of our work. Below, we address each point in detail.

 

R1-1. Regarding Experimental Results

Following the reviewer’s suggestion, we increase the number of random seeds from 5 to 10 to strengthen the claim regarding the effectiveness of OPT.

In Table A-C of the anonymous link (https://sites.google.com/view/icml2025opt/), we report the mean and 95% confidence interval (CI) rather than standard deviation to convey the reliability of the results. OPT consistently outperforms the baseline, and the minimal CI overlap supports the statistical significance of the results.

We also provide Interquartile Mean (IQM) [1-1] comparisons in Figure D for each domain, which further confirm that our method achieves strong performance. Additionally, we will increase the number of random seeds for Tables 4 and 7 and include the corresponding updated results in the revised manuscript.

 

R1-2. Intuitive guide of balancing strategies

We appreciate the reviewer for pointing out this insightful consideration. Although we do not provide a theoretical analysis for selecting the weighting coefficient $\kappa$, we offer the following intuitive guideline based on our empirical observations:

• If the dataset quality is moderate, we recommend setting $\kappa$ such that the emphasis on $Q^{\text{on-pt}}$ gradually increases during online fine-tuning.
• Conversely, if the dataset quality is relatively low, it is beneficial to set $\kappa$ to utilize $Q^{\text{on-pt}}$ from the early stage of online fine-tuning.

We will include this practical guideline in the revised manuscript to provide clearer guidance on applying our method across various tasks.

 

R1-3. Regarding Additional Experiments on Value Estimation

Since the motivation of our method is to improve value estimation during online fine-tuning, we conduct additional experiments to verify whether the combined value function benefits from OPT.

To evaluate this, we compare the value estimation of TD3 and TD3+OPT in three environments where OPT shows significant performance gains: halfcheetah-medium-replay-v2, hopper-random-v2, and walker2d-medium-v2. As an approximation of the optimal value function, we train a TD3 agent with sufficient steps until it reaches near-optimal performance and use its value function as a reference.

For a fair comparison, we sample 10 fixed state-action pairs, where states are drawn from the initial state distribution and actions are selected using the optimal policy. We then measure the value estimation bias of TD3 and TD3+OPT throughout online fine-tuning.

Figure A of the anonymous link (https://sites.google.com/view/icml2025opt/) presents how the estimation bias evolves during online fine-tuning. These results indicate that TD3 exhibits noticeable esimation bias due to distribution shift, whereas our method reduces bias early in training by leveraging $Q^{\text{on-pt}}$ trained specifically to handle online samples.

To further support this, we conduct an additional experiment using the same estimation bias metric, but this time comparing $Q^{\text{off-pt}}$ and $Q^{\text{on-pt}}$ individually. As shown in Figure C, $Q^{\text{on-pt}}$ reduces estimation bias more rapidly than $Q^{\text{off-pt}}$, indicating that the improvement observed above stems from the effective adaptation of $Q^{\text{on-pt}}$ to online samples.

These findings confirm that our method enhances value estimation, leading to more stable and effective online fine-tuning. We will include this analysis in the revised manuscript to support our claims.

 

We appreciate the reviewer's detailed and insightful feedback once again. The suggestions have meaningfully contributed to improving the clarity and rigor of our work. We hope that our responses have adequately addressed the concerns raised.

 

[1-1] Agarwal, Rishabh, et al. "Deep reinforcement learning at the edge of the statistical precipice." NeurIPS 2021.