Online Pre-Training for Offline-to-Online Reinforcement Learning

We sincerely thank the reviewer for their thoughtful feedback and for recognizing the significance of our work and its contributions.

 

W1. Regarding Motivation

The value function learned by offline RL, trained on a fixed dataset, often provides inaccurate value estimation for out-of-distribution samples [4-1]. Overfitting to the offline dataset may also contribute to this issue, particularly when the dataset is of low quality [4-2]. During online fine-tuning, the value function naturally encounters such OOD samples, and these inaccurate estimations can negatively impact policy learning and hinder performance improvement, as illustrated in Figure 1 of our paper.

Recent studies such as [4-3,4-4] address this challenge indirectly by adopting conservative policy learning strategies during the online phase to mitigate the effects of inaccurate value estimation. In contrast, OPT is specifically designed to directly tackle this issue of inaccurate value estimation during the transition from offline to online learning, making it a distinct approach with a focused objective.

 

W2. Regarding Sample Efficiency

We appreciate your alternative suggestion regarding the use of a new policy during Online Pre-Triaining.

The characteristic of the backbone model influences the relatively modest performance of OPT in the early stages of training. Since OPT builds upon the pre-trained model, the initial performance reflects the early stage performance of backbone model. However, as training progresses, $Q^{\text{on-pt}}$ effectively adapts to the samples generated by the improving policy, resulting in a noticeable performance increase, as shown in our learning curves.

The suggestion to incorporate a new policy during Online Pre-Training is an intriguing alternative. However, we utilize $\pi^{\text{off}}$ because it already represents a policy derived from the offline dataset, providing a stable and efficient foundation for subsequent online fine-tuning while avoiding the additional exploratory overhead of training a new policy. Exploring the integration of a new policy remains an interesting avenue for future work.

 

W3. Regarding OEMA and Balanced Replay

We provide a detailed explanation of the components of the proposed algorithm in Common Response C.1, where we also demonstrate the contribution of each component through experimental analysis. Furthermore, Common Response C.3 elaborates on the necessity and role of meta adaptation within OPT.

Balanced Replay[4-5] is employed in OPT to ensure that $Q^{\text{on-pt}}$, which is designed to adapt effectively to online samples can fully utilize the online data during fine-tuning. To analyze the impact of Balanced Replay on OPT, we conduct experiments using an alternative replay strategy, Online Replay, which exclusively utilizes online samples for training. As detailed in Appendix C.1, the results show that even with Online Replay, OPT outperforms existing baselines. This indicates that the success of OPT is not solely reliant on Balanced Replay.

Additionally, since OEMA[4-6] also employs Balanced Replay, the comparison between OPT and OEMA highlights that OPT's performance gains are not attributed to Balanced Replay or OEMA individually but rather to their effective combination within the overall algorithm. By integrating these components thoughtfully, OPT achieves strong performance without replying on any single technique as the primary driver of its success.

 

Q1 & Q2. Utilizing $Q^{\text{off-pt}}$ and $Q^{\text{on-pt}}$

At the beginning of online fine-tuning, since the policy has been trained on the offline dataset, it generates samples similar to those in the offline dataset. In this setting, $Q^{\text{off-pt}}$, thoroughly trained on the offline dataset, is expected to support the policy learning. Based on this, we continue updating $Q^{\text{off-pt}}$ during online fine-tuning and initially assign a lower $\kappa$ value to prioritize its influence.

However, as the policy improves and its distribution begins to diverge from the offline dataset, we gradually increase the weight of $Q^{\text{on-pt}}$, utilizing the adaptation effectively to the online samples. This dynamic balancing leverages the $Q^{\text{off-pt}}$ in the early stages and $Q^{\text{on-pt}}$ in subsequent stages, enabling the method to integrate the advantage of both value functions throughout online fine-tuning.

We sincerely thank the reviewer for their thoughtful feedback and positive remarks on our work. We address the raised points below to provide additional clarity and support for our approach.

 

W1 & Q4. Computational Cost of OPT

To evaluate computational efficiency, we compare the wall-clock running time of TD3+OPT and TD3 in the walker2d-random-v2 environment, where TD3+OPT achieves the most significant performance gains. The comparison results are presented in Figure 1 of our anonymous link (https://sites.google.com/view/iclr2025opt).

For TD3, the wall-clock time is approximately 4000 seconds. In contrast, TD3+OPT requires 6000 seconds, primarily due to the additional steps for Online Pre-Training ($N_{\tau}$ and $N_{\text{pretrain}}$ are set to 25,000 and 50,000 steps, respectively) and the updates associated with the addition of the new value function during online fine-tuning.

Although OPT requires additional time compared to TD3, this increase reflects the added effort to train the new value function, which is integral to enhancing adaptability and performance during online fine-tuning. The trade-off is balanced, as the additional computational cost results in substantial improvements in overall performance.

 

W2. Regarding Hyper-Parameters

OPT introduces a small number of hyperparameters to support its components, including Online Pre-Training and the use of two value functions during online fine-tuning. These hyperparameters are chosen to require minimal tuning while supporting the method’s effectiveness:

• $N_{\tau}$ and $N_{\text{pretrain}}$: Consistently set across all domains, with $N_{\tau}$ analyzed in detail in Section 5.4.
• $\alpha$: Matches the learning rate of the backbone model, ensuring straightforward configuration.
• $\kappa$: Balances the contributions of $Q^\text{off-pt}$ and $Q^\text{on-pt}$. An analysis of its impact is provided in Common Response C.3.

Overall, while OPT introduces these hyperparameters, their number is minimal, and they are designed to work effectively with simple configurations. Detailed analyses in the paper further support their practical application across diverse settings.

We sincerely thank the reviewer for their insightful feedback and for highlighting key aspects of our work. Your comments have helped us further clarify our approach, and we provide detailed responses to the raised concerns and questions below.

 

W1. Regarding Meta Adaptation Strategy

We provide a detailed explanation of meta adaptation and its role in Common Response C.3. Meta adaptation is a key component that enables $Q^{\text{on-pt}}$ to adapt effectively to online samples during online fine-tuning.

The second initialization method mentioned in Section 5.1 aligns with the approach described in the question, where the new $Q^{\text{on-pt}}$ is trained solely on online data collected from a frozen pre-trained policy. However, as the data is generated by a fixed policy, the experimental results show that this approach leads to overfitting to $B_{\text{on}}$. This overfitting hinders $Q^{\text{on-pt}}$'s ability to adapt to the new samples generated as the policy improves, resulting in degraded performance.

As a result, this highlights the necessity of meta adaptation in ensuring effective learning and adaptability of $Q^{\text{on-pt}}$ during online fine-tuning.

 

W2. Regarding Figures

Thank you for your comment regarding the blurry figure legends in Figure 5 and Appendix G. We have revised the figure legends to improve clarity and readability. Additionally, we have added detailed captions and included appropriate x/y axis labels. These updates have been incorporated into the revised manuscript for your review.

 

Q1. Regarding $\kappa$

In Common Response C.2, experiments with alternative $\kappa$ scheduling (Appendix C.2, Table 6 of the revised paper) demonstrate that performance is consistent regardless of the specific scheduling values. This highlights that the gradual transition from $Q^{\text{off-pt}}$ to $Q^{\text{on-pt}}$ is the key factor for effective adaptation, rather than the precise scheduling details.

We sincerely thank the reviewer for their constructive feedback and for highlighting the strengths of our work. We address the raised concerns and questions below to provide further clarification and support for our approach.

 

W1 & W3. Clarification of Algorithm Components

We explain each component of the proposed method and their respective contribution in the Common Response C.1. The explanation of meta adaptation strategy and its role are discussed in Common Response C.3.

Performing TD learning solely on $B_{\text{on}}$ is indeed a potential approach for Online Pre-Training. However, since $B_{\text{on}}$ is generated from a fixed policy, it lacks diversity and contains a limited number of samples. As as result, $Q^{\text{on-pt}}$ is prone to overfitting, which is evidenced by the results presented in Section 5.1 of our paper.

 

W2. Regarding $\kappa$ Scheduling

We provide additional analysis and experiments related to $\kappa$ in the Common Response C.2. These experiments demonstrate that the specific values of $\kappa$ have a minimal impact on overall performance, as long as the balance shifts gradually over time. Further details on the scheduling parameters can be found in Appendix D of our paper.

 

Q1. Regarding of t-SNE Visualizations

To complement the observations presented in the t-SNE visualizations in Section 5.2, we further evaluate the distributional differences using Maximum Mean Discrepancy (MMD) in the walker2d environment.

The MMD results reveal a clear distributional difference in random dataset, while the medium and medium-replay datasets exhibit relatively closer. These findings align closely with the trends observed in the t-SNE visualization, confirming the consistency of our results across different evaluation methods.

 

Q2. Clarification of Algorithm Components

As detailed in Appendix D, we use TD3 with an update-to-data (UTD) ratio of 5 in the MuJoCo environment as the backbone model for our experiments. For a fair comparison, we applied the same UTD ratio to TD3 in Table 1. This adjustment appears to result in TD3 outperforming other baseline algorithms.

In the revised paper, we explicitly clarify that TD3 is evaluated with a UTD ratio of 5 in Section 4.1.

 

Q4. Regarding Ablation for Online Pre-Training

To validate the importance of combining Online Pre-Training with a new value function, we conduct an ablation study in Section 5.1 by directly using a random initialized value function for online fine-tuning, without any pre-training. This setting, referred to as Random Initialization, evaluates whether adding a new value function alone is sufficient to achieve strong performance.

The experiment results, presented in Figure 7 of our paper, demonstrate that $Q^{\text{on-pt}}$, when initialized randomly and fine-tuned without pre-training, generates random and inaccurate estimates during the early stage of online fine-tuning. These poor initial estimates limit policy improvement, resulting in significantly degraded overall performance.

This confirms that the mere addition of a new value function is insufficient. Instead coupling it with Online Pre-Trianing enables $Q^{\text{on-pt}}$ to effectively adapt to online samples, facilitating stable and efficient learning during online fine-tuning.

 

Q3 & Q6. Regarding Table 4 and Figure 2

In response to the observation regarding the confidence intervals in Table 4, we revise the table to ensure that entries are not bolded in cases where confidence intervals overlap significantly.

Regarding the use of the fire symbol in Figure 2, we understand your concern and remove the symbol to avoid potential confusion, following your suggestion.

 

Q5. Comparison with TD7 [1-1]

TD7 adopts a similar approach by using a fixed policy to collect batches of data, which are then used for updates. This design is indeed comparable to our method. However, while TD7 employs policy checkpoints primarily for stability, our approach focuses on enabling $Q^{\text{on-pt}}$ to adapt effectively to online samples.

Additionally, our method applied this strategy only once during the learning process, whereas TD7 continuously relies on this mechanism throughout training. This distinction highlights the differing objectives and applications of the two methods.

 

[1-1] Fujimoto, Scott, et al. "For sale: State-action representation learning for deep reinforcement learning." NeurIPS 2024.