Offline-to-Online Reinforcement Learning with Prioritized Experience Selection

We thank the reviewer for insightful comments. Please find our clarifications to the concerns below. If we are able to address the concerns, we hope the reviewer can reconsider the score.

Concern 1: On the mitigation of distribution shift issue

First, we want to say that generating high-performance trajectories with online policy is not difficult because the initial trajectories in the priority queue come from the dataset, and the policy has already been pre-trained on the offline dataset, usually performing better than the behavioral policy of the offline dataset. Therefore, the priority queue is likely to be continuously updated. Additionally, we dynamically relax the selection threshold, which means that more diverse data will be added to the replay buffer, enhancing the exploration capability of the policy.

Concern 2: Baseline of FIFO buffer

We thank the suggestion of the additional baseline. We implement the FIFO baseline as follows: instead of initializing the priority queue with the offline dataset, we continuously added online sampled trajectories in a FIFO manner, with the rest of the process being the same as PES. We conduct experiments on 12 D4RL MuJoCo datasets and present the results below. It can be seen that PES significantly outperforms the FIFO baseline, achieving higher performance on 9 out of 12 datasets. We believe this is because, although FIFO mitigates distribution shift, it cannot ensure the high quality of the samples used for training, as online samples may be of low quality. In contrast, PES can both mitigate distribution shift and ensure the high quality of samples.

Concern 3: The detail of IQL-Expert

First, we will elaborate on the process of the IQL-Expert baseline. Initially, we load the data from the halfcheetah-medium-v2 dataset into the replay buffer and train the IQL algorithm for 1 million steps, which is the same as the training process for general offline algorithms. Subsequently, we sequentially add the data from the halfcheetah-expert-v2 dataset (approximately 1 million data points) to the replay buffer (which still contains medium quality data), and we train the IQL algorithm once for each added data point, totaling 1 million steps. We believe that the reason the reviewer's results differ from ours is that the reviewer fine-tuned directly on the entire dataset of halfcheetah-expert-v2, whereas we sequentially added data to the replay buffer (which is more in line with the situation during online fine-tuning). Secondly, we argue that even if the final performance of the algorithm is high, the initial performance collapse caused by distribution shift is still important. For example, in safe reinforcement learning, we not only need good final performance but also need to satisfy certain safety constraints. Sudden performance collapse may violate safety constraints and bring hidden dangers.

Concern 4: On the Pixel inputs

Although PES is not an algorithm designed for pixel inputs, we understand the reviewer's concern. Therefore, we select several datasets from V-D4RL and compare the performance and computation time of IQL and IQL+PES. We present the performance comparison below, which shows that under visual inputs, IQL+PES still outperforms IQL on 3 out of 4 datasets, demonstrating the superiority of PES. As for computation time, the average training time for IQL across the 4 datasets is 9 hours and 14 minutes, while the average training time for IQL+PES is 13 hours and 22 minutes, which is significantly longer than IQL. We acknowledge that the training efficiency under visual inputs is reduced due to KNN search, but this can be mitigated by using visual encoders to reduce dimensionality, which we will consider as future work.

Many thanks to the authors for the explanations and additional experiments. I would like to raise my score to 5. I didn't raise to a higher score because: (1) I still find the Theorem 3.1 is less directly connected to the central claim of this paper. I would like to see some analysis about how does the current policy distribution shift affect the policy learning, and when does it hurt. (2) Authors mentioned about the initial performance collapse issue in offline-to-online fine-tuning in the rebuttal, but this point is not well-discussed in the main paper. A closely related work on this issue is not compared (https://github.com/YiqinYang/BOORL). (3) Adding one sample at a time is closer to the fine-tuning setting, however, adding a sample from the expert dataset is not close to the fine-tuning setting. If we use the offline policy to collect online samples, we can only obtain some similar samples from the offline dataset, which has a much smaller distribution shift than the one from the expert dataset.

We thank the reviewer for raising the score, and we are pleased to address the remaining concerns. Please find our clarifications to the concerns.

Concern 1: Theorem 3.1 is less connected to the central claim

Our central claim is that distribution shift may lead to performance collapse, and we introduce PES to mitigate this issue. Theorem 3.1 is connected to our central claim as it reveals that when the empirical MDP between two consecutive steps differs greatly (i.e., the distribution shift is severe), performance collapse may occur. From theorem 3.1, we can also infer that if the distribution shift between two steps is severe and the empirical distribution is greatly different from the true distribution, the policy may not improve steadily, thus hindering policy learning. We note that theoretically analyzing the impact of policy distribution shift is challenging since the policy is optimized via SGD. If we were to analyze the deterministic policy improvement, many assumptions would need to be introduced, and these assumptions may not hold under practical scenarios. Therefore, the theoretical guarantee of policy improvement is beyond the scope of this paper, but we believe this is an interesting point and can be further investigated in our future work.

Concern 2: The initial performance collapse is not well-discussed

Firstly, we respectfully argue that we have discussed the phenomenon of initial performance collapse in our paper since it is our central claim. We mention this issue in our paper in line 13, 45, 310 and so on, and our theoretical analysis is also related to the initial performance collapse phenomenon. Secondly, we thank the reviewer for providing BOORL as another baseline. We therefore integrate PES into BOORL and conduct experiments on various D4RL datasets, and present the results below. We can see that PES can benefit BOORL in 8 out of 12 datasets, indicating the effectiveness and versatility of PES. Although the improvements are not quite significant on some datasets (e.g., from 97.2 to 99.5 on half-me), this is because BOORL is already well-performing on D4RL datasets.

Concern 3: Adding samples from expert dataset is not close to fine-tuning

Yes, we agree that adding samples from expert dataset is not close to fine-tuning manner. However, the aim of the motivating example is not to simulate the fine-tuning phase, but to create a setting with severe distribution shift to better illustrate the superiority of PES, although this setting may not occur in practice. We add samples from expert dataset since the samples in the expert dataset has a distinct distribution shift from the medium dataset.

We thank the reviewer for insightful comments. Please find our clarifications to the concerns below. If we are able to address the concerns, we hope the reviewer can reconsider the score.

Concern 1: On the naive design of PES

We respectfully disagree with that. On the one hand, we have clarified in Appendix B that under the Lipschitz assumption (a widely used assumption in reinforcement learning), PES has the ability to select high-quality samples. Furthermore, in Step 3 of PES, we gradually loose the selection threshold to alleviate the issue of insufficient exploration, ensuring the diversity of samples in the later stages of training. On the other hand, using KNN distance to measure the similarity of samples is widely used in reinforcement learning [2,3,4], so we believe that the method based on distance measurement is not naive but a simple and effective approach.

Concern 2: On the low score of Cal-QL

The performance difference of Cal-QL might be due to the differences in the codebases used. To facilitate the deployment of PES, we uniformly used the baselines implemented in the CORL [1] codebase, including CQL and Cal-QL, rather than the official code, as mentioned in Appendix E.1. We directly ran the Cal-QL implementation provided by CORL and presented the results in our paper. We did not intentionally suppress the baseline performance; we have provided our source code in the supplementary materials to ensure the reproducibility of the results.

Concern 3: Applying PES in Off2On

We accept the suggestion and replace BR with PES in Off2On. We conduct experiments on 12 D4RL MuJoCo datasets and present the results evaluated by 5 random seeds below. We can see that PES is comparable with BR when both applying Q ensemble, achieving higher score in 6 out of 12 datasets respectively. However, we argue that the advantage of PES over BR is the better performance without Q ensemble which increases the complexity and computational burden.

Concern 4: Applying PES to ReBRAC and FamO2O

According to the suggestion, we supplement ReBRAC and FamO2O which are two recent SOTA algorithms as additional baselines. We apply PES to ReBARC and FamO2O and conduct experiments on 12 D4RL MuJoCo datasets. We present the results below. The results indicate that ReBRAC+PES surpasses ReBRAC on 8 out of 12 datasets and FamO2O+PES outperforms FamO2O on 8 out of 12 datasets. We believe this demonstrates the versatility and superiority of PES.

We thank the reviewer for insightful comments. Please find our clarifications to the concerns below. If we are able to address the concerns, we hope the reviewer can reconsider the score.

Concern 1: Comparison to other O2O methods

We first supplement ReBRAC and FamO2O as additional baselines for comparison with PES. We present the results evaluated by 5 random seeds below. It can be seen that although ReBRAC and FamO2O already have strong performance, PES is able to further enhance the performance of ReBRAC and FamO2O, which illustrates the versatility of PES. Next, we elaborate on the advantages of PES compared to other O2O methods. First, PES has versatility, meaning that PES can be applied to various offline and O2O methods to improve performance, including ReBRAC and FamO2O. Second, due to the improvements PES makes at the data level, it is orthogonal to algorithm design and can enhance performance without changing the original algorithm process. In contrast, other methods like FamO2O introduces additional training steps, increasing the complexity of training.

Concern 1: Comparison with the offline trained policy

Yes, we have provided extensive comparisons with offline trained policies. In Appendix H.2, we present the learning curves of the online stage of PES combined with various algorithms, including CQL, TD3-BC, Cal-QL, PEX, and AWAC. The starting point of the curves (i.e., the point at step=0) represents the performance of the offline-trained policy. It is evident that with the integration of PES, the curves generally show an upward trend, indicating that compared to pure offline training, PES can achieve a general performance improvement.

Concern 3: Ablation on Step 2 of PES

To evaluate the impact of data quality in the priority queue on final performance, we conduct an ablation study. Instead of initializing the priority queue with high-return trajectories, we randomly sample trajectories from the dataset for initialization. We conduct experiments on 12 D4RL MuJoCo datasets and present the comparative results below. It can be seen that the priority queue initialized with random trajectories performs clearly worse than the one initialized with high-return trajectories. This is because random initialization may cause the algorithm to only use low-quality samples for training, leading to reduced performance.

We thank the reviewer for insightful comments. Please find our clarifications to the concerns below. If we are able to address the concerns, we hope the reviewer can reconsider the score.

Concern 1: The differences between PES and Balanced Replay

First, we elaborate on our experimental setup. BR not only uses the balanced replay technique but also employs Pessimistic Q ensemble, which is crucial for the performance of BR. To fairly compare PES with BR, we removed the Q ensemble from BR and only used the balanced replay technique (as mentioned in line 303 of our paper), which is why the performance of BR in our experiments is not as good as reported in the original paper. Secondly, we elaborate on the superiority of PES over BR. BR selects more on-policy samples from the offline dataset, which is an optimistic sample selection perspective that may cause severe distribution shift, hence the need for Q ensemble to improve the robustness of the Q network. On the other hand, PES reduces distribution shift by filtering online samples and avoids over-conservatism by dynamically adjusting the selection threshold, achieving good performance without relying on Q ensemble.

Concern 2: More baselines for Comparison

We accept the suggestion of comparing PES with more baselines. We have added RLPD and FamO2O as additional baselines. We apply PES to RLPD and FamO2O and conducted experiments on 12 D4RL MuJoCo datasets. We report the results evaluated by 5 random seeds below. It can be seen that PES still manages to enhance the performance of RLPD and FamO2O: PES+RLPD outperformed RLPD on 8 out of 12 datasets, and PES+FamO2O outperformed FamO2O on 9 out of 12 datasets. We believe this further demonstrates the versatility and superiority of PES.

Concern 3: Underlying geometric assumption on KNN search

That is an interesting point. Actually, the geometric assumption of the environment can be considered as a Lipschitz assumption, which we have elaborated on in detail in Appendix B. In Assumption B.3, we assume that the Q function has Lipschitz continuity, that is, samples with similar state-action spaces also have similar Q values, from which we derive the conclusion that PES can select the optimal samples. This is actually a widely used assumption in reinforcement learning [1,2], so we believe our method is reasonable. Moreover, in addition to our experimental results in Table 1, many previous works [1,3] have also used KNN to measure similarity and have proven its effectiveness in environments like Antmaze, so we believe that the KNN method is effective in a variety of different geometric environments.

Concern 4: The implication of Theorem 3.1

Good catch! After being reminded, we also notice the flaw in Theorem 3.1. After rectifying, we have re-derived the performance difference bound of the optimal policy for consecutive steps under the true MDP, which is still related to the difference between the two empirical MDPs. Therefore, our theoretical motivation remains applicable, and we should reduce the distribution shift by reducing the difference between the empirical MDPs. Please refer to our revision for the specific corrections (including Theorem 3.1 in the main text and Appendix A).

Thank the authors for the detailed responses and the additional experiments. Upon a careful read through the manuscript and the rebuttal, my concerns about the baselines and the differences between PES and Balanced Replay have been addressed.

That being said, I still have some concerns about the new Theorem 3.1: The upper bound of performance discrepancy now depends on three factors: (1) $D_{TV}$ between $\hat{p}\_{M\_t}$ and $\hat{p}\_{M\_{t+1}}$; (2) $D_{TV}$ between $p_{M}$ and $\hat{p}\_{M\_t}$; (3) $D_{TV}$ between $p_{M}$ and $\hat{p}\_{M\_{t+1}}$.

In the updated manuscript, it is mentioned that Theorem 3.1 suggests that if the data distribution in the replay buffer between two consecutive steps can evolve smoothly, such that the difference between the two estimated MDPs is small, then the learned policy can avoid the abrupt performance drop, facilitating a smooth policy transfer.

I find the above statement rather inaccurate as the upper bound also depends on (2) and (3), which can be large as the true $p_{M}$ is typically fairly different from the empirical MDPs due to limited data coverage. Hence, the performance discrepancy is not necessarily small if only the $D_{TV}$ between $\hat{p}\_{M\_t}$ and $\hat{p}\_{M\_{t+1}}$ is small. Please correct me if I missed anything.

Also, I will be willing to reconsider my rating based on the authors’ response to this issue.