Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data

Thank you for your constructive and encouraging feedback! In response to your comments, we 1) answer below your question on why online RL algorithms still suffers from Q-value underestimation issues, 2) added Figures showing WSRL with different levels of expertise in the offline policy, 3) point you to our baseline comparison on RLPD-initialized with prior policy, and 4) anwer your question of why JSRL does not retain offline data. In addition, we also added extensive experiments on nine Gym locomotion tasks and eight antmaze environments, as well as many other ablations. Please let us know if you have any other questions.

It is hypothesized in section 4.1 that backups of small, over-pessimistic Q-values from pessimistic offline algorithms cause the Q-values divergence. However, in Figure 9 you mention the warm-up helps avoid over-pessimistic values for WSRL. I had understood that WSRL used an online, non-pessimistic algorithm (RLPD) as its base, so what would cause the underestimation in this case?

Good question! As you mentioned, Section 4.1 explains how over-pessimistic Bellman backup leads to Q-values divergence. You’re right that WSRL uses an online non-pessimistic algorithm (RLPD), but it initializes its Q function from the pessimistic offline pre-training, e.g., with Cal-QL or CQL. By the nature of pessimistic pre-training, OOD state-action values may be especially smaller in value compared to the state-action pairs seen in the offline dataset. During fine-tuning, if there were no warm-up, the algorithm will collect OOD data into the buffer, and so Bellman backups with pessimistic target Q values will lead to Q-divergence. However, warmup solves this problem by putting more offline-like data into the replay buffer where Q-values are not as pessimistic, thereby preventing the pathologies in Section 4.1.

It would have been interesting to see WSRL results when initialized from pre-trained policies of different quality levels or trained for varying numbers of steps.

Following your suggestion, we added Appendix J to show WSRL on varying levels of expertise on the offline policy. Specifically, Figure 21 shows WSRL performance on kitchen-complete-v0 and relocate-binary-v0, two especially hard tasks for offline RL where CalQL has pre-trained performance 0. Not surprisingly, WSRL is also not able to learn: when pre-training fails, indicating that offline training is not able to learn useful behavior, initializing with the pre-trained networks may not bring any information gain from pre-training, and may actually hurt fine-tuning by reducing the network’s plasticity [1].

A comparison of RLPD initialized with prior policy, but without the additional online data should have been included as a baseline.

This is actually included in Figure 11 as an ablation; When WSRL is not initialized with the Q-function, it is equivalent to RLPD with no offline data retention (Note that RLPD is initialized with random actions, but policy-initialized-RLPD is equivalent to doing warmup). Results show that WSRL outperforms this baseline. Specifically, we showed that WSRL is much better than policy-initialized-RLPD on Antmaze environments, moderately better on Adroit, and similar on Kitchen.

In Figure 7, JSRL is shown as not retaining offline data. However, JSRL is implemented over IQL in Uchendu et al., and the IQL implementation linked in the paper does retain offline data. Did the authors specify to you that they chose not to retain offline data?

You are correct, IQL usually retains the offline data during offline-to-online fine-tuning. However, In the Uchendu el al.[2], Algorithm 1 (page 4) bullet point #2 says that buffer D is initialized to the empty set, which indicates that they do not retain offline data. Therefore, we implemented JSRL with no data retention.

RE: Added Experiments and Ablations

In addition, We have also added new experiments showing WSRL’s performance on 1) nine mujoco locomotion environments as well as 2) eight antmaze environments in Appendix A and B. The experiments all follow the same trend as the ones in the main paper, and support our main claims.

We also added new ablation experiments in Appendix D, E, I, J, further analyzing the design decisions we made in WSRL.

Minor Line 84: an warmup

Line 207: on which we measure metrics on

Line 253: onto the previous action pair that the TD error (?)

Figure 4: Mixed, complete, partial (what does this refer to?)

Thanks for pointing them out, we have fixed them. In Figure 4, the “Mixed”, “Complete”, and “partial” refers to three different kinds of kitchen environments.

[1] Nikishin, Evgenii, et al. "The primacy bias in deep reinforcement learning." International conference on machine learning. PMLR, 2022.

[2] Uchendu, Ikechukwu, et al. "Jump-start reinforcement learning." International Conference on Machine Learning. PMLR, 2023. https://arxiv.org/abs/2204.02372

I am not sure how the warm-up phase solves the miscalibration problem present in CalQL. For example when you initialize the policy and Q with the CQL, although the warmup data will help to combat the distribution mismatch, the change of the RL algorithm will still make the Q function miscalibrated which in turn make the policy unlearn. How do you think of this issue?

Good question! As you mentioned, the change in RL algorithm from offline to online could lead to miscalibration issues, where the pre-trained values are more pessimistic than ground truth values. This miscalibration issue is amplified by the no-retention setting, as opposed to the problem setting in the CalQL paper where offline data is retained during fine-tuning. The reason warm-up mitigates this issue to a large extent is because, as you mention, warmup data helps combat distribution mismatch. We provide a detailed analysis of how this affects the Bellman update in Appendix F, and instead give a high-level intuition here. With a large number of training steps on this warmup data (i.e., with a high UTD) that is offline-like, we can obtain a better Q-function that retains pre-trained knowledge and successfully fine-tune. Note that while the no retention fine-tuning problem setting does not retain offline data, it allows as many updates needed on warmup data before more online data is collected and in the process, it can address this miscalibration issue by simply training on warmup data. Please let us know if this clarifies this question, we are happy to clarify further!

Won't it be beneficial to just run a bunch of gradient steps only on the Q function after you collect the warmup data until it recalibrates?

Following your suggestion, we added a new ablation in Appendix E, showing the effect of freezing the policy and only updating the Q-function for some period of time after the warmup phase. As Figure 18 shows, this generally does not improve performance across six environments and two different freezing-periods, and does not resolve the initial dip in policy performance (again, more detailed discussion on why this is unavoidable in Appendix D).

RE: Added new experiments on diverse domains

Additionally, We have also added new experiments showing WSRL’s performance on 1) nine mujoco locomotion environments as well as 2) eight antmaze environments in Appendix A and B. The experiments all follow the same trend as the ones in the main paper, and support our main claims.

Could you clarify Figure 3? The caption says "Fine-tuning starts at step 0.", but the plots are start from around 250k.

Our apologies for the typo. The agents were pre-trained for 250k steps, and fine-tuning starts at step 250k.

Thank you for your feedback! We have addressed your comments by 1) adding zoomed-in plots for the start of fine-tuning, 2) adding analysis on why WSRL does not suffer from catastrophic forgetting of its offline pre-training, 3) answering your question on the warm-up phase solves the mis-calibration problem, and 4) adding ablations on only updating the Q function at the start of fine-tuning. We also added extensive experiments on nine Gym locomotion tasks and eight antmaze environments, as well as many ablations. Please let us know if any further concerns remain, or if these changes address the issues.

Some of the plots are truncated, i.e. not starting from 0. Could you please clarify on that? I would even suggest the authors to have a zoom-in version of that parts [the beginning of fine-tuning phase] of the plot.

Thanks for the suggestion, we have added Figure 16 in Appendix D, showing the zoomed-in version of the first 50k steps of fine-tuning at small evaluation intervals. Some of the original plots did not start from 0 because the first performance evaluation and logging in our codebase was done $N$ steps into fine-tuning. Figure 16 shows performance starting from step 0. We also updated Figure 6 and 7 to start from step 0. These updates do not affect any of the conclusions in the paper.

From the results in Figure 7, seem like the warmup phase does not really solve the unlearning problem, i.e. the initial value for WSRL is much lower than the CalQL initialization. Is this a desired behaviour?

You are right – Figure 7 shows that for Kitchen environments the initial value of WSRL is worse than CalQL, which in this comparison is provided access to retaining offline data. This is because, at the very early stage of fine-tuning, WSRL’s policy performance does take a little dip (this is better seen in the zoomed-in plots in Figure 16). However, Figure 16 also shows that WSRL is able to quickly recover from such a dip, and learns faster than baseline methods with no-retention. As Figure 7 shows, WSRL eventually learns a much better policy despite its initial dip; although CalQL has better initial performance, it never really improves much in Kitchen envs.

We hypothesize that such a dip might be inevitable at the start of online fine-tuning in the no offline data retention setting because the policy is experiencing different states than what it was trained on and potentially states it has never seen (See Appendix D for an example and more details). Moreover, in some environments (e.g., binary reward environments), one might expect small fluctuations in the policy to manifest as large changes in the actual policy performance. However, such brief performance dip does not mean the policy/Q function has unlearned, which is evidenced by the fact that WSRL recovers faster than its peer algorithms and learns faster than online RL algorithms such as RLPD (See Figure 7). If this initial dip would have destroyed all pre-training knowledge from the policy, then we would not expect quick recovery.

Therefore just deducing whether an algorithm has unlearned or not based on performance may not be the most informative. Instead, a more meaningful metric to measure unlearning is to evaluate how much a fine-tuning algorithm with no data retention deviates from its pre-training, and how fast it can adjust to the online state-action distribution. Therefore, we measure the KL divergence between the fine-tuned policy and the pre-trained policy on both online and offline data distributions, and find that WSRL remains stable during fine-tuning and does not destroy priors learned from offline pre-training. See a more detailed analysis in Appendix D.

We acknowledge the term “unlearning” was not clearly defined in the paper, and claims of WSRL resolves the unlearning issue might have caused confusion. We have updated the paper to remove ambiguous claims of “unlearning”, and state that even though WSRL experiences an initial dip in policy performance, it quickly recovers and does not destroy priors learned from offline pre-training. We would appreciate any feedback you might have in making the terminology more clear in the paper.

Thanks for your replies to my questions. Many concerns has been resolved by the rebuttal, except:

• I can agree with you from Appendix D that somehow WSRL managed to forget fast and adapt fast and these add up together to make the final performance of WSRL better. But it is still unclear what does WSRL forget during the shift. And why is it only harmful for the short term (the performance dip) but not and even beneficial for the long term (the asymptotic performance). Like you said, the term "unlearning" needs to be more properly defined. I will encourage the authors to work more on this direction.
• I think you tried to use Appendix I to answer my question about the buffer initialization instead of doing warmup. But that is not what I mean. What I mean is, as WSRL is doing, by default, 5000 steps for warmup with data collected by the pretrained policy, what if we instead randomly sample 5000 transitions from the pretrain offline dataset and use that for warmup. In this way, the method hopefully doesn't slowed down as you shown in Figure 20 with the entire offline dataset.

Despite of these, I do think the new contents strengthen the paper. I will increase my score to reflect that.

Thanks for your fast response! We are glad to hear that most of the concerns have been resolved. We wanted to provide some clarity on your last two questions. Please let us know if anything still remains unclear, we are very happy to discuss further.

what if we instead randomly sample 5000 transitions from the pretrain offline dataset and use that for warmup

Thanks for the suggestion! We added a new experiment in Figure 22 for this ablation, along with analysis in Appendix K, highlighted in purple. The results show that using 5000 transitions from the offline dataset for warmup generally performs worse than WSRL. Note that this experiment highlights the utility of our approach: despite not having access to any offline data, WSRL is able to achieve similar or better performance than using transitions from the offline dataset in the no retention fine-tuning setting.

To dive into the offline dataset result in more detail, while the two methods are similar on Adroit environments, WSRL is slightly better in Kitchen and much better on Antmaze. This is perhaps because the 5000 randomly sampled transitions might not be relevant to the online fine-tuning task, especially in Antmaze where the dataset has diverse state-action coverage (as explained below and in Appendix D). When the replay buffer is initialized with less relevant data, it is less effective at preventing Q-value divergence (Section 4) and recalibrating the online Q-function and policy.

But it is still unclear what does WSRL forget during the shift. And why is it only harmful for the short term (the performance dip) but not and even beneficial for the long term (the asymptotic performance).

That’s a great question! We added more analysis to Appendix D (highlighted in purple) to further aim to understand this, though of course just studying unlearning in all can be a full paper in itself and we do not mean to imply that we provide an extensive analysis studying this.

In short, our experiments highlight that due to the distribution shift from offline pre-training to no-retention fine-tuning, WSRL is forgetting experience in the offline dataset that was learned during offline pre-training but is in reality irrelevant for specializing to the online task. In contrast, it is instead specializing to the online task. These intuitions of “forgetting” and “specializing” can be seen in the two KL divergence plots that we have added in Figure 17: concretely, these plots show the KL divergence between the offline policy and the fine-tuned policy on offline and online state distributions. The KL divergence increases over the offline state distribution, which suggests that the policy learns to forget at least some behaviors appearing in the offline dataset that it had learned during offline pre-training; but the KL divergence remains small on the online state distribution, suggesting that the policy did not forget the behavior it had learned from the offline data relevant to the online distribution. We do note the caveat that KL divergence may not be entirely indicative of forgetting and plan to continue to investigate the broader question in our future research.

That said, to give some specific examples, consider the Antmaze environments and the Adroit environments. The Antmaze offline dataset exhibits a very diverse state-action distribution, covering almost all the locations in the entire maze. However, the online task is single-goal navigation. Therefore, in no-retention fine-tuning, the agent is incentivized to forget irrelevant states (because it is no longer training on those experiences) and instead only specialize in the fine-tuning task. Therefore, the KL divergence on the offline state distribution increases. However, the KL divergence on the online distribution remains small because the pre-trained policy is somewhat capable on such states and the policy only needs to adjust to some unseen states. On the other hand, the Adroit datasets consist of expert human demonstrations and rollouts of a good behavior cloning policy. One can imagine that these experiences are all relevant to solving the same task as what we wish to solve during fine-tuning. As expected, we see that the KL divergence on offline distribution on Adroit environments barely increases (as opposed to Kitchen and Antmaze), suggesting that the policy did not forget useful priors in pre-training.

[1] Zhang, Yinmin, et al. "A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 15. 2024.

[2] Nakamoto, Mitsuhiko, et al. "Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning." Advances in Neural Information Processing Systems 36 (2024).

[3] Li, Jianxiong, et al. "Proto: Iterative policy regularized offline-to-online reinforcement learning." arXiv preprint arXiv:2305.15669 (2023).

[4] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. "Offline reinforcement learning with implicit q-learning." arXiv preprint arXiv:2110.06169 (2021).

To demonstrate the absence of unlearning, it would help to show early performance at smaller timestep intervals

This is a great suggestion! To address this question, we have now added a new result in Figure 16 in Appendix, showing the fine-tuning performance of WSRL and baseline algorithms during the first 50k steps with very small evaluation intervals.

Figure 16 shows that WSRL experiences an initial dip in policy performance after 5, 000 steps of warmup, but recovers much faster than other baselines. We hypothesize that such a dip might be inevitable at the start of online fine-tuning in the no offline data retention setting because the policy is experiencing different states than what it was trained on and potentially states it has never seen (See Appendix D for an example and more details). Moreover, in some environments (e.g., binary reward environments), one might expect small fluctuations in the policy to manifest as large changes in the actual policy performance. However, such brief performance dip does not mean the policy/Q function has unlearned, which is evidenced by the fact that WSRL recovers faster than its peer algorithms and learns faster than online RL algorithms such as RLPD (See Figure 7). If this initial dip would have destroyed all pre-training knowledge from the policy, then we would not expect quick recovery.

Therefore just deducing whether an algorithm has unlearned or not based on performance may not be the most informative. Instead, a more meaningful metric to measure unlearning is to evaluate how much a fine-tuning algorithm with no data retention deviates from its pre-training, and how fast it can adjust to the online state-action distribution. Therefore, we measure the KL divergence between the fine-tuned policy and the pre-trained policy on both online and offline data distributions, and find that WSRL remains stable during fine-tuning and does not destroy priors learned from offline pre-training. See a more detailed analysis in Appendix D.

We acknowledge the term “unlearning” was not clearly defined in the paper, and claims of WSRL resolves the unlearning issue might have caused confusion. We have updated the paper to remove ambiguous claims of “unlearning”, and state that even though WSRL experiences an initial dip in policy performance, it quickly recovers and does not destroy priors learned from offline pre-training. We would appreciate any feedback you might have in making the terminology more clear in the paper.

Could you provide experimental results using other offline algorithms (e.g., IQL, CQL) with the proposed methods applied?

In the submission PDF, we show in Figure 19 in Appendix G that WSRL performs similarly across three environments regardless of whether we use IQL, CQL, CalQL for the offline pre-training. We are working on obtaining similar results in more environments.

The paper employs strong language, such as “Should Not” and “Completely unnecessary,” though the evidence provided may not be fully convincing.

Thank you for the suggestion! Following your suggestion, we have removed strong language from the paper, such as “Should Not” and “completely unnecessary”. For example, we propose to change the title to “Efficient Online Reinforcement Learning Fine-Tuning *Need Not Retain Offline Data”, instead of the original title. We have also added many experiments in additional environments (Figure 12, 13) to support our claims. All changes in the paper are highlighted in blue. Please let us know if there are other instances of strong claims that you would want us to change, and we are happy to make changes!

This paper excludes offline data during online fine-tuning, implements a warmup phase, and increases the UTD ratio. What would happen if we retained the offline data while implementing the warmup phase and increasing the UTD ratio?

We added an additional ablation in Figure 20 in Appendix I to compare performance of WSRL vs. WSRL with the replay buffer initialized with offline data. We find that the performance is mostly similar, though initializing with offline data sometimes provides stronger performance. This is not surprising and perhaps expected because in certain environments, retaining the offline dataset could give the fine-tuning more information, possibly information that was not well-learned during pre-training. That said, it does not conflict with our goal in this paper which is to build a method without data retention that can outperform existing fine-tuning approaches which seem to entirely fail when offline data is not retained (as we show in Figure 6), and WSRL does indeed satisfy this desideratum, even if WSRL + offline data performs similarly to WSRL.

Thank you for your feedback! We have addressed your comments by 1) added experiments on all eight antmaze environments and nine Mujoco locomotion environments, 2) including baseline comparison against SO2 agent from Zhang et al., which we find still underperforms our method WSRL, 3) added zoomed-in plots for start of fine-tuning, 4) clarified on offline pre-training performance and added analysis on whether WSRL “unlearns”, and 5) added many ablation experiments such as IQL/CQL pre-training and WSRL with data retention. Please let us know if any further concerns remain, or if these changes address the issues.

Could you provide experimental results from environments like the MuJoCo dataset? Could you provide experimental results for all 8 Antmaze datasets?

We have added results for WSRL (along with baselines) on nine Mujoco locomotion environments in Appendix B, as well as results on all eight Antmaze environments in Appendix A. The trends observed in these environments are consistent with those presented in the submission: WSRL outperforms previous state-of-the-art algorithms, such as CalQL and RLPD, despite some baselines benefiting from the unfair advantage of retaining offline data (See Figure 12, 13). We believe these additional experiments should address your concern on comprehensive evaluation of WSRL on diverse domains.

Could you provide a comparison with “A Perspective of Q-Value Estimation on Offline-to-Online Reinforcement Learning”?

Thanks for pointing out this work. We have added this work as a baseline in Figure 7. The results show WSRL performing similarly on Adroit environments, but much better in Kitchen and Antmaze environments.

While experiments were conducted on Antmaze ultra-diverse, ultra-play was not included, which may suggest some inconsistency in [dataset] selection.

We picked the three antmaze environments out of the eight because they were three of the hardest antmaze environments to solve, as measured by performance of previous offline RL and online RL fine-tuning algorithms (CalQL [2], Proto [3], IQL [4]). We only presented results on antmaze-ultra-diverse and not antmaze-ultra-play because their dataset collection mechanism is very similar. However, we agree with you that the paper would benefit from more comprehensive results, so we added experiments for all eight antmaze environments in Appendix A (See Figure 12). Across the board, we find that WSRL outperforms other methods that do retain offline data.

The offline performance in the experimental setup is poor from the start (Section 6, Figures 6 and 7)

Sorry for the confusion. The offline performance on the environments in the paper actually is good (See updated Figures 6 and 7, first data point), some reaching 60-80 points, as you mentioned. The confusion stems from the fact that the original Figures 6 and 7 were plotted at large evaluation intervals, and the first evaluation data point did not reflect the offline performance. We have fixed Figures 6 and 7 to show offline performance as the first data point.

Thank you for addressing the concerns with additional experiments, as noted in the previous response. However, there are still unclear aspects, particularly in the writing and presentation of experimental results.

The paper's purpose and contributions, as I understand them, are as follows:

“In offline-to-online RL, retaining offline data during online fine-tuning is impractical. Therefore, the paper proposes a novel method to perform well in online fine-tuning without retaining offline data, supported by various analyses and a new warm-up phase.”

This premise is reasonable and represents a meaningful contribution.

However, there appear to be some areas where the content of the paper and the experimental results are not fully aligned.

 

1. Inconsistent Use of "Unlearning" and "Catastrophically Forgetting"

From my understanding:

• "Unlearning" in this paper refers to the performance drop at the start of online fine-tuning due to distribution shift.
• "Catastrophically forgetting" refers to information loss so severe that recovery becomes impossible even with continued training.

The terms are used in the following contexts in the paper:

<Unlearning>

• Line 20: "We find that continued training on offline data is mostly useful for preventing a sudden unlearning of the offline RL value function at the onset of fine-tuning, caused by a distribution mismatch between the offline data and online rollouts."
• Line 23: "Our approach, WSRL, mitigates this sudden unlearning..."
• Line 73: "This recalibration phase can lead to unlearning of the offline initialization, and even divergence, when no offline data is present for training."
• Line 79: "We show that the main culprit behind the unlearning is the distribution mismatch between the offline data and online training distribution, and retaining offline data attenuates the effect of this mismatch, playing an essential role in the working of current offline-to-online fine-tuning methods."

<Catastrophically Forgetting>

• Line 82: "...catastrophically forgetting offline pre-training."
• Line 85: "'Simulates' offline data retention can greatly facilitate recalibration, preventing catastrophic forgetting that never recovers with more learning."
• Line 325: "How can we tackle both catastrophic forgetting of the offline initialization and attain asymptotic sample efficiency online?"
• Line 330: "The remaining question is: how do we tackle catastrophic forgetting at the onset of fine-tuning that prevents further improvements online, without offline data?"

As explained in the revised Appendix D (the initial performance dip might be inevitable due to the interaction of the learning policy with new states not present in the pretraining data), the paper does not resolve unlearning but rather addresses catastrophic forgetting. This distinction must be clarified explicitly in the paper without any confusion. Claims that the paper resolves unlearning are incorrect and misleading.

 

2. Step 0 Discrepancy in Figures 6 and 7

• In Figures 6 and 7, the values for WSRL at step 0 differ. Is there an explanation for this? The trends appear identical after step 0, making this discrepancy seem unusual.

 

3. Details on MuJoCo Experiments

• Thank you for conducting additional MuJoCo experiments. However, the details of these experiments are not provided. Were they conducted under the same settings as described in Line 1065 (hyperparameter section)?

 

4. Minor

• Y-Axis Format in Figures 3 and 4: The y-axis uses exponential notation (e.g., 10^{−2.1}) in decimal form. Is there a specific reason for this choice?
• Consistency in Terminology: The term "Cal-QL" is written both as "CalQL" and "Cal-QL." Please standardize its usage throughout the paper.
• Dataset Order in Figures: In Figures 6 and 7, the order of datasets differs. Unless there is a specific reason, aligning the order would improve clarity.

 

I believe the contribution is valuable and the experiments are thorough, but it’s important for the claims in the paper to align with the actual findings to maintain clarity and accuracy. It would be helpful if the authors could indicate whether the writing can be improved, even if immediate revisions are not possible due to the limited rebuttal period. Alternatively, if my understanding is mistaken and the current writing already accurately reflects the experimental results, I would appreciate clarification.

Thank you for your time and your review! We answer below your question about WSRL’s compatibility with on-policy RL algorithms. We also added extensive experiments on nine Gym locomotion tasks and eight antmaze environments, as well as many ablations. Are there additional questions we could address to improve the score?

While off-policy algorithms are naturally suited for the online phase, is it possible to apply online policy methods with monotonic improvement properties (e.g., PPO, TRPO) and still achieve comparable performance during online training?

That’s a great question! In this paper, we choose off-policy algorithms for fine-tuning because they are most compatible with the standard offline RL pre-training (e.g. CQL, IQL), where off-policy algorithms are necessary to learn from a pre-collected dataset. With off-policy pre-training, using the same type of learning objective online minimizes catastrophic forgetting. However, we do agree that on-policy fine-tuning from off-policy pre-training could be an interesting and important area for future work.

RE: new experiments and ablations

We have also added new experiments showing WSRL’s performance on 1) nine Gym locomotion environments as well as 2) eight antmaze environments in Appendix A and B. The experiments all follow the same trend as the ones in the main paper, and support and strengthen our main claims.

We also added new ablation experiments and analysis in Appendix D, E, I, J, further analyzing the design decisions we made in WSRL.

Typos: Ofline --> Offline (In the box of Takeway 3, Line 3)

Thanks for pointing it out, we have corrected it.