[Q3] Origin of PARS’s innovation

Thank you for the important question! PARS achieves good performance by effectively combining the two sources mentioned by the reviewer. Looking into each of them in more detail:

1. Reward scaling and layer normalization (RS-LN)

   As mentioned in our response to [W1], the analysis experiments on the effects of reward re-scaling and layer normalization can be found in Figures 7 and 18 of the manuscript. The results indicate that when these two components are combined, they have a significantly positive impact on performance. Notably, the combination of reward scaling with LN demonstrates significantly enhanced performance compared to the LN used in RLPD. Additionally, incorporating PA into this setup further amplifies the performance gains.

2. The combined use of TD3+BC

   Moreover, according to [Ref.13], DDPG+BC is more effective than AWR [Ref.9] in achieving optimal performance offline. Citing the explanation from [Ref.13]: “AWR only has a mode-covering weighted behavioral cloning term, while DDPG+BC has both mode-seeking first-order value maximization and mode-covering behavioral cloning terms. As a result, actions learned by AWR always lie within the convex hull of dataset actions, whereas DDPG+BC can ‘hillclimb’ the learned value function, even allowing extrapolation to some degree while not deviating too far from the mode.”

   As a variant within the same framework as DDPG+BC, TD3+BC similarly enjoys comparable advantages. In our [Q4] response, we present a more detailed analysis of the outcomes when the PARS method is applied to IQL, which incorporates AWR. The findings reveal that while IQL+PARS demonstrates noticeable improvements, TD3+BC, which integrates both “mode-seeking” and “mode-covering”, proves to be more effective in achieving superior performance.

[Q4] Regarding extensibility of PARS

We previously provided experiments in Appendix F demonstrating the application of the PARS idea to SAC-RND, which showed improved performance compared to the original SAC-RND.

Furthermore, we extended the PARS idea to in-sample learning-based methods such as IQL. As shown in Appendix F.2, since IQL does not perform OOD sampling, PA does not provide benefits for this method. Conversely, RS-LN demonstrated noticeable effectiveness, highlighting its potential for broader applicability and impact when integrated into in-sample learning-based approaches. However, it did not surpass the performance of TD3+BC-based PARS, as discussed in our response to [Q3] and detailed in Appendix F.2.

[Ref.9] Peng, Xue Bin, et al. "Advantage-weighted regression: Simple and scalable off-policy reinforcement learning." arXiv 2019.

[Ref.13] Seohong Park, Kevin Frans, Sergey Levine, Aviral Kumar. Is Value Learning Really the Main Bottleneck in Offline RL?

We are thankful for the encouraging feedback on PARS's effectiveness and for the chance to further highlight its strengths in comparison to previous offline-to-online works, as well as to better articulate the underlying factors contributing to its effectiveness.

[W1] Relationship between the performance improvements and the algorithm's design.

We would appreciate it if the reviewer could check our analysis presented in Figure 7, Figure 8, and Appendix H (Figure 18). We have already conducted an ablation study on various components of PARS. Figures 7 and 18 examine how LN, $c_\text{reward}$, and PA impact performance during the offline phase, while Figure 8 further analyzes the effects of PA during the online fine-tuning phase. The analysis shows that without LN, performance decreases as $c_\text{reward}$ increases, but with LN, performance improves as $c_\text{reward}$ grows. Additionally, when PA is incorporated, the performance becomes even stronger and more robust.

[W2 & Q1] Comparison with RLPD

We want to clarify that RLPD, reference [1] mentioned by the reviewer, is not an offline-to-online algorithm because it does not include an offline pretraining phase. Instead, it leverages offline data to make online learning more sample-efficient. While it shares a similar goal with offline-to-online algorithms—achieving high online performance with minimal samples—its approach is different. Offline-to-online algorithms aim to adapt well to distribution shifts and effectively utilize offline knowledge. In contrast, RLPD learns from scratch without explicit offline pretraining while utilizing offline data to achieve sample-efficient learning.

Therefore, RLPD does not explicitly consider the offline phase, and as shown in Table R5, RLPD alone does not perform well offline. Since our goal is to achieve robust performance both offline and during the online fine-tuning phase, we applied the method of using layer normalization from RLPD but improved it by incorporating reward scaling and infeasible action penalties in addition to layer normalization. This improvement effectively enhances OOD mitigation, which was insufficient with layer normalization alone, as observed in the 'LN' column with $c_\text{reward}$ 1 in Figure 3. Consequently, as demonstrated in Table 2 and Figure 10 of our manuscript, PARS outperforms RLPD in more challenging environments like Antmaze Ultra and Adroit because it effectively utilizes the information learned during the offline phase in the online phase.

[Table R5] Comparison of the offline learning results using the RLPD algorithm using five random seeds with the PARS offline results.

[Q1 & Q2] Comparison with other offline-to-online algorithms

In the revised paper's Appendix B, Figure 13, we added a comparison experiment to highlight the benefits of PARS over prior offline-to-online algorithms, including Cal-QL. Cal-QL builds on CQL by addressing its limitation of being overly conservative when predicting Q-values for OOD actions, which hinders online fine-tuning. It achieves this improvement by learning calibrated conservative value functions. While Cal-QL makes notable advancements over CQL through allowing the predicted Q-value to decrease only when it exceeds $V^{\mu}(s)$, challenges remain. Specifically, if the Q-value within the data distribution surpasses $V^{\mu}(s)$, particularly near the boundaries where data coverage ends, minimizing the Q-value at these edges may still result in significant underestimation.

On the other hand, PARS, which naturally reduces the Q values of OOD actions through RS-LN and PA, benefits during online fine-tuning by promoting exploration within the potential region and effectively adapting to new samples. While the reviewer mentioned that LN could be conservative, LN distinguishes itself from conservative loss function-based methods by allowing Q values to naturally decrease, offering a unique advantage.

In addition to the offline-to-online algorithms discussed in Appendix B, PEX integrates offline-trained policies into online learning through a policy expansion strategy that preserves offline behaviors while enabling further learning. This strategy freezes the offline policy to prevent degradation and introduces a new learnable policy, allowing both to collaborate adaptively. PARS focuses on critic regularization, whereas PEX emphasizes policy and can be considered an orthogonal line of work. The performance comparison in Table 2 shows that PARS significantly outperforms PEX.

Thank you for the additional comments, and we're glad to hear that our response regarding RLPD was helpful. We've included answers to the reviewer's additional questions in our revised paper as follows:

• Regarding Q1 & Q2. → Appendix B.2

  We previously compared offline-to-online algorithms in Section 5, Related Works, and added a more detailed explanation of two works, RLPD and Uni-O4, which we believe are particularly related to PARS, in Appendix B.2. If there are any works that the reviewer think should be compared but we haven't, please let us know!

• Regarding Q3. → Appendix K

  We additionally analyzed in Appendix K why RS-LN and PA are more effective when combined with TD3+BC. The policy extraction methods differ between TD3+BC and IQL: TD3+BC employs a behavior-constrained policy gradient, while IQL relies on weighted behavioral cloning. Specifically, TD3+BC employs both mode-seeking first-order critic maximization and mode-covering behavioral cloning. Consequently, as illustrated in Figure 24 of our revised paper, TD3+BC enables the generation of actions that extend beyond the boundaries of $\mathcal{A}_\mathcal{D}$ to some extent. This approach can achieve greater synergy when the critic's predictions are both accurate and well-generalized. The superior performance of PARS within TD3+BC can largely be attributed to the role of RS-LN and PA in fostering a critic with high accuracy and improved generalization capabilities. For more details, please refer to Appendix K.

  • We added this at the end of the appendix to avoid disrupting section numbering, which could confuse reviewers when referencing section numbers to read our previous responses. As the reviewer noted, these explanations enhance the paper's comprehensiveness, so we're considering moving them to the front of the appendix or main text.

• Regarding Q4. → Appendix D.2

  We further analyzed how RS-LN can also be effective when combined with in-sample learning-based methods by comparing the representative IQL critic learning objective with the TD3+BC critic learning objective. For more details, please refer to Appendix D.2.

Let us know freely if you have any additional questions or if your concern has been addressed.

Thanks for answering my question. Regarding Q1 and Q2, I have another related question: could you please investigate any potential connections between RLPD and ENOTO[1]? ENOTO adapts Q ensemble. Q ensemble originates from SAC-N [2], and through ensemble Q, it can indirectly penalize samples that appear relatively infrequently, thereby mitigating the Out-Of-Distribution (OOD) problem. The approach adopted by ENOTO is to normally use Q ensemble during offline pre-training and gradually relax the ensemble during online fine-tuning. Thank you.

Reference

[1] Kai Zhao, Jianye Hao, Yi Ma, Jinyi Liu, Yan Zheng, Zhaopeng Meng, ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles

[2] Gaon An, Seungyong Moon, Jang-Hyun Kim, Hyun Oh Song, Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble

Thanks for your response, which has addressed my concerns. Additionally, I believe that the experiments and evaluations conducted on RLPD are quite thorough. Therefore, I am prepared to increase the score to 6. RLPD is also a comprehensive paper, and as such, the author has successfully balanced the length and expression in their writing to some extent. Meanwhile, I need to lower my confidence from 3 to 2.

We appreciate the comments aimed at clarifying the contributions and unclear explanations of PARS, as well as suggesting ways to present more robust experimental results. Incorporating this feedback is expected to further improve the quality of the paper.

[W1] Regarding the motivation of PARS

Thank you for giving us the opportunity to further strengthen our motivation by comparing it with prior works. In our common response [C1], we have clarified our motivation and contributions more explicitly. As pointed out by the reviewer, in Figure 2, while both Penalizing OOD actions within the feasible region (c) and RS-LN and PA (d) aim to achieve the same goal of finding the max Q policy offline, (d) demonstrates a smoother decrease in the Q-function in OOD regions. This characteristic is particularly advantageous during online fine-tuning. As we show through various figures and additional experiments in Appendix B of our revised paper, if the OOD Q-values are learned too conservatively or are fitted to small values offline, it could lead to adverse effects during online fine-tuning. A more detailed response can be found in Appendix B of the revised paper.

[W2] Regarding the ensemble

We regret any confusion that may have arisen from our prior wording, as we did not intend to criticize the use of critic ensembles per se. Recent approaches such as MSG [Ref.3] with 64 critics and RLPD [Ref.4] with 10 critics illustrate that critic ensembles are both straightforward and effective. Our point, however, is that relying solely on critic ensembles for regularization, as seen in methods like SAC-N [Ref.5], may demand as many as 500 critics for certain datasets, leading to an excessive complexity overhead. Moreover, even with this added complexity, SAC-N is unable to solve the AntMaze task (as also mentioned in CORL [Ref.6]).

As shown in Table R3, for AntMaze, PARS performs well without critic ensembles, although it shows a slight decrease in score compared to PARS with four critics.

[Table R3] PARS AntMaze performance without critic ensembles.

In tasks like MuJoCo and Adroit, ensemble integration proves beneficial, but a simple method significantly reduces the number of ensembles required compared to using only the ensemble approach. We envision that future research will focus on advancing network expressivity and generalization ability, leading to powerful RL algorithms that operate with minimal complexity.

[Q1] Didactic examples with more seeds and increased reward scale

We provide results for additional seeds and for setting $c_\text{reward}$ to values greater than 100, such as 10,000, in Appendix J of the revised paper. First, varying the seeds showed slight differences, but the overall trend remained consistent. Additionally, when $c_\text{reward}$ was increased, applying only LN caused the shape to change slightly depending on $c_\text{reward}$, but no clear pattern emerged. However, when both LN and PA were applied, similar patterns persisted even as $c_\text{reward}$ increased further.

[Q2] Regarding the interpretation of Srank

The effective rank of the representation, Srank, was proposed in [ref.7], which stated that a reduction in effective rank results in implicit under-parameterization and can lead to performance degradation. However, in “Small Batch Deep Reinforcement Learning (NeurIPS 2023)” [ref.8], based on previous findings that deep RL networks tend to overfit during training, it was observed that the network adapts better to an earlier rank collapse than to a later one. Specifically, when RL is trained with smaller batch sizes, the Srank collapse occurs earlier in training and remains lower compared to larger batch sizes. During this earlier collapse, performance improves more effectively.

We referred to the analysis results of these two studies. As shown in Figure 4, when LN is applied, an increase in the reward scale initially causes the effective rank to decrease, but it then converges. In contrast, when LN is not applied, the effective rank continues to decrease during training as the reward scale increases. Consistent with the analysis results on small batch sizes, applying LN with an increasing reward scale also leads to more effective performance improvements. This suggests that the reduction in effective rank in this context actually prevents overfitting and promotes more effective performance enhancement.

We appreciate the positive comments on the simple and intuitive strengths of PARS, as well as the feedback to clarify the underlying causes and unclear points. Responding to these comments allowed us to improve the clarity and explanatory depth of our work.

[W1] Why the RS-LN can mitigate OOD overestimation.

Thank you for the important question. In Section 4.1, we illustrate through a didactic example that when reward scaling is combined with layer normalization, OOD Q-values progressively decrease further. Furthermore, the analysis in Section 4.2.1 demonstrates that this phenomenon is related to network expressivity, analyzed through factors such as dormant neurons. As highlighted in Appendix C, the ReLU activation function also plays a role.

In the revised paper, Appendix D further addresses the potential reasons why RS-LN can mitigate OOD overestimation. LN not only normalizes inputs to have a mean of 0 and a variance of 1 but also adjusts the scale through learnable parameters. This scaling is trained to progressively increase from an initial value of 1 in response to reward scaling. During the joint optimization of the scaling parameter and other parameters in the Q-network, we observed a phenomenon where the feature norm of ID samples becomes more distinguishable from that of OOD samples, as shown in Figure 17. Specifically, the feature norm of ID samples is larger than that of OOD samples. While this observation requires further analysis, it could serve as a crucial topic for future work.

We have identified important phenomena related to RS-LN and demonstrated its empirical effectiveness through analyses using various metrics. These findings could inspire important future research directions, and we look forward to seeing further advancements in this area.

[W2] Could PA also show benefits over methods using behavior models?

• The first advantage of PA, compared to methods that rely on behavior models, is that it eliminates the need to train an auxiliary model. Additionally, because a behavior model is a predictive model, its accuracy inevitably decreases as the data distribution becomes more complex. In such cases, the Q-values for ID actions may also be incorrectly decreased. Moreover, as noted by AWR [Ref.10] and SPOT [Ref.11], continuously tuning a behavior model in an online environment where the data distribution is constantly changing is impractical.
• Moreover, compared to the approach of using a behavior model to distinguish between ID/OOD and lowering the OOD to a specific value, PA demonstrates significant advantages in online fine-tuning, particularly in terms of online exploration and adapting well to new samples, as we have shown through various figures and additional experiments in Appendix B of the revised paper.

[W3] Regarding writing

Thank you for providing comments on the writing. We have incorporated them into the revised paper.

[Q1] Regarding the sign of the rewards

In the domains we benchmarked, MuJoCo and Adroit have mixed ± rewards, while Antmaze follows a sparse reward setting where a positive reward of +1 is given upon reaching the goal. For example, the reward statistics in the MuJoCo and Adroit datasets are as follows:

[Table R4] Reward statistics of MuJoCo and Adroit datasets.

Therefore, PARS performs well in scenarios where rewards are either all positive or a mix of positive and negative values. However, as the reviewer pointed out, if all rewards are negative, our method cannot demonstrate its benefits. In such cases, the rewards in the offline data can be converted to positive values before applying our method.

Thank you once again for the reviewer's efforts in providing thoughtful comments on our work. In our new common response, we have summarized the paper revisions made during the rebuttal period. We would be delighted if our revised paper helps provide a clearer understanding of our work.

In particular, we discussed [W1] why RS-LN can mitigate OOD overestimation in Appendix D, and delved deeper into the content of [W2] in Appendix B. Other responses can be found in our previous responses. Please feel free to reach out if you have any additional questions.

We appreciate the reviewer for their positive comments on the strengths of PARS, such as its simplicity and effectiveness, as well as for highlighting further promising discussion points about PARS.

[W1] Regarding hyperparameters

Thank you for pointing out an important point. In RL, unlike in NLP or vision tasks, there is no clear validation set or environment. Therefore, offline evaluation metrics that can be utilized without online evaluation would be highly advantageous. However, as the reviewer mentioned, many prior works ([Ref.1,2]) allowed a certain degree of hyperparameter tuning, and we followed these works for tuning. Nevertheless, we agree with the reviewer’s point and further verified that comparable performance in the AntMaze domain can be achieved with just a single hyperparameter setting. This is particularly significant because many offline algorithms have struggled in the AntMaze domain, especially in the Ultra environment. Achieving significantly better performance than the baseline with a single hyperparameter setting in the AntMaze domain—even though it is slightly lower than the PARS score presented in the paper—underscores a key advantage of PARS. The hyperparameters we used for the single-hyperparameter experiment are: reward scale of 10,000, alpha of 0.001, and beta of 0.01.

[Table R1] PARS AntMaze performance using a single hyperparameter setting.

[W2] Comparison with in-sample learning-based methods

Thank you for suggesting an insightful discussion point. Based on your suggestion, we analyzed how the ideas of PARS could be applied to a representative in-sample learning-based method, IQL, and included the analysis in Appendix F.2 of the revised paper. As shown in Appendix F.2, since IQL does not perform OOD sampling, PA does not provide benefits for this method. Conversely, RS-LN demonstrated noticeable effectiveness, highlighting its potential for broader applicability and impact when integrated into in-sample learning-based approaches.

[Q1] Regarding feasible bound

In this work, all the domains we benchmarked have a feasible action range of -1 to 1. Consequently, we conducted experiments with $L_i = -1$ and $U_i = 1$. We believe this does not introduce additional assumptions compared to other algorithms, as they also leverage the knowledge of the feasible action range being -1 to 1 and typically use a tanh activation function in the actor network as a standard practice.

However, in practical scenarios, as pointed by the reviewer, assuming an unknown feasible action range could further improve the algorithm's scalability. To explore this, as suggested by the reviewer, we conducted additional experiments by extracting the minimum and maximum bounds for each action dimension from the dataset. As shown in Table R2, training with the feasible bounds calculated from the dataset resulted in no difference in performance. This is likely because we impose the infeasible penalty not near the feasible bounds but at a point significantly farther away, making it less critical to know the exact values and feasible to use the calculated ones.

[Table R2] PARS with a calculated feasible bound.

[Ref.1] Tarasov, Denis, et al. "Revisiting the minimalist approach to offline reinforcement learning." NeurIPS 2023.

[Ref.2] Nikulin, Alexander, et al. "Anti-exploration by random network distillation." ICML 2023.

Thank you once again to the reviewer for your thoughtful comments on our work. In our new common response, we have summarized the paper revisions made during the rebuttal period. We would be delighted if our revised paper provides a clearer understanding of our work.

In particular, we have added content on [W2] comparison with in-sample learning-based methods in Appendix F.2, explained why RS-LN can also be effective in in-sample learning-based methods in Appendix D, and discussed its particular effectiveness in TD3+BC in Appendix K. Other responses can be found in our previous responses. Please feel free to reach out if you have any additional questions.