Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Data

We sincerely appreciate the careful review of our work, the clarification of various components of PARS, and your thoughtful suggestions.

[R1] Ablation study separating the effects of RS-LN and PA

Beyond the ablation in Figure 9, we isolated PA in a separate experiment. As shown in Fig B of the link (https://sites.google.com/view/pars-icml25).

• None (without LN or PA): Training becomes unstable as the reward scale increases.
• PA: Helps mitigate extrapolation error, though its effect diminishes at larger reward scales.
• LN: Performance improves with increasing reward scale.
• LN and PA: Combining LN with PA further enhances performance by reinforcing downward Q extrapolation.

 

[R2] Hyperparameter justification

As the reviewer mentioned, $\alpha$ in Eq. (3), the infeasible action distance, and the reward scale are important factors in PARS. The infeasible action distance is analyzed in Figure 11, reward scale in Figure 9, and $\alpha$ in our response to Reviewer QMiG [R3].

 

[R3] Generalization beyond D4RL

Thank you for the helpful suggestion. Additionally, we evaluated PARS on four environments from the NeoRL-2 [Ref.4] benchmark, which includes more complex and realistic problems, covering a variety of real-world challenges such as delay, external factors, limited data, and safety constraints. As shown in Table D of the link (https://sites.google.com/view/pars-icml25), PARS surpasses previous SOTA even under more realistic benchmark.

 

[R4] Action space dimensionality and PA behavior

As shown in Eq.(1), an action is considered infeasible in dimension $i$ if it falls outside predefined thresholds. Thus, $\mathcal{A}_I$ is defined as the union of these infeasible regions across all dimensions. From this union, we sample the same number of infeasible actions as dataset actions per gradient update and assign them a penalty of $Q\_{\text{min}}$.

 

[R5] Computational efficiency of PARS

In Appendix H, we compared the computational cost of various offline RL algorithms, showing that PARS is more efficient in training time and GPU memory usage.

 

[R6] Comparison with uncertainty-based approach

Thanks for the insightful point. Both distributional and model-based RL incorporate forms of uncertainty—distributional RL models the distribution over Q-values, while model-based RL estimates uncertainty in learned environment dynamics. These uncertainties help with exploration and OOD detection, albeit often at the cost of added complexity and computation. PARS offers a simple and efficient solution built on TD3+BC, without complex uncertainty modeling, yet still achieves strong performance.

We see promise in combining PARS with uncertainty-based methods. In offline RL, penalties may need to differ between interpolation and extrapolation regions, as the latter, lacking clear bounds, can lead to more severe value errors. Penalizing OOD while encouraging downward extrapolation could improve robustness. Similarly, uncertainty-guided exploration during fine-tuning, as reviewers suggested, may boost performance. We'll incorporate these insights in the revision.

 

[R7] Applicability to stochastic environments

Even in stochastic settings, actions are generally confined within a certain min and max range. Thus, the span defined by these bounds can be regarded as the feasible region. Moreover, instead of sampling infeasible actions that lie just outside the feasible action set, we sample from a sufficiently distant region, using a guard interval. This allows PARS to function without precise knowledge of the feasible-infeasible boundary, supporting robust performance under stochastic dynamics.

 

[R8] Extension to policy extrapolation

We're not entirely sure what the reviewer means by policy extrapolation, but we interpret it as the policy incorrectly predicting actions for given states. Since the policy is trained via the Q-function, reducing Q-function errors may help mitigate policy errors. However, as Q-function extrapolation and OOD action selection are distinct, directly applying PARS to improve policy accuracy is not straightforward. If we have misunderstood, we would be grateful if the reviewer could kindly clarify.

 

[R9] Effectiveness of PARS under sparse or varying reward scales

Among our evaluated domains, AntMaze meets the reviewer’s criteria with its sparse reward setting. As shown in Figure 9, performance improves with larger reward scales, highlighting a clear advantage over other baselines. We also analyzed the case where noise is added to the reward; for details, please refer to our response to reviewer snRz [R4].

 

[Ref.4] Gao, Songyi, et al. "NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios." arXiv preprint (2025).

 

Thank you once again for your thoughtful review. We hope that our responses have thoroughly addressed your concerns.

We sincerely appreciate your positive feedback and your constructive suggestions that allowed us to strengthen areas we may have initially missed.

[R1] Empirical evidence on Q-value overestimation reduction

Thanks for the suggestion. Measuring extrapolation error in high-dimensional state-action spaces is challenging, as it requires sampling outside the dataset’s convex hull—a non-trivial task in such high-dimensional settings. Therefore, following the approach in [Ref.3], we visualized the Q-values learned by applying LN, RS-LN, and RS-LN&PA on the Inverted Double Pendulum with a 1D action space. The visualizations can be found at link (https://sites.google.com/view/pars-icml25). As shown in Fig A of the link, and consistent with the discussion in Figure 5 of the manuscript, we observe that training the Q-function without LN leads to divergence. Applying LN prevents this divergence. Furthermore, when reward scaling and PA are also applied, the extrapolation curve bends downward while maintaining a similar overall shape within in-sample actions. We also observe that with reward scaling and PA, the average magnitude of Q-values decreases, suggesting a general reduction in overestimation.

 

[R2] Applicability to discrete action spaces

In our work, we focused on the problem of Q-function extrapolation error in continuous action spaces. In discrete action spaces, the action set is finite, making the notion of being “outside” the feasible set less clear—and upward extrapolation of the Q-function less relevant. Since many practical tasks, such as robot control, involve continuous control, and many offline RL algorithms target this setting, we also concentrate on continuous control. We believe that achieving strong performance in this setting, compared to other offline RL algorithms that also focus on continuous control, is a substantial contribution.

 

[R3] Sensitivity analysis of $\alpha$ hyperparameter

As the reviewer pointed out, when applying PA, the hyperparameter $\alpha$ in eq.(3) is important. It should be set such that assigning a $Q_\text{min}$ penalty to infeasible actions does not interfere with the TD update for actions in the dataset. Accordingly, we set $\alpha$ to a small value in the range of 0.0001 to 0.001. We additionally conducted a sensitivity analysis on $\alpha$. As shown in Table C of the link (https://sites.google.com/view/pars-icml25), PARS performs well when $\alpha$ is in the range of 0.001 to 0.1. On the other hand, if $\alpha$ is too small (e.g., 0.0001), the penalty does not take effect properly, and if it is too large, it starts to interfere with the TD update on the dataset, leading to a decrease in performance.

 

[R4] Missing y-axis labels in figures

Sorry for the confusion, and thank you for the helpful comment. We'll make sure to clearly specify the y-axis in the revised version.

 

[Ref.3] Kim, et al. "Adaptive q-aid for conditional supervised learning in offline reinforcement learning." NeurIPS 2024.

 

Once again, we sincerely thank you and hope that our responses have adequately addressed all of your concerns.

We sincerely appreciate the positive feedback on our work and the opportunity to clarify any remaining uncertainties.

[R1] Theoretical justification of PARS

Thank you for the suggestions. As theoretical analysis of deep neural networks with nonlinearities is highly challenging, we empirically validated the effectiveness of RS-LN and leave theoretical aspects for future work. We will clarify this point in the limitations. As noted in Appendix B, we used ReLU due to its popularity and its frequent use in prior works, and exploring other activations better suited to RS-LN, especially with PA, is a promising future direction.

We can show the convergence of PARS, and we will add this to the paper. For this we define the following sets:

• $\mathcal{ID} = \{(s, a) \mid \beta(a \mid s) > 0\}$
• $\mathrm{OOD}_{\text{in}} = \mathrm{ConvexHull}(\mathcal{ID}) \setminus \mathcal{ID}$
• $\mathrm{OOD}_{\text{out}} = \left( \mathrm{ConvexHull}(\mathcal{ID}) \right)^c$

We define the operator $T^\pi_{\mathrm{pars}}$ as:

$T^\pi\_{\mathrm{pars}} Q(s,a) = \begin{cases} T^\pi Q(s,a), & \text{if } (s,a) \in \mathcal{ID}, \\\\ \mathbb{E}\_{(s',a') \in \mathrm{kNN}(s,a; \mathcal{ID})}[T^\pi Q(s',a')], & \text{if } (s,a) \in \mathrm{OOD}\_{\text{in}}, \\\\ Q\_{\min}, & \text{if } (s,a) \in \mathrm{OOD}_{\text{out}}, \end{cases}$

where $\mathrm{kNN}(s,a; \mathcal{ID})$ denotes the set of the k-nearest neighbors within $\mathcal{ID}$. We show that $T^\pi_{\mathrm{pars}}$ is a contraction under the$\| \cdot \|_\infty$ norm:

• Case 1:$\left|T^\pi_{\mathrm{pars}} Q_1 - T^\pi_{\mathrm{pars}} Q_2\right| = \left|T^\pi Q_1 - T^\pi Q_2\right| \le \gamma \| Q_1 - Q_2 \|_\infty$

• Case 2:$\bigl|T^\pi\_{\mathrm{pars}} Q_1 - T^\pi_{\mathrm{pars}} Q_2\bigr| = \left| \mathbb{E}\_{\mathrm{kNN}}[T^\pi Q_1] - \mathbb{E}\_{\mathrm{kNN}}[T^\pi Q_2] \right| \le \gamma | Q_1 - Q_2 |_\infty$

• Case 3:$\left| Q_{\min} - Q_{\min} \right| = 0 \le \gamma \| Q_1 - Q_2 \|_\infty$

Thus, $T^\pi\_{\mathrm{pars}}$ is a contraction, and the Q-function converges accordingly.

In Case 2, computing $\mathrm{kNN}(s,a; \mathcal{ID})$ is costly. Since the Q-function is approximated by a neural network, we let it implicitly learn this behavior instead of adding it to the loss. For the same reason, assigning $Q_{\min}$ can unintentionally affect nearby values. To prevent this, we introduce a “guard interval” that excludes actions just outside $\mathcal{A}_F$ from being evaluated as $Q\_{\min}$. In practice, a guard interval of about 1000 (see Figure 11) is sufficient to avoid influencing Q-updates.

 

[R3] Regarding convex hull definition

PARS aims to be a simple, efficient algorithm. While nonlinear boundaries are certainly possible, we opt for a convex hull to maintain simplicity.

 

[R4] Effectiveness of reward scaling in noisy environments

Assuming the reward is added by noise, scaling the noisy reward increases both true reward and noise proportionally, leaving the signal-to-noise ratio unchanged. To investigate further, we referred to [Ref.2], which analyzes offline RL under reward noise. Following this, we examined how PARS performs in such noisy reward settings. As shown in Table A of the link (https://sites.google.com/view/pars-icml25), in the presence of reward noise the trend of improved performance with increased reward scale still holds. So the core idea of PARS remains valid.

 

[R5] Regarding presentation

In the revision, we will move Appendix D to the main body, possibly by shortening Fig. 1 and others.

 

[R6] Role of Q-ensemble under the presence of PA penalty

As noted in Appendix B, PARS primarily targets extrapolation errors outside the convex hull and does not explicitly address OOD actions within it. These actions may still face approximation errors, which could be mitigated through ensemble-based uncertainty estimation. However, as shown in Table B (https://sites.google.com/view/pars-icml25), PARS outperforms SOTA in AntMaze even without an ensemble.

 

[R7] Dependency on policy expressiveness

As the reviewer noted, policy expressiveness is important, and methods like Diffusion-QL use diffusion models to enhance it. But, despite using diffusion, Diffusion-QL underperforms compared to PARS and suffers from the high inference cost of diffusion models.

Since policies are derived from the Q-function, we argue that improving Q-network expressiveness is even more crucial. Our work focuses on this, and notably, PARS achieves strong performance with just a simple MLP policy. Ultimately, we believe that combining efforts to improve both policy and Q-network expressiveness is key to building better offline RL algorithms.

 

[Ref.2] Yang, Rui, et al. "Towards Robust Offline Reinforcement Learning under Diverse Data Corruption." ICLR 2024.

 

Thank you again for the careful review. We hope that our responses have sufficiently addressed the concerns.

We sincerely appreciate your positive evaluation of our contribution and suggestions regarding various prior studies and potential directions for extending PARS.

[R1] Comparison of tuning search budget for hyperparameters with baselines

As noted in our manuscript, we tuned around 8 hyperparameter configurations, varying $\alpha$, TD3+BC $\beta$, and reward scale. For context, prior offline RL works we compare against also performed environment-specific tuning: ReBRAC used 25 configurations for MuJoCo and Adroit and 192 configurations (grid over 4 actor betas, 4 critic betas, 4 actor learning rates, and 3 critic learning rates) for AntMaze, SAC-RND used 19 (MuJoCo) and 9 (AntMaze), and SPOT, MSG, and MCQ used 6, 12, and 7, respectively. SAC-N and EDAC used 7 and 12.

As shown above, most of the prior works we compare against adopt a similar level (some even more extensive) of hyperparameter tuning, which supports the fairness of our comparisons. For TD3+BC, IQL, and CQL, we report results from their original papers (1–2 configurations), and ReBRAC [Ref.1] provides extensively tuned results for them—yet PARS still performs strongly.

Importantly, as shown in Appendix G, PARS achieves strong AntMaze performance even with a single hyperparameter configuration, outperforming other baselines.

 

[R2] Motivation for PA

As discussed in Section 3.2, our goal is to encourage the learned Q-function to exhibit downward extrapolation for OOD actions that lie outside the convex hull of the dataset. To achieve this, RS-LN first increases the expressivity of the neural network, which naturally enables downward extrapolation by preventing gradient updates that would otherwise increase the Q-value positively for OOD actions. In addition, PA makes this process more explicit by penalizing infeasible actions, thereby enforcing downward extrapolation. As demonstrated in the ablation study in Figure 9, RS-LN has a significant effect on its own, but applying PA on top further enforces downward extrapolation, contributing stability and performance of offline RL training.

 

[R3] Relation to prior work

Thank you for pointing out the relevant prior work. We will include the study in the revision. In particular, R-BVE also aims to mitigate Q-function extrapolation errors using SARSA-based in-sample Q-learning and ranking loss, performing policy improvement only once at the end. However, this single-step approach may achieve insufficient performance improvement in complex environments or where rewards are sparse. While the ranking loss aligns Q-values based on high-reward trajectories, it risks performance degradation when applied to low-reward data, requiring additional mechanisms like soft filtering that increase implementation complexity.

In contrast, PARS directly addresses Q-value extrapolation, especially outside the convex hull of the data, by combining RS-LN and PA. This provides a stable foundation for repeated policy improvement and achieves both simplicity and superior performance.

 

[R4] Regarding policy churn

Thanks for the insightful suggestion. The referenced paper studies policy churn—sudden shifts common in value-based methods like DQN. In contrast, PARS builds on TD3+BC with a behavior cloning term in the actor, which helps suppress such changes. While this likely reduces policy churn, continual critic updates may still cause some churn, potentially aiding exploration. Quantitatively analyzing churn in critic-regularized methods like PARS would be a valuable direction for future work.

 

[R5] Implications for safe RL

In our study, we define infeasible actions as those lying outside the set of feasible actions. However, this definition can be extended by incorporating various constraints depending on the objectives of different tasks. In particular, by establishing clear criteria for unsafe infeasible actions and designing the system to penalize such actions accordingly, the framework can be naturally extended to safe RL.

 

[R6] Where layer normalization is applied

We applied LN after all linear layers, including the input layer, except for the final output layer.

 

[R7] Regarding figure 10 (left)

As the reviewer mentioned in another comment, a key issue during online fine-tuning is that changes in the distribution can lead to unstable learning. Since PA was applied during offline training, we observed that removing PA during online fine-tuning led to instability. On the other hand, maintaining PA throughout resulted in more stable learning and led to significant performance improvements.

 

[Ref.1] Tarasov, Denis, et al. "Revisiting the minimalist approach to offline reinforcement learning." NeurIPS 2023.

 

We sincerely appreciate your thoughtful review once again. We hope that our responses have thoroughly addressed all of your comments.