Safety-Polarized and Prioritized Reinforcement Learning

We would like to sincerely thank the reviewer for the thorough reading of our paper. We address your valuable questions in the following responses.

Q1: Sparse cost challenge

In our setup, episodes terminate immediately upon safety violations, as we treat safety as a hard constraint. The learner receives no cost feedback until a violation occurs, resulting in at most one non-zero cost per episode—what we refer to as sparse safety cost. This sparsity makes it difficult to learn safety estimations, as the agent must infer early indicators of unsafe outcomes without intermediate cost signals. These challenges motivate our use of PER, which prioritizes unsafe transitions to improve policy safety.

Q2: Question about proving $\gamma$-contraction

A brief proof here: https://anonymous.4open.science/r/rebuttal-6740/proof_details.png.

Q3: Two more hard-constraint baselines

We conduct experiments with PPOBarrier on highway benchmarks and find it performs conservatively, with consistently low rewards across all tasks. In contrast, our SPOM and SPOM_PER achieve a better tradeoff. Detailed SWU scores, crash rates, and episode rewards are provided in https://anonymous.4open.science/r/rebuttal-6740/ppo_barrier_table.png, along with training curves in https://anonymous.4open.science/r/rebuttal-6740/ppo_barrier_training_curve.png. As for the second referenced work, its setting assumes access to analytical violation functions, while our SA-REF is learned without any domain-specific knowledge.

Q4: Literature on HJ reachability and REF

We will include all relevant literature on HJ reachability and REF in a new section of our Related Works. Briefly, early works employed HJ reachability value functions to assess state feasibility, relying on known system dynamics and numerical methods. Some studies approximate unknown dynamics using Gaussian Processes (Zhang, W. et al. 2023)or symbolic regression (Wang, Y. et al. 2024). Once reachability is computed, the state space is partitioned into feasible/infeasible regions to guide policy optimization (Zheng, Y. et al. 2024). Other works focus on formal safety verification of DRL systems (Dony, Y. et al. 2024). However, HJ-based value functions are not well-suited for stochastic MDPs during RL training. To address this, REF estimates unsafe probabilities via backward reduction (Ganai, M. et al. 2024). Our work extends REF to the state-action level, enabling state-dependent action masking to reduce safety violations.

Q5: Literature on hard-constraint safe RL

Thanks for the suggestions. We indeed discuss hard-constrained safe RL in our related works. Based on policy optimization strategies, we categorize prior works into: action correction and action masking. Action correction modifies actions after they are proposed (e.g., via projection), while action masking alters the action distribution directly. The papers mentioned by the reviewer largely fit within this framework and will be incorporated into our Related Works. For instance, Tan,D. et al. 2024 uses value functions as control barrier functions in a shielding setup, aligning with action correction. Yang,Y. et al. 2023 learns certificates that softly penalize unsafe actions, guiding policies toward safer regions—similar in spirit to action masking. Suttle, W. et al. 2024 introduces truncated sampling from state-dependent safe action sets, which we view as a general form of action masking. We will include all the literature in our paper for completeness.

Q6: How does the “off-policy” affect the theoretical analysis?

We assume that actions are sampled from a behavior policy $\pi_b$ in Section 4.2, which is a common assumption in Q-learning to ensure sufficient exploration of the state-action space. Our theoretical analysis follows the classical Q-learning framework: we prove the $\gamma$-contraction property of our operator and demonstrate convergence in tabular MDPs. This directly parallels traditional Q-learning theory.

Q7：Gating operator

The gating operator in Section 5.1 represents our hard masking rule. In Theorem 4.4, we prove its convergence in tabular MDPs. When extending this method to deep Q-learning, we found that using Equation (17) as the objective yields better empirical performance. We report the ablation results in Section 6.3, where ‘OAM’ refers to Optimal Action Masking, alongside other polarization function choices.

Q8: Key terms

•Maximal safety: A policy that achieves the lowest probability of safety violations.

•Safety polarization: The use of our polarization function to mask actions with relatively high SA-REF values.

•REF: can be regarded as the probability for the current state to reach unsafe regions under the current policy.

We hope our responses have addressed your concerns and we welcome any further suggestions to improve the paper. If you find our clarifications satisfactory, we would greatly appreciate it if you could consider increasing your score accordingly.

We would like to sincerely thank the reviewer for the valuable feedback of our work.

Q1: The soundness of our assumption that there exists a sufficiently large policy space with minimal unsafe probability.

We assume that in the environments we consider, there exists a sufficiently large policy space in which the probability of unsafe events is minimal. Such environments are common in autonomous driving and robotic control. For instance, there are many feasible driving policies that allow a vehicle to operate safely on the road without collisions, as well as a sufficiently large set of policies enabling a robot to navigate without falling or colliding with obstacles—indicating that a large policy space with minimal unsafe probability indeed exists in these environments. This assumption aligns well with many practical scenarios. Building on this assumption, our MAXSAFE policy selects from within this space and aims to optimize reward performance while ensuring safety.

Q2: The proposed method is limited to discrete action spaces.

Our action masking method is currently only suitable for RL tasks with discrete action spaces, as simple action masking strategies may not perform well in continuous settings. We view this as an interesting direction for future work, which would require more advanced techniques to better optimize the trade-off between safety and performance.

Nevertheless, the proposed safety Prioritized Experience Replay (safety PER) is a general approach and can also be applied to the learning of the cost critic. To validate this, we run experiments on two environments—SafetyCarCircle and SafetyPointGoal—from the Omnisafe benchmark, https://github.com/PKU-Alignment/omnisafe. The results suggest that the simple SAC-Lag algorithm can benefit from being combined with safety PER to achieve better performance. We use 6 random seeds for evaluation, and we provide both the training curves and the mean/std of episode return and episode cost. The results are available at:https://anonymous.4open.science/r/per_test/results.png, https://anonymous.4open.science/r/per_test/training_curve.png.

Q3: Is a SAC-Lagrangian baseline (modified to handle discrete actions) with different trade-off settings a better baseline?

We include a variety of off-policy baseline RL algorithms. Since our tasks involve discrete action spaces, we adopt DQN-based off-policy methods. For example, our baseline RCDQN is an off-policy algorithm, and all algorithms with names ending in “DQN” are off-policy methods designed for discrete action settings.

Q4: Usefulness of the SWU score.

We adopt the evaluation setting from previous literature to demonstrate the trade-off between safety and reward. In our paper, we also report the training results for episode reward and crash rate in Table 5.

Q5: Choice of benchmarks

Regarding the concern about why we do not use Safety-Gym as our benchmark suite, the primary reason is that our setting fundamentally differs from that of Safety-Gym. In our framework, safety is treated as the highest priority. Specifically, we assume that an episode terminates immediately upon the execution of any unsafe action (i.e., entering an unsafe region). In contrast, Safety-Gym employs a scalar cost signal and allows temporary safety violations as long as the expected cumulative cost remains within a predefined budget. For example, the “circle” environment in Safety-Gym’s safe navigation task is conceptually similar to the “circle” environment in our benchmarks. However, the underlying assumptions differ: in Safety-Gym, safety is treated as a constraint, and episodes continue despite violations, provided the overall cost remains within budget. In contrast, our formulation enforces strict safety—any violation leads to immediate episode termination. This setting better reflects real-world scenarios where safety is non-negotiable and must take precedence over performance. A formal comparison of the two formulations is provided in the Preliminaries section of our paper. We refer the reviewer to that section for detailed definitions. Due to our strict safety assumption, the cost signal in our setting only appears at the end of an episode, resulting in significantly sparser cost feedback. To address this challenge and improve learning efficiency, we introduce Prioritized Experience Replay (PER) in our SPOM_PER method. This design is motivated by the need to balance reward maximization with strong safety guarantees.

We hope our responses have addressed your concerns and we welcome any further suggestions to improve the paper. If you find our clarifications satisfactory, we would greatly appreciate it if you could consider increasing your score accordingly.

We would like to sincerely thank the reviewer for the positive recognition of our work.

Regarding the concern about why we did not choose Safety-Gym as our benchmark suite, the primary reason is that our setting fundamentally differs from that of Safety-Gym. In our framework, safety is treated as the top priority. Accordingly, we assume that an episode terminates immediately upon the execution of any unsafe action (i.e., entering into any unsafe region). In contrast, Safety-Gym employs a scalar cost signal and allows safety violations as long as the expected cost-to-go remains within a predefined budget. For example, the “circle” environment in Safety-Gym’s safe navigation task is conceptually similar to the “circle” environment in our benchmarks. However, the underlying assumptions are different: in Safety-Gym, safety is treated as a constraint, and episodes continue even when temporary violations occur, as long as the total cost remains within the safety budget. In contrast, our formulation assumes strict safety—any safety violation immediately terminates the episode. This setting better reflects scenarios where safety is non-negotiable and takes precedence over performance. The mathematical comparison between these two setups is provided in the Preliminaries section of our paper, and we refer the reviewer to that part for detailed formulation. As a result of our strict safety assumption, the cost signal in our setting only appears at the end of an episode, making it significantly more sparse. To address this challenge and improve learning efficiency under such sparse cost feedback, we propose the use of Prioritized Experience Replay (PER) in our SPOM_PER method. This design decision is also motivated by the need to balance reward maximization with strong safety guarantees.

We truly appreciate the reviewer’s positive feedback and encouragement. We hope our responses have addressed your concerns, and we welcome any further suggestions to improve the paper.

We would like to express our sincere gratitude to the reviewer for the valuable feedback and constructive comments. To better quantify the trade-off between reward and safety, we adopt the SWU score introduced by Yu, H., Xu, W., and Zhang, H. in Towards Safe Reinforcement Learning with a Safety Editor Policy (NeurIPS 2022b) as our final evaluation metric to compare the performance of different algorithms. The following sections address the two main questions raised by the reviewer.

Q1: The soundness of our assumption that there exists a sufficiently large policy space with minimal unsafe probability.

We assume that in the environments we consider, there exists a sufficiently large policy space where the probability of unsafe events is minimal. Such environments are common in autonomous driving and robotic control. For instance, there are many possible driving policies that allow a vehicle to operate safely on the road without collisions, and a sufficiently large set of policies that enable a robot to navigate without falling or colliding with obstacles—indicating that a large policy space with minimal unsafe probability indeed exists in there environments. This assumption aligns well with many practical scenarios. Building on this assumption, our MAXSAFE policy selects policies from this space and aims to optimize reward performance while ensuring safety.

Q2: The proposed methods SPOM_PER and SPOM do not always guarantee the lowest crash rate.

In our experiments, we use the SWU score to evaluate the trade-off between safety and reward. As shown in Table 1 of our paper, SPOM_PER and SPOM achieve the best overall performance across all six benchmark environments. Although the Recovery baseline achieves the lowest crash rates in Roundabout and Intersection, its reward performance is significantly worse, indicating that it suffers from an overly conservative policy in these two environments. In contrast, our SPOM_PER and SPOM attain the highest SWU scores across all benchmarks. Also, the results demonstrate that applying Prioritized Experience Replay (PER) to our proposed SA-REF effectively reduces safety violations while preserving strong reward performance.

We sincerely appreciate the reviewer’s constructive feedback, which has been invaluable in improving the clarity and rigor of our work. We hope our detailed responses have addressed your concerns, and we welcome any further suggestions to strengthen our paper.