Safety Representations for Safer Policy Learning

We sincerely thank the reviewer for recognizing the practicality and flexibility of our approach and for helping identify inconsistencies in the paper. We believe we have addressed all of your concerns in the manuscript with the changes highlighted in blue for ease of reference, but we remain available to address any further questions or comments before the end of the discussion period.

[W1]: Imprecise definition of risk representation (now safety representation).

1. Thanks to the reviewer’s comment, we’ve identified that this specific description of $R_H(s)$ (now $\mathfrak{S}_{H}(s)$) was imprecise. We’ve corrected this in the revised paper by replacing the line with: “$\mathfrak{S}_H (s)$ represents the probability that the agent remained safe throughout the fixed safety horizon $H_s << H$ without encountering an unsafe state based on the agent’s past experience.” As such, the safety representation encodes information about the agent’s past experience and cannot accurately predict whether the agent will fail following a novel policy.

2. We agree with the reviewer’s observation regarding the tradeoff between on-policy and off-policy data for learning the safety representations. Following common RL practice, we store all policy rollouts in a queue within a replay buffer, prioritizing recent policies. Specifically, we use a 1M-size buffer for all environments bar SafeMetaDrive where the buffer size is even smaller (200K), removing old trajectories in a FIFO manner. Further details are provided in Sec. 3.3 and Appendix A.1. The effect of modelling the safety distribution with off-policy vs on-policy data is explored in Fig. 8.

3. We have removed the theoretical analysis (previously in Appendix A) as it did not contribute relevant insights to the paper and could potentially confuse the reader.

4. In response to the reviewer’s feedback, we have added the necessary implementation details and design discussions in Sec. 3.3 and Appendix A.1, including a dedicated discussion on design choices in Appendix A.1.1.

[W2]: Safe Bellman Operator Based Methods

We thank the reviewer for pointing out this reference, which we have now included in the Related Work section. Safe Bellman operator-based methods, such as those in [4], CSC [5], and SQRL [6], also aim to model the likelihood of agent failure. However, unlike our approach, these methods use this information to constrain policy learning rather than informing it. In contrast to [4, 5, 6], we do not model a scalar probability of failure; instead, we learn a distributional representation of the state, characterizing it as a distribution over the proximity to unsafe states based on the agent’s past experience. Results for CSC [5], a popular off-policy baseline that uses the Safe Bellman operator to model the likelihood of failure for constraining policy actions, show that such methods can lead to overly conservative policy learning (Fig. 5 in the revised paper). Additionally, our experiments demonstrate that the proposed method, SRPL, significantly improves the performance of CSC [5], both in terms of constraint satisfaction and sample efficiency (Fig. 5).

Being a representation learning approach, our method indeed provides no new theoretical guarantees, but it does inherit the theoretical guarantees of the baseline method to which it is applied.

[W3]: Thanks for pointing this out. We’ve corrected the caption for Figure 3.

[W4]: Additional Baselines and More Environments

1. Taking the reviewer’s comment about limited and outdated baselines into consideration, we’ve added three additional baselines from recent online SafeRL literature: CRPO [1], CVPO [3], and SauteRL [2]. The results for these baselines have been shown in Fig. 4 & 5, for the sake of better clarity we’ve included pairwise plots in Fig. 18-21 in the Appendix.

2. In the original paper, our focus was to show the effectiveness of our method across a diversity of robotics tasks. We’ve since added results for three more environments: SafeMetaDrive, Hopper and Walker2d. The importance of SafeMetaDrive as pointed out by the reviewer in [Q3] convinced us to put the results for SafeMetaDrive in the main paper (Fig. 4) and move the results for Ant to the Appendix (Fig. 12).

We'll continue in the next comment.

[1] Xu et al. CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee In ICML 2021

[2] Sootla et al. Sauté RL: Almost Surely Safe Reinforcement Learning Using State Augmentation In ICML 2022

[3] Liu et al. Constrained Variational Policy Optimization for Safe Reinforcement Learning In ICML 2022

[4] Fisac, Jaime F., et al. "Bridging hamilton-jacobi safety analysis and reinforcement learning." 2019 International Conference on Robotics and Automation (ICRA). IEEE, 2019.

[5] Bhardwaj et al. Conservative Safety Critics for Exploration In ICLR 2021

[6] Srinivasan et al. Learning to be Safe: Deep RL with a Safety Critic

We would like to express our sincere gratitude for the thorough and detailed review of our work. We believe that the clarity and quality of the paper have been significantly enhanced by incorporating your valuable suggestions. We have addressed all of your concerns in the revised version, with the changes highlighted in blue for ease of reference.

[W1]: Confusing use of the term “risk” throughout the paper

We acknowledge the reviewer’s concern regarding the potential ambiguity of the term "risk," which is often interpreted as a risk measure. To address this, we have revised the terminology throughout the paper and method. Specifically, "risk representation" has been updated to "safety representation," and "risk model" is now referred to as the "Steps-to-Cost (S2C) model." Additionally, the framework has been renamed to "Safety Representation for Policy Learning (SRPL)" in the revised manuscript.

[W2]: The definition of the risk model is inconsistent across the paper

We agree with the reviewer about the inconsistency of the definition of the risk model in the original version of the paper. In our approach, we model the distance to unsafe states, which may not always correspond to a constraint violation or failure. We agree that modelling the distance to constraint violation is also a sensible choice. To address this in the paper, we’ve made a single definition of the S2C model (earlier risk model) in Sec. 3.2 and 3.3. Our design choice is based on our intuition of a state-centric safety representation that is independent of task specifications, enabling generalization across different tasks and constraints, as long as the definition of unsafe states remains consistent. In Appendix Figure 15, we present results demonstrating the transfer of the learned safety representation across tasks for different cost thresholds. Specifically, the original representation, trained with a safety threshold of $\beta=10$, was then frozen and applied to the target task for different cost thresholds.

[W3 & Q1]: Experimental Clarifications

Upon further reflection, we agree with the reviewer that the results were not presented clearly. The reasoning behind presentation choices like showing costs for the first $2M$ steps in the case of Safety Gym environments was to highlight the low episodic cost incurred by the SRPL (earlier RIPL) agents in comparison to the baseline algorithms. All algorithms converge around 2M timesteps in terms of Episodic cost. As a result, the early benefits of SRPL were difficult to see when plotted on 10M time steps. The choice behind not showing all of the SRPL (earlier RIPL) versions of the baseline algorithm was to prevent excessive overlap which was reducing the clarity of the figure. The same applies to some of the baselines plotted as horizontal lines.

To address the reviewers’ concerns and improve the clarity of the paper, we’ve modified the presentation of the results in the following way:

1. We’ve plotted the results for Episodic Cost for Safety Gym with the x-axis on the log-scale in Fig. 4 thus showing the benefits of using SRPL on Episodic Cost clearly. We also now include a line indicating the constraint threshold for Safety Gym environments.
2. For AdroitHandPen and SafeMetaDrive environments, we’ve included a plot in the Appendix (Fig. 17) that shows the episodic cost along with the constraint threshold.
3. To show all versions of the vanilla algorithms and their SRPL counterparts, we’ve added pairwise plots for each environment and algorithm combination in Figures 18-21.
4. With the inclusion of CVPO (an off-policy algorithm) as a baseline, we decided to show off-policy algorithms (CVPO and CSC) with their SRPL counterparts separately in Fig. 5.

We are almost at the character limit for this comment so we will continue with our response in a separate comment.

We sincerely thank the reviewer for their thoughtful feedback and for recognizing the value of our work. We are pleased that the motivating example and ablation study were insightful for the reviewer.

Weaknesses:

[W1]: Additional Baselines

We concur with the reviewer’s suggestion to include additional baselines. In response, we have incorporated three baselines: CRPO [1], CVPO [3], and SauteRL [2]. The results for these baselines are presented in Figures 4 and 5 of the revised paper. Additionally, we have expanded our evaluation to include three new environments: SafeMetaDrive, Hopper, and Walker2d. To enhance result visualization, we have included pairwise comparison plots between each baseline and its SRPL counterpart across all environments, provided in Figures 18–21 of the Appendix.

[W2 & Q1]: Accumulated Cost as a Risk Measure

Thank you for bringing paper [4] to our attention. We have added it to the Related Work section. Additionally, we have incorporated results for SauteRL [2], which aligns with the concept proposed in [4] and benefits from widely available implementations. SauteRL introduces an accumulated cost-based risk measure by augmenting the state space with this information. However, a notable limitation of this approach is its ineffectiveness when the cost threshold is 1 or lower. In such cases, the accumulated cost remains effectively zero for all states except the one causing a constraint violation, which can hinder effective constraint satisfaction. We have included supplementary results demonstrating that while SauteRL performs well on Safety Gym tasks with a cost threshold of 10, it fails to generalize effectively to environments like SafeMetaDrive and AdroitHandPen. These results are presented in Figure 4 of the revised manuscript. Furthermore, we show that integrating SRPL with SauteRL enhances its performance across all evaluated environments.

[1] Xu et al. CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee In ICML 2021

[2] Sootla et al. Sauté RL: Almost Surely Safe Reinforcement Learning Using State Augmentation In ICML 2022

[3] Liu et al. Constrained Variational Policy Optimization for Safe Reinforcement Learning In ICML 2022

[4] Solving Richly Constrained Reinforcement Learning through State Augmentation and Reward Penalties. arXiv preprint arXiv:2301.11592.

Thank you for your positive feedback and for acknowledging our contributions. We have revised the paper to address the questions and concerns raised by the reviewer. All modifications to the original manuscript have been highlighted in blue for ease of reference.

Weaknesses:

[W1]: Additional Baselines and Related Work

Thank you for bringing this to our attention. In response, we have incorporated three additional baselines from recent literature: CRPO [1], SauteRL [2], and CVPO [3]. The results for these baselines are presented in Figures 4 and 5. Furthermore, we have expanded the Related Work section to include more recent research in Safe RL.

[W2]: Visualization of Safety Representations

We acknowledge the challenge of modelling long-term dependencies in the current implementation of the safety representation, a difficulty shared by standard reinforcement learning algorithms. We are particularly interested in exploring approaches that can effectively capture long-term dependencies and demonstrate their utility in environments where long-term safety considerations are critical. To illustrate how the risk model—now referred to as the S2C model—assesses the safety of individual states, we have included its output at convergence for three distinct states in the Ant environment. These visualizations are presented in Figure 9 of the Appendix.

Questions:

[Q1]: Sensitivity of RIPL (now SRPL) to Bin Size

In our experiments, we consistently utilized a bin size of 2 and a safety horizon $H_s$ of 40 steps, except for the Safety Gym environments, where a bin size of 4 and a safety horizon $H_s$ of 80 were employed. We did not perform fine-tuning across different bin sizes or safety horizons. Detailed implementation specifics are provided in Appendix A.1. Additionally, an ablation study exploring various bin sizes and safety horizons is presented in Figures 13 and 14 of the Appendix. The results indicate that smaller bin sizes generally yield superior performance; however, the marginal gains diminish as the bin size decreases further. Figure 14 demonstrates that increasing the safety horizon enhances performance, although the improvement plateaus beyond a certain threshold. This phenomenon can be attributed to the greater significance of short-term safety information over long-term considerations in SRPL agents.

[Q2]: Finetuning safety representations on the target task

In response to the reviewer’s feedback, the revised version of the paper now includes results on fine-tuning the risk model—now referred to as the S2C model—on the target task, as presented in Figure 6. The findings demonstrate that fine-tuning the S2C model significantly enhances both sample efficiency and constraint satisfaction. This improvement is particularly notable when transferring both the policy and the S2C model, leading to superior overall performance.

[1] Xu et al. CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee In ICML 2021

[2] Sootla et al. Sauté RL: Almost Surely Safe Reinforcement Learning Using State Augmentation In ICML 2022

[3] Liu et al. Constrained Variational Policy Optimization for Safe Reinforcement Learning In ICML 2022