Reply to Reviewer qrLu

We appreciate the reviewer for recognizing our contributions in both practice and theory, and for providing constructive suggestions.

Q1: The sample manipulation is an interesting mechanism with the gradient conflict as a signal. Is it solely applied to PCRPO or can be extended to other safe RL algorithms?

A1: Many thanks to the reviewer's recognition of the potential generalization of our sample manipulation approach. The reviewer is insightful that using the gradient conflict as a manipulation signal (metric) can improve the sample efficiency of extensive safe RL algorithms, not only PCRPO. Please refer to Q1 in the general response.

Q2: How to set $x^r_t$ or $x^c_t$ ? Explain Eq. (3)(4) of PCRPO and how is Eq. (3) functioning as 'projecting reward and cost gradients onto their normal planes'? Add more details about PCRPO.

A2: Thank the reviewer for the clarification suggestions.

• $x_t^r$ and $x_t^c$ represent the weights of the reward gradient and the safety cost gradient in the final gradient $w_{t+1}$. Please refer to Q2 of the general response for more details, where we explain how to choose/set $x_t^r,x_t^c$ and also conduct new ablation studies using varying $x_t^r$ and $x_t^c$.
• As the reviewer suggested, we have added an intuitive description for Eq.(3) in the main text and a more detailed introduction and illustration for PCRPO in the appendix. Recall the first line of Eq.(3) serves as the update rule when there is some conflict between the reward gradient and the safety cost gradient. In that case, to explain "project the reward and cost gradients onto their normal planes", we take the first term of Eq.(3) $g_r - \frac{g_r \cdot g_c}{ ||g_r||^2} g_c$ as an example. $\frac{g_r \cdot g_c}{ ||g_r||^2} g_c$ represent the vector projection of reward gradient $g_r$ on the cost gradient $g_c$. And the corresponding surrogate reward gradient $g_r - \frac{g_r \cdot g_c}{ ||g_r||^2} g_c$ becomes a vector that is perpendicular to $g_c$ --- the surrogate reward gradient not only improves reward but also won’t conflict with the cost gradient $g_c$ (since it is perpendicular to $g_c$, which we called the normal plane).

Q3: I see the dynamically change in rows 4, 7 in Algorithm 1 ESPO. Is it an existing component of PCRPO or a new trick in ESPO?

A3: We apologize for this confusion. The dynamic change in rows 4 and 7 of Algorithm 1 (ESPO) is indeed an existing component from PCRPO, not a new trick introduced in ESPO. This mechanism in PCRPO is designed to dynamically adjust the range of soft region (by setting $h^+, h^-$) to adapt the optimization behaviors. This trick can improve the trade-offs between reward performance and safety constraints objectives, which we maintained in ESPO to leverage its efficiency.

The contributions of ESPO focus on developing the new sample manipulation paradigm using gradient conflict as the metric/signal, which shows powerful advantages in sample efficiency, as shown in the paper and our new experiments (please refer to the previous answers A1 to Q1).

Q4: Are the theoretical results in Theorem 4.1 and Proposition 4.2 conclusions about PCRPO but not tailored to ESPO?

• A4: This is indeed one of the essential theoretical contributions of this work. The reviewer is correct that the convergence guarantees (Theorem 4.1) and the advantages of reduced optimization oscillation (Proposition 4.2) for our ESPO also hold for the prior work PCRPO, which are direct implications of our results. Notably, PCRPO is currently one of the state-of-the art safe RL algorithms, while still needing theoretical guarantees. This work closes this gap for PCRPO. In addition, ESPO is a broader algorithm framework --- PCRPO can be seen as a special case with identical sample size choices for all training iterations, which all the theoretical results hold for.

Q5: Assumption A.7 serves as a key to verify the sample size manipulation. Give more intuitive explanations and discussions to justify this assumption.

• A5: We have added discussions to justify Assumption A.7 in the new version: In words, Assumption A.7 assumes a local Lipschitz property of the Q function error term $\delta_t = \|Q_r^{\pi_{w_t}} - \overline{Q}_t^r\|_2$ with respect to the sample size $s_t^B$ used for estimating $\overline{Q}_t^r$ at any time step $t$. Typically, this can be satisfied easily since $\delta_t$ generally will decrease monotonically as the sample size increases in a polynomial order, such as $\delta_t \approx O(\sqrt{\frac{1}{s_t^B}})$ that usually holds with high probability verified by high-dimensional statistics (some concentration inequalities).

Q6: Change the x-axis of Figures 2 and 3 to sample steps rather than training epochs for fair comparisons. Besides, Table 1 should report the sample steps used to reach the same performance rather than that used in a fixed number of epochs.

A6: As the reviewer suggested, to provide a more fair and informative evaluation for the proposed method, we have made the following changes to show the results:

• For Figures 2 and 3: We have revised them to use the number of sample steps as the x-axis rather than training epochs, shown in Figures 5 and 6 in the pdf of the general response. This more reasonable illustration further underscores the sample efficiency of our proposed ESPO compared to baselines. For both Table 1 and Table 2: We have revised them (in the general response pdf file Table 1 and Table 2) to report the number of sample steps required to reach reasonable performance thresholds, rather than after a fixed number of epochs.
• Using these more fair metrics that the reviewer suggested, the results demonstrate even more efficiency of our ESPO compared to the baselines, in terms of not only sample efficiency, but also reward and safety performance.

Thanks for the authors' response. The supplementary experimental results are comprehensive, and the additional clarifications address most of my confusions. Thus, I decide to raise the score to 4. However, I still hold a slightly negative opinion due to two main concerns: (1) the limited applicability of sample manipulation to broader algorithms (the paper structure seems overly tailored to PCRPO, even though some additional experimental results about TRPO-Lag and CRPO are provided), and (2) the weak connection between the theoretical analysis and the main contribution (sample manipulation).

Just a reminder, it seems there is a typo in your response A2 where $g_r-\frac{g_r\cdot g_c}{||g_r||^2}g_c$ should be corrected to $g_r-\frac{g_r\cdot g_c}{||g_c||^2}g_c$ according to the paper.

Reply to Reviewer by8k

Many thanks to the reviewer for recognizing our contributions in terms of theory and practice.

Q1: Adding evaluations on more diverse, complex, or real-time environments would strengthen the generalizability claims.

A1: Thank the reviewer for insightful comments. We agree that testing on additional benchmarks or safe RL tasks is essential to substantiate the generalizability of ESPO. Following the reviewer's suggestion,

• We add three new sets of experiments on either a more complex class of safe RL problem --- multi-objective safe RL, or integrating our proposed technical modules with other safe RL algorithms to show the generalizability. Please refer to Q1 in the general response for details. The results from these additional experiments affirm that our method not only generalizes well to diverse, safe RL problems but can potentially bring benefits for extensive safe RL algorithms.
• More experiments on large and real-time environments leave to future work: (1) extending ESPO to safe MARL tasks, which involve multi-agent strategic interactions while maintaining safety constraints with large-scale applications in collaborative robotics or autonomous systems. (2) Deploying ESPO in real-robot control tasks requires addressing real-world challenges such as sensor noise, actuator delays, and environmental uncertainties.

Q2: More in-depth comparisons with a broader range of SOTA methods

• A2: Thank you for raising this point. First, we recall that we include the SOTA and classical ones of both primal methods (CRPO, PCRPO) and primal-dual methods (PCPO, CUP, PPOLag), which are the main classes of safe RL algorithms. So, instead of including more similar baselines, we choose to extend our ESPO to a new safe RL problem --- multi-objective safe RL and compare it to this problem's SOTA baseline CRMOPO. Details can be referred to the previous answer A1. These additional experiments verify the efficiency of ESPO (or its variants) not only in the standard safe RL problems, but also in more complex tasks.

Q3: How does ESPO perform in environments with high-dimensional state and action spaces compared to low-dimensional ones?

A3: Thank you for your insightful question. We observe that the proposed method ESPO demonstrates superior performance over baselines in both low-dimensional tasks (SafetyHopperVelocity-v1 (action: 3D, state: 10D) and SafetyReacher-v4 (action: 2D, state: 10D)) and relatively high-dimensional tasks (SafetyAntVelocity-v1 (action: 8D, state: 27D), SafetyWalker-v4 (action: 6D, state: 17D), and SafetyHumanoidStandup-v4 (action: 17D, state: 45D). ), but with particularly strong performance in low-dimensional tasks. For details, please refer to Figures 5 and 6 in the general response file.

• It is an interesting direction to implement ESPO for tasks with higher dimensional state and action, such as using an image as the state. This will involve more challenges, such as representation learning and computer vision. ESPO has potential in such high-dimensional tasks due to its sample efficiency advantages compared to prior art matches the pressing need of reducing required samples in those tasks -- such as the video game benchmark Atari (with the image as input) is typically one of the hardest benchmarks to solve and cost a lot of times to collect samples [1].

[1] Ye, Weirui, et al. "Mastering atari games with limited data." NeurIPS 2021.

Q4: What is the impact of noisy gradient estimates on the performance and stability of ESPO due to its reliance on gradient conflict signals?

A4: We appreciate the reviewer's insightful question. Typically, for all policy-based or actor-critic (safe) RL algorithms, the noisy gradient estimates can bring challenges since they are usually estimated with a batch of samples but not infinite samples.

• Inherent challenges for safe RL problems: The reviewer is correct that the noisy gradient estimates bring challenges for ESPO, while it is not primarily brought by algorithm design, but the inherent daunting challenges for safe RL problems ---a good balanced update direction is hard to find using noisy reward gradient and safety cost gradient estimates (there may be conflict happens between reward and cost).
• The sample manipulation using gradient conflict signals (our key technical module in ESPO) can potentially reduce the effects of noisy gradient estimates. Instead of limiting the applicability, the proposed sample manipulation is actually inspired by the noisy gradient estimate issue and designed to reduce its effect. In summary, the sample manipulation module will use more samples to estimate the final gradient when it needs more accuracy (when there is possibly a conflict between the reward and cost and a balance is required) and fewer samples when it tolerant more error --- improve sample efficiency. We acknowledge that the gradient conflict signal metric may also be noisy, but it won't directly hurt the final gradient direction. ESPO's superior performance implicitly shows that the performance is not sensitive to the gradient conflict signal errors.

Q5: Can ESPO be effectively integrated with off-policy or model-based RL methods?

• A5: The answer is Yes. The core capability of ESPO lies in its use of sample manipulations based on reward and cost gradients to improve sample efficiency. This mechanism can be seamlessly incorporated into the frameworks of off-policy and model-based approaches, where gradients are readily available. While it has not been tested, its fundamental principles suggest that such integration is feasible and potentially very beneficial. This is an exciting direction for future work.

Q6 The sensitivity of ESPO to various hyperparameters, such as learning rates and sample size thresholds.

• A6: Thank the reviewer for the comments on hyperparameters. Please refer to Q3 in the general response for details.

Reply to Reviewer oXwv

Q1: The paper experimented on SafetyReacher-v4, SafetyWalker2d-v4, and SafetyHopper/AntVelocity. Those safety tasks do not test generalization, such as those with safety gym.

A1: We appreciate the reviewer's insightful comments. To the best of our understanding of the reviewer's question, we clarify the relationship between the benchmarks (Safety-MuJoCo and Omnisafe) used in this paper and prior popular benchmark (safe gym). We have added this important clarification in the new version.

• Safety-MuJoCo is a new environment benchmark adapted from MuJoCo and Omnisafe is an algorithm benchmark (does not include new safe RL environments) that support the environments in Safety-Gymnasium. Here, Safety-Gymnasium is a popular suite, an evolved version of the original Safety-Gym, developed to support the latest gym API and maintain compatibility with ongoing research needs, given that Safety-Gym is no longer actively maintained.
• We use SafetyReacher-v4, SafetyWalker-v4, SafetyHumanoidStandup-v4 in Safety-MuJoCo. Safety-Gymnasium employs cost constraints such as velocity limits and rewards based on the robot’s speed. In contrast, Safety-MuJoCo benchmarks enable more kinds of safety cost constraints, such as the robot’s health—monitoring falls and joint forces— which haven’t been included/emphasized in Safety-Gymnasium. This provides a more comprehensive evaluation of the algorithms to generalize across more types/numbers of safety-critical constraints.
• Other environments (SafetyHopper/AntVelocity) that we used in this paper are exactly from Safety-Gymnasium. Omnisafe mainly serves as an algorithm platform that is used to test safe RL baselines in environments from Safety-Gymnasium.

Apologies for the confusion.

Notice that the experiments conducted in the paper mainly use velocity as the safety constraint and is a fixed threshold. In the original safety gym environment (openai/safety-gym), the safety constraints are typically obstacles and hazards (e.g., a car reaching the goal without hitting obstacles/overlapping with hazards). The location of the obstacles/hazards varies in each run, thus testing generalization in some sense.

I felt the safety-velocity benchmarks are easier than the openai safety-gym original settings (commonly used in SafeRL community). However, I acknowledge this is a weakness but not a solid rejection reason.

Reply to Reviewer qn1v

Q1: More discussion of the critical coefficients $x_t^r$ and $x_t^c$ of the algorithm's performance. How are the coefficients $x_t^r$ and $x_t^c$ computed?

A1: Thanks for raising this point! We conduct new ablation experiments and have added a detailed discussion for $x_t^r$ and $x_t^c$ in the algorithm design section with an intuitive explanation. Please refer to Q2 in the general response.

Q2: Do the hyperparameters of ESPO require careful tuning? (Tables 5 and 6 use different hyperparameters for different benchmarks.)

A2: Thanks for raising this point; we appreciate the opportunity to clarify it.

• Tuning hyperparameters for different tasks is not unique to ESPO, but widely considered in designing safe RL algorithms. For instance, PCPO [1] requires fine-tuning the projection distance $D$ within a primal-dual safe RL framework. CUP [2] involves fine-tuning the safety hyper-parameter $v$, and FOCOPS [3] necessitates fine-tuning the safety temperature $\lambda$. For practical considerations, hyperparameter tuning is somewhat necessary for heavily distinct tasks, while it's often a one-time process for each new kind of environments.

• Table 5 and Table 6 show the choices of the hyperparameters for different tasks: sample size adjustment parameters ($\zeta^+, \zeta^-$ defined in Equation (6-7)), safety constraint threshold (limit $b$), and soft region threshold parameters ($h^+, h^-$). We indeed provide an ablation study in Figure 4 in Appendix C lines 688-709, over the benchmark SafetyWalker2d. Specifically, Figures 4(a)-(c) show the results of ESPO when the safety constraint threshold (limit $b$) is different (30 or 40) with comparisons to baseline CRPO, which exhibits similar and stable reward performance when the limit $b$ varies. In addition, Figures 4(d)-(f) show the ablation w.r.t. sample size adjustment parameters ($\zeta^+, \zeta^-$) and demonstrate the stability of the performance under different ($\zeta^+, \zeta^-$) settings. We acknowledge the need of reasonable fine-tuning hyperparameters and are actively working on adaptive techniques in future work.

• [1] Yang, T., et al. (2020). Projection-Based Constrained Policy Optimization. ICLR 2020.

[2] Yang, L., et al. (2022). Constrained update projection approach to safe policy optimization. NeurIPS 2022.
[3] Zhang, Y., et al. (2020). First order constrained optimization in policy space. NeurIPS 2020.

Q3: In some cases either $h^-$ or $h^+$ is infinite, meaning the algorithm never performs a pure constraint satisfaction step or a pure reward optimization step. How are they chosen and do they require tuning for each environment?

A3: Our ESPO algorithm framework intentionally designs and allows for such choices. Different tasks have different preferences for optimizing rewards or prioritizing safety constraints. So generally, we need to choose/tune diverse $h^+, h^-$ to both satisfy the goal of different tasks and also the efficiency of the learning process, with intuitions shown below:

• We can choose $h^- = -\infty$ (i.e., always consider the safety constraint objective even if constraints are already satisfied). Such a choice will potentially make less oscillation around the safety constraint requirement threshold and always update the reward objective and safety constraint objective together. So this choice fits the tasks that the agent also wants to keep safe during the learning process, but not only exploiting the process, or less oscillation of the optimization process, which can accelerate the learning process.
• We can choose $h^+ = \infty$ (i.e., always consider the reward objective even if constraints are violated). This choice is suitable for those tasks that have less priority for satisfying the safety constraint, or less oscillation of the optimization process can accelerate the learning process.
• When $h^-, h^+$ are finite (the common choice): creating a soft region $[h^- +b, h^+ +b]$. When the safety cost objective falls into this soft region around the required limit $b$, both the reward objective and the safety cost objective will be considered for the optimization update rule, which can reduce the oscillation around the safety limit to some extent. With such finite choices, the algorithm keeps the possibility of optimizing only the reward or safety cost objective when the safety objective is pretty good or is largely violated, respectively.

Q4: How are the other hyperparameters ($X^+$, $X^-$, $h^-$, $h^+$) chosen? Does it require careful tuning for each environment?

A4: The reviewer raises an important point about hyperparameter selections. To the best of the author's understanding, the reviewer is asking about how to choose the sample size adjustment parameters ($\zeta^+, \zeta^-$ defined in Equation (6-7)), and soft region threshold parameters ($h^+, h^-$).

• The intuition of choosing $h^-$, $h^+$ are provided in above answer A3. Note that different tasks have different preferences for optimizing rewards or prioritizing safety constraints. So generally, we need to choose/tune diverse $h^+, h^-$ to both satisfy the goal of different tasks and also the efficiency of the learning process.
• The sample size adjustment parameters $\zeta^+$ and $\zeta^-$ determine the final sample size $X(1+\zeta^+)$ or $X(1+\zeta^-)$ in different scenarios (whether the gradient conflict occurs). As such, the final sample size typically is set close to the original sample size $X$. We provide ablation study w.r.t. $\zeta^+, \zeta^-$ in Figure 4(d)-(f) in Appendix C lines 688-709, over the benchmark SafetyWalker2d. Figures 4(d)-(f) demonstrate the stability of the performance under different ($\zeta^+, \zeta^-$) settings. It verifies that the ESPO is typically not sensitive to $\zeta^+, \zeta^-$ and just needs reasonable tuning.