Risk-Averse Constrained Reinforcement Learning with Optimized Certainty Equivalents

Thank you for your detailed review and thoughtful comments. We hope the responses below address your questions clearly.

---

Questions

1. Thank you for the question! We originally included a table clarifying this distinction, but omitted it due to page constraints. We will take care to make this distinction clearer in the text. The key difference is that the constraint in [12] applies the risk measure to the discounted return of a cost function (see Eqs. (2) and (3), and the definition of $\mathcal{J}^\theta(x^0)$ at the bottom of p.5 in [12]). We refer to this as the return-based formulation, in contrast to the reward-based formulation used in our work. In our framework’s notation, the return-based formulation (like in [12, a]) can be written as:

$\max_{t} \left(t - \frac{1}{\beta} \left(t - \mathbb{E}\left[\sum_{\tau=0}^\infty \gamma^\tau r(s_\tau, a_\tau) \right]\right)_+ \right),$

while the reward-based formulation (our work), which places the risk measure inside the occupancy measure, becomes:

$\max_{t} \mathbb{E}\left[ \sum_{\tau=0}^\infty \gamma^\tau \left(t - \frac{1}{\beta}(t - r(s_\tau, a_\tau))_+ \right) \right].$

Similarly, in [a], the spectral risk measure is applied to $G_{C_i}^\pi$ (Eq. (5)), which is also a return (defined at the bottom of p.2). This places it within the return-based framework of [12], though extended to more general spectral risk measures.

We hope this clarifies the distinction and will make these points more explicit in the manuscript. There is more that we can say and we are happy to further elaborate or follow up as needed.

2. We will add this to the background section. The conditions on $g$ appear as Definition 2.1 in Ben-Tal and Teboulle [4]. Specifically, $g$ is a utility function that is proper, closed, concave, and non-decreasing, and it satisfies $g(0) = 0$ along with $1 \in \partial g(0)$, where $\partial g(\cdot)$ denotes the subdifferential map.

3. As noted in our response to Reviewer WADh, we will make sure to include this explicitly. The subgradient can be computed using Equation (8) and the definition of $r'$ provided on line 149, and we will ensure that this is clearly stated in the manuscript.

4. This is a good question! Like many nonconvex problems that arise in DL/RL, we can usually only guarantee convergence to different notions of local stationary points rather than true global optima. In our setting, we use the Moreau envelope which is standard in nonsmooth weakly convex optimization. As written in Theorem 3.12, the quality of the policy depends on the scale of $\delta$ to be small (we want Assumption 3.6 to say that the RL solver returns nearly the optimal policy for the inner problem) and we’d ideally want the non-convexity to be such that all $\epsilon$-stationary points have values close to the global optima. We could state Theorem 3.12 in the main body more simply (e.g., assuming $\delta = O(\epsilon^2)$) if it would help the presentation.

5. The objective function—that is, the modified reward implementing the risk-aware constrained objective—is explicitly defined in terms of the state, action, and original reward function in Appendix B.3.1, Equation (25).

The state and action spaces are defined by the joint positions, velocities, and other physical properties of the agents in the MuJoCo simulator. These elements vary across environments and are governed by the physics engine, as documented on the MuJoCo website. The Safety-Gymnasium environments we use are built directly on MuJoCo v4, and we do not modify the state, action, or reward definitions provided therein. Therefore, loading any environment from the suite automatically applies the default state-action-reward configuration.

Given the complexity and high dimensionality of these environments, we were unable to include some of these details in the main body. However, we will point readers to the simulator's official documentation and, if the paper is accepted, elevate the content of Appendix B.3.1 to the main body using the additional page allowance.

6. Thank you! We now fixed it.

7. We will define Slater’s condition in the Appendix and reference it accordingly. Slater’s condition, a strict feasibility constraint, is often necessary to establish strong duality. It states that there exists a policy which is strictly feasible (i.e., there exists some policy which satisfies the constraints with strict inequality).

Safe Navigation Results Added

Thank you for acknowledging our responses and for your participation in the rebuttal period. We would like to follow up with additional experiments conducted during the rebuttal phase. These new experiments were performed on the Safe Navigation tasks in the Safety-Gymnasium suite.

---

The table below summarizes the converged episodic cumulative reward and cost, denoted by $R$ and $C$, respectively, for our algorithm and the unconstrained vanilla PPO.

• Our method is able to effectively minimize cost while maintaining high rewards, consistent with our findings in the Safety-Velocity experiments—even in the presence of discrete constraints.

• Notably, in the Push environment, our algorithm achieves higher rewards than vanilla PPO while maintaining zero cost. This is due to the fact that the vanilla PPO agent occasionally gets stuck on hazards and fails to reach the goal—suggesting that it struggles to find a global optimum in the absence of constraints and our algorithm works like a regularizer of the path.

• As expected from our Safety-Velocity experiments, reducing cost generally results in some reward degradation—highlighting the inherent trade-off in risk-constrained optimization.

---

We will incorporate these results into the final version if the paper is accepted. Once again, thank you for your constructive comments—they have helped improve the quality of our work.

Thank you for your thoughtful review and for raising this question, which we had considered during the development of the manuscript. We hope the following response effectively addresses the identified weaknesses and provides a clear answer to your question.

---

Weaknesses

1. We primarily selected Safety-Velocity tasks as they are based on MuJoCo environments, which are widely recognized benchmarks in deep RL. These tasks are also the most realistic and complex within the Safety-Gymnasium suite, featuring high-dimensional state and action spaces, 3D spatial structure, and intricate physics dynamics. In contrast, many of the other tasks in the suite are either toy problems, Atari-style environments, or limited to 2D settings.

Nonetheless, we have initiated experiments on Safety-Navigation as well and will include the results if they are completed in time for a camera-ready version.

2. Please refer to our response to your question below.

---

Question

Thank you for the thoughtful question! Our experiments were primarily designed to demonstrate the theoretical properties and practical behavior of our algorithm, rather than to benchmark against prior methods. That said, we acknowledge the value of such comparisons.

We were unable to implement the approach of Bonetti et al. [7], as their block-coordinate descent algorithm as written in the paper is missing implementation detail and is not accompanied by publicly available code. (Our attempts to obtain it from the authors were also unsuccessful.) Nonetheless, our framework can be adapted to approximate their setting—for example, by fixing the Lagrange multiplier to zero and disabling updates—allowing us to mimic an unconstrained optimization scenario similar to [7]. If the reviewer believes such a comparison would improve the clarity or impact of our presentation, we are happy to include it.

We also note that our approach, which leverages stochastic gradient methods, is conceptually distinct from the optimization scheme in [7]. Even in the absence of a direct implementation, we will consider including ablations or visualizations to highlight behavioral differences, should space and time permit.

New Experiments Added

We appreciate your continued engagement during the discussion period. Following your comments, we began experiments on SafetyNavigation, which have been completed now. Below, we provide the experimental details and results in numerical form, in accordance with rebuttal guidelines that prohibit the use of external links, images, or plots.

---

Experimental Details

We have added three new tasks from the Navigation section of Safety-Gymnasium:

• Circle: The agent must move in a circular trajectory around the center while staying within boundaries. Exiting the circular area from the inside results in a cost of +1. Two walls are created by constraining the agent along the x-axis.
• Goal: The agent must navigate to the goal while avoiding 8 hazards. Each contact with a hazard incurs a cost of +1, with a maximum total cost of 8.
• Push: The agent must push a box to the goal location while avoiding 2 hazards. Each contact with a hazard incurs a cost of +1, with a maximum total cost of 2.

To manage time constraints, we used the point agent (with 12-dimensional state and 2-dimensional action space), since the constraint optimization task is independent of the agent’s body. Using a more complex agent would primarily affect the PPO solver’s efficiency, not our wrapper algorithm. This setup differs from Safety-Velocity, where the agent’s physical design (e.g., joints) directly impacts constraint violations such as velocity.

As in our original experiments, we introduced action noise to simulate uncertainty and highlight the risk-awareness capabilities of our algorithm.

Note: Constraints in these navigation tasks are inherently discrete, in contrast to the continuous constraint shaping in Safety-Velocity. As a result, dual and CVaR variable learning is less smooth, making the optimization problem inherently more challenging.

---

Results

The table below summarizes the converged episodic cumulative reward and cost, denoted by $R$ and $C$, respectively, for our algorithm and the unconstrained vanilla PPO. The vanilla PPO results are consistent with those reported in Table 5 of the Safety-Gymnasium paper.

• Our method is able to effectively minimize cost while maintaining high rewards, consistent with our findings in the Safety-Velocity experiments—even in the presence of discrete constraints.

• Notably, in the Push environment, our algorithm achieves higher rewards than vanilla PPO while maintaining zero cost. This is due to the fact that the vanilla PPO agent occasionally gets stuck on hazards and fails to reach the goal—suggesting that it struggles to find a global optimum in the absence of constraints and our algorithm works like a regularizer of the path.

• As expected from our Safety-Velocity experiments, reducing cost generally results in some reward degradation—highlighting the inherent trade-off in risk-constrained optimization.

---

Conclusion

We will include these results, along with relevant discussion, in the camera-ready version if the paper is accepted.

That said, we would be grateful if you would consider raising your score. We welcome any additional suggestions or feedback.

Thank you for your review and insightful comments. We hope the responses below effectively address the identified weaknesses and answer your questions.

---

Weaknesses

1. Please see our response to Question 1 below.

2. Thank you for your feedback. While our work builds on established tools, we believe the contributions lie in their nontrivial integration and the resulting theoretical framework. The conclusions we draw and the guarantees we establish are not straightforward applications of existing results but require careful analysis and novel synthesis across these domains.

3. We will revise the exposition to make this connection clearer. Specifically, the problem in Eq. (6) corresponds to the left-hand side of the equation on line 180. We will add a reference to the earlier constrained problem and briefly remind the reader of the Lagrangian duality framework to improve clarity and readability.

---

Questions

1. This is a valid point—thank you for raising it. Although this condition may be hard to check in practice, the framework still serves as a valid Lagrangian relaxation—even when this assumption is not met and in the absence of strong duality—as is often the case in constrained optimization.

2. We focus on CVaR as it is one of the most widely used and well-understood risk measures, which we believe helps with clarity of exposition. That said, our framework is designed to accommodate a broad class of OCEs. We can add explicit equations and explanations in the Appendix to illustrate how the formulation and algorithm would adapt to other OCEs to further highlight the flexibility of our approach.

3. Thank you for the suggestion! We will aim to include additional experiments, including comparisons with state-of-the-art baselines, if the paper is accepted. As noted in the limitations section, we anticipate that computational cost may become a bottleneck in high-dimensional settings.

In our preliminary experiments, we evaluated a vanilla PPO agent in a deterministic, non-constrained Ant environment. Across five seeds, the agent failed to converge to meaningful rewards, indicating that the solver was insufficient for this setting. Consequently, we excluded Ant from our main experiments.

We also ran vanilla PPO on the reported environments as a baseline for comparison. However, to better isolate the practical effectiveness of our proposed risk-aware method, we chose not to report those results. The vanilla PPO agent violated constraints in 85–90% of episode steps, and the rewards were highly volatile due to noise injected into the actions.

If preferred, we can include the vanilla PPO results in the final version. We will also add a brief note in Appendix B.1 explaining our findings in the Ant environment.

4. While the algorithm itself is not directly sensitive to the $\beta$ parameter, the constraint-adhering behavior of the final policy—i.e., the average frequency of constraint violations—is. By definition, $\beta$ specifies the risk level by setting the percentile of the worst-case outcomes over which the expected cost is computed. For instance, if we use a larger $\beta$, and assuming the dual and CVaR variables converge, we would generally observe more constraint violations, as the policy is optimized over a broader range of outcomes and is less focused on avoiding the worst-case scenarios.

New Experiments Added

Thank you for your participation in the discussion period! We are glad that your concerns have been addressed. In response to your feedback, we carried out experiments on SafetyNavigation tasks, which are now complete. Below, we present the experimental details and results numerically, in accordance with rebuttal guidelines.

---

Experimental Details

We have added three new tasks from the Navigation section of Safety-Gymnasium:

• Circle: The agent must move in a circular trajectory around the center while staying within boundaries. Exiting the circular area from the inside results in a cost of +1. Two walls are created by constraining the agent along the x-axis.
• Goal: The agent must navigate to the goal while avoiding 8 hazards. Each contact with a hazard incurs a cost of +1, with a maximum total cost of 8.
• Push: The agent must push a box to the goal location while avoiding 2 hazards. Each contact with a hazard incurs a cost of +1, with a maximum total cost of 2.

To manage time constraints, we used the point agent (with 12-dimensional state and 2-dimensional action space), since the constraint optimization task is independent of the agent’s body. Using a more complex agent would primarily affect the PPO solver’s efficiency, not our wrapper algorithm. This setup differs from Safety-Velocity, where the agent’s physical design (e.g., joints) directly impacts constraint violations such as velocity.

As in our original experiments, we introduced action noise to simulate uncertainty and highlight the risk-awareness capabilities of our algorithm.

Note: Constraints in these navigation tasks are inherently discrete, in contrast to the continuous constraint shaping in Safety-Velocity. As a result, dual and CVaR variable learning is less smooth, making the optimization problem inherently more challenging.

---

Results

The table below summarizes the converged episodic cumulative reward and cost, denoted by $R$ and $C$, respectively, for our algorithm and the unconstrained vanilla PPO. The vanilla PPO results are consistent with those reported in Table 5 of the Safety-Gymnasium paper.

• Our method is able to effectively minimize cost while maintaining high rewards, consistent with our findings in the Safety-Velocity experiments—even in the presence of discrete constraints.

• Notably, in the Push environment, our algorithm achieves higher rewards than vanilla PPO while maintaining zero cost. This is due to the fact that the vanilla PPO agent occasionally gets stuck on hazards and fails to reach the goal—suggesting that it struggles to find a global optimum in the absence of constraints and our algorithm works like a regularizer of the path.

• As expected from our Safety-Velocity experiments, reducing cost generally results in some reward degradation—highlighting the inherent trade-off in risk-constrained optimization.

---

Bottom Line

We will incorporate these results and the corresponding discussion into the camera-ready version, if the paper gets accepted.

We would also appreciate it if you could consider updating your score. We remain open to any further suggestions or feedback you may have.

Thank you for the careful review and thoughtful suggestions! We address the identified weaknesses and respond to your specific questions below.

---

Weaknesses

1. Please see our responses below.

2. We included some discussions of assumptions in Appendix A.6 and A.7 (for Assumptions 3.6 and 3.8, respectively). We will take care to add a brief discussion of Assumption 3.4 as well but in the main body to improve readability.

3. Please see our responses below.

---

Questions

1. We did not carefully or extensively tune any of the step sizes.

• PPO agent: We did not experiment with alternative step sizes for the PPO solver. We adopted a value previously found effective in our other projects using the same MuJoCo environments.

• CVaR and Dual Variable: For the CVaR and dual variables, we used a smaller learning rate relative to that of the networks (e.g., the policy), based on the common practice of using lower step sizes for scalar parameters due to their increased sensitivity to gradient noise and lack of averaging effects present in high-dimensional updates. We tested three magnitudes: $5 \times 10^{-4}$, $5 \times 10^{-5}$ (chosen), and $5 \times 10^{-6}$. The middle value provided better stability than the first and faster convergence than the last.

2. We will add additional references and relevant discussion on probabilistic (chance) constraints in safe RL. If the reviewer has specific examples of related work they believe are particularly relevant to our approach, we would be happy to consider and include them.

3. We primarily chose Safety-Velocity tasks because they are based on MuJoCo environments, which are widely used benchmarks in deep RL. These tasks are among the most realistic and challenging within the Safety-Gymnasium suite, involving high-dimensional state and action spaces, 3D spatial configurations, and complex physics dynamics. In contrast, many other tasks in the suite, such as those in Safe-Navigation and Safe-Vision, tend to be simpler—often limited to 2D or relying on lower-dimensional inputs.

That said, in response to your suggestion, we have begun experiments on Safety-Navigation. We did not include Safety-Vision due to the additional need for CNN-based image processing, which may not be feasible within the current timeline. If completed in time, we will include Safety-Navigation results in the camera-ready version.

4. Thank you for the suggestion! The subgradient in Line 5 of Algorithm 1 can be computed using Equation (8) along with the definition of $r'$ provided on line 149. Still, we will make sure to include this formulation explicitly in the paper for clarity.

Safe Navigation Results Added

Thank you once again for your constructive suggestions and for recognizing the merits of our work. As you had encouraged, we conducted additional experiments on the SafetyNavigation tasks in Safety-Gymnasium.

These experiments are now complete, and we would like to follow up by providing the details and numerical results below, in accordance with rebuttal guidelines that prohibit the use of external links, images, or plots.

---

Experimental Details

We have added three new tasks from the Navigation section of Safety-Gymnasium:

• Circle: The agent must move in a circular trajectory around the center while staying within boundaries. Exiting the circular area from the inside results in a cost of +1. Two walls are created by constraining the agent along the x-axis.
• Goal: The agent must navigate to the goal while avoiding 8 hazards. Each contact with a hazard incurs a cost of +1, with a maximum total cost of 8.
• Push: The agent must push a box to the goal location while avoiding 2 hazards. Each contact with a hazard incurs a cost of +1, with a maximum total cost of 2.

To manage time constraints, we used the point agent (with 12-dimensional state and 2-dimensional action space), since the constraint optimization task is independent of the agent’s body. Using a more complex agent would primarily affect the PPO solver’s efficiency, not our wrapper algorithm. This setup differs from Safety-Velocity, where the agent’s physical design (e.g., joints) directly impacts constraint violations such as velocity.

As in our original experiments, we introduced action noise to simulate uncertainty and highlight the risk-awareness capabilities of our algorithm.

Note: Constraints in these navigation tasks are inherently discrete, in contrast to the continuous constraint shaping in Safety-Velocity. As a result, dual and CVaR variable learning is less smooth, making the optimization problem inherently more challenging.

---

Results

The table below summarizes the converged episodic cumulative reward and cost, denoted by $R$ and $C$, respectively, for our algorithm and the unconstrained vanilla PPO. The vanilla PPO results are consistent with those reported in Table 5 of the Safety-Gymnasium paper.

• Our method is able to effectively minimize cost while maintaining high rewards, consistent with our findings in the Safety-Velocity experiments—even in the presence of discrete constraints.

• Notably, in the Push environment, our algorithm achieves higher rewards than vanilla PPO while maintaining zero cost. This is due to the fact that the vanilla PPO agent occasionally gets stuck on hazards and fails to reach the goal—suggesting that it struggles to find a global optimum in the absence of constraints and our algorithm works like a regularizer of the path.

• As expected from our Safety-Velocity experiments, reducing cost generally results in some reward degradation—highlighting the inherent trade-off in risk-constrained optimization.

---

We will include these results, along with the relevant discussion, in the camera-ready version if the paper is accepted.

Thank you for your questions and observations! We hope that we are able to address the points of weakness and related questions below.

---

Weaknesses

1. Thank you for pointing out these additional references. We will include these comparisons in the related work section.

   • Reference [1]: We cite the later work of Chow et al. (reference [12] in our paper), which addresses constraints, whereas [1] focuses on replacing the expectation in the objective with CVaR in a return-based formulation without constraints.

   • Reference [2]: It applies risk measures to constraints in the return-based setting and appears to focus on a risk-neutral objective with a CVaR constraint (see Eq. (11) in [2]).

   • Reference [3]: It proposes solving CMDPs via a local policy search algorithm, where the constraint enforces proximity to the previous policy iterate using a distance metric. In contrast, our method is not based on local policy search and does not impose constraints based on policy distance.

2. Our goal is to position this work primarily as a theoretical contribution, with the following key strengths:

   • a parameterized strong duality,
   • an interesting interplay of convex analytic and Lagrangian duality,
   • a modular algorithmic framework that accommodates a broad class of OCEs,
   • and the ability to handle per-step cost CVaR constraints, which differs from most prior work that focuses on return-based CVaR.

Notably, strong duality is a property that does not hold for many constrained problems, which are often solved via relaxation. We provide a complete theoretical treatment, including a novel convergence analysis and detailed discussion of assumptions and their practical relevance (see Appendices A.1, A.6, and A.7). In this regard, our approach aligns with prior theoretical works such as the given reference [1], which similarly emphasizes rigorous analysis over extensive empirical comparisons.

That said, we acknowledge the importance of this trade-off and will aim to include additional experiments and return-based baselines, if time permits and the paper is accepted.

---

Questions

1. The PPO solver employs a stochastic policy, modeled as a Gaussian distribution with learnable mean and standard deviation, as noted on line 727. Importantly, our framework does not require the final policy to be deterministic. As you pointed out, recovering the true stochastic optimum of the original CMDP requires maintaining stochasticity, which our algorithm preserves through the use of such policies.

We will add 1–2 clarifying sentences to the paragraph starting on line 281 to make this point explicit.

2. In the min-max optimization literature, theoretical guarantees are typically established for average or randomly selected iterates, as last-iterate convergence is generally difficult to prove without additional structural assumptions. Uniform sampling of an iterate is a standard approach in nonconvex optimization, including for gradient descent and stochastic gradient descent, to ensure convergence to stationary points. In practice, one usually returns the iterate when the learning curve stops changing significantly.

That said, in our practical implementation—given the use of a deep reinforcement learning solver in high-dimensional environments—we return the final policy observed when the learning curve stabilizes. This is noted on line 235 in our wrapper code (wrapper > RA_C_RL.py).

Thank you for bringing this observation; we will clarify this point with 1–2 sentences in the text.

Choice of Per-Step Cost CVaR

Further elaborating on the choice of per-step cost CVaR (as opposed to the return-based like most other works), we would like to point out that a general comparison of return-based and reward-based risk measures in the context of RL has been done in Bonetti et al. [7]. Our work explores the choice of using reward-based risk measures in both objective and constraints (or even a combination of risk-neutral and risk-aware) with theoretical guarantees, as solving constrained problems is more difficult (theoretically and empirically). We will take care to highlight these existing comparisons of the per-step cost risk measure and the return-based counterpart (citing Bonetti et al. [7]) in the text to include these critical trade-off discussions.

---

New Experiments Added

Moreover, we conducted additional experiments on the SafetyNavigation tasks in Safety-Gymnasium and compared them to a PPO baseline (which was the method of comparison for the return-based Chow et al. [12]). These experiments are now complete, and we provide the details and numerical results below, in compliance with rebuttal guidelines that restrict the use of external links, images, or plots.

We have added three new tasks from the Navigation section of Safety-Gymnasium:

• Circle: The agent must move in a circular trajectory around the center while staying within boundaries. Exiting the circular area from the inside results in a cost of +1. Two walls are created by constraining the agent along the x-axis.
• Goal: The agent must navigate to the goal while avoiding 8 hazards. Each contact with a hazard incurs a cost of +1, with a maximum total cost of 8.
• Push: The agent must push a box to the goal location while avoiding 2 hazards. Each contact with a hazard incurs a cost of +1, with a maximum total cost of 2.

To manage time constraints, we used the point agent (with 12-dimensional state and 2-dimensional action space), since the constraint optimization task is independent of the agent’s body. Using a more complex agent would primarily affect the PPO solver’s efficiency, not our wrapper algorithm. This setup differs from Safety-Velocity, where the agent’s physical design (e.g., joints) directly impacts constraint violations such as velocity.

As in our original experiments, we introduced action noise to simulate uncertainty and highlight the risk-awareness capabilities of our algorithm.

Note: Constraints in these navigation tasks are inherently discrete, in contrast to the continuous constraint shaping in Safety-Velocity. As a result, dual and CVaR variable learning is less smooth, making the optimization problem inherently more challenging.

---

Results

The table below summarizes the converged episodic cumulative reward and cost, denoted by $R$ and $C$, respectively, for our algorithm and the unconstrained vanilla PPO. The vanilla PPO results are consistent with those reported in Table 5 of the Safety-Gymnasium paper.

• Our method is able to effectively minimize cost while maintaining high rewards, consistent with our findings in the Safety-Velocity experiments—even in the presence of discrete constraints.

• Notably, in the Push environment, our algorithm achieves higher rewards than vanilla PPO while maintaining zero cost. This is due to the fact that the vanilla PPO agent occasionally gets stuck on hazards and fails to reach the goal—suggesting that it struggles to find a global optimum in the absence of constraints and our algorithm works like a regularizer of the path.

• As expected from our Safety-Velocity experiments, reducing cost generally results in some reward degradation—highlighting the inherent trade-off in risk-constrained optimization.

We will include these results, along with the relevant discussion if the paper is accepted.