Embedding Safety into RL: A New Take on Trust Region Methods

How sensitive is C-TRPO to the accuracy of the estimated cost function? Could you provide an analysis or experiments showing how performance degrades with increasingly noisy or misspecified constraints?

This is an excellent question. While we do not yet have a comprehensive theoretical analysis, we are running experiments to assess C-TRPO’s robustness to noisy or misspecified cost function estimates and will report results in the updated appendix.

How does C-TRPO handle multiple, potentially conflicting constraints? Could you provide theoretical insights or empirical results for scenarios with multiple safety criteria?

C-TRPO naturally extends to multiple constraints. The benchmark environments we consider already include multiple cost signals, which are aggregated into a single constrained objective. While our current experiments adhere to this setup, it is possible to modify the benchmark tasks to enforce separate constraints on individual cost functions.

We will expand our discussion on handling multiple constraints in the final version of the paper. Theorem 4.5 already implies that the optimal constrained policy $\pi^*_{safe}$ found by C-NPG satisfies as few constraints with equality as required to be optimal, which is discussed in the respective paragraph. We expect similar results to hold for C-TRPO as well.

Additionally, we recognize the importance of empirical evaluation in scenarios with distinct, potentially conflicting constraints, and we consider this an important direction for future work.

Thank you for your detailed comments and suggestions! We respond to each of your points in detail below.

I had a problem understanding the results depicted in figure 3. Why are the results aggregated together across multiple tasks? How can we compare the average of normalized rewards and constraint violations across 8 tasks together?

We follow a standard procedure from unconstrained RL to provide a concise summary of results, adopting the methodology from rliable:

• Each algorithm is trained across multiple seeds per environment.

• Performance is normalized per environment relative to a reference algorithm.

• Normalized results are then aggregated across seeds and tasks to enable a high-level comparison.

To ensure robustness, we use interquartile mean aggregation and report bootstrapped confidence intervals, following best practices in RL evaluation. While this approach provides an informative summary, we acknowledge that individual task-level variations are important, which is why we also include per-task sample efficiency curves in the appendix.

The barrier function over state distributions is a clever way of inducing such a safe trust region.

Thank you for the positive assessment!

C-TRPO inherits methodology from CPO’s hysteresis idea that allows for safe policy updates.

We would like to clarify that hysteresis was introduced in our work, not in the original CPO paper. However, our results show that applying hysteresis to CPO improves its performance, as reported in our ablation studies. To achieve the full performance, C-TRPO’s safe trust region is necessary.

On line 233 (left column), the paper discusses divergence before it is formally defined (Equations 16 and 17).

Thank you for this suggestion! There are two key divergences:

• The theoretical divergence $D_C$ (Equation 10).
• The approximated divergence $\bar{D}_C$ (Equation 16).

We will clarify this distinction earlier in the manuscript.

Algorithm 1 should be moved after the full methodology description to improve readability.

We agree and will restructure the manuscript accordingly.

Question regarding Theorem 4.4, Isn't CPO also safety invariant? It would be sufficient to show the CPO is invariant and then, C-NPG with any choice of beta would be invariant too.

Neither CPO nor C-TRPO are strictly safety-invariant as defined in Theorem 4.4 due to approximation and estimation errors. However, Proposition B.2 shows that in the limit of small step sizes, C-TRPO converges to the safety-invariant C-NPG.

In contrast, CPO does not follow a natural gradient update, making its small-stepsize limit unclear. While a bound similar to Proposition 4.3 exists for CPO, it is less conservative than C-TRPO’s. Importantly, both methods provide only a bounded approximation of the ideal safety invariance property of C-NPG, even with perfect value function knowledge. When value functions are estimated from finite samples, these bounds are further affected.

In practice, C-TRPO offers a more conservative safety guarantee with minimal computational overhead or performance loss. Since estimation errors also impact safety, we plan to explore the finite-sample safety properties of both methods in future work.

I feel the property that needs to be more emphasized in the text is the conservativeness of the policy update under higher beta values w.r.t. CPO.

Thank you for this suggestion. We will highlight this aspect in the manuscript.

Thank you for your valuable feedback! We have addressed your specific comments in detail below.

Log barrier baselines are not considered in the experiment.

We considered IPO as a practical log-barrier baseline, along with P3O, a proximal adaptation.

Regarding the specific works mentioned:

• LB-SGD (Usmanova et al.) is discussed in our related work section. However, our focus is on practical deep RL algorithms, whereas LB-SGD primarily offers theoretical insights. Given the lack of large-scale empirical evaluations and implementation details for deep function approximation in LB-SGD, we did not include it as a baseline in our experiments.

• Saute RL presents an interesting comparison but tackles a subtly different problem formulation than standard CMDPs. As seen in Definition 4 of their work, Saute RL imposes stricter constraints by disallowing stochastic policies from occasionally violating constraints and assuming non-negative safety costs. While this formulation is reasonable for safety, it does not always correspond to a general CMDP. A key strength of C-TRPO is its reliability in solving standard CMDPs, making it applicable to broader tasks like diverse policy optimization (e.g., [1]). A rigorous theoretical and empirical comparison would require additional work beyond the current scope. However, we will mention Saute RL in the manuscript and highlight its comparison with C-TRPO as an important direction for future research.

• ActSafe is a newly accepted work that we were not previously aware of. Since it falls under model-based safe RL, which we briefly discuss in our related work section, we will add a reference in the final manuscript. However, we want to highlight key differences from our approach. Specifically, in Definition 4.7, ActSafe indeed defines a distance measure between policies, which is informed by cost continuity. In contrast, our Bregman divergence explicitly captures differences in the cost values of the policies, representing a key conceptual distinction in how safety is enforced.

[1] Zahavy, Tom, et al. "Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality." The Eleventh International Conference on Learning Representations.

I am not sure about analysing cost-regret with trust region methods given that they are rarely applied for learning in the real world due to sample inefficiency

This is an important point. While near on-policy approaches like TRPO tend to be sample inefficient, they remain valuable due to their stability and well-established theoretical foundations in the function approximation setting. While they may not be ideal for online real-world learning, understanding how to minimize cumulative constraint violation regret is essential, as any algorithm used for fine-tuning on a real system must consider it.

We believe that the insights from C-TRPO’s regret analysis will contribute to the development of future algorithms that are both sample-efficient and safety-aware.

Thank you for your positive assessment of our work!

We agree that a regret analysis would further strengthen the theoretical contribution and see this as a valuable direction for follow-up work.