Safety-Prioritizing Curricula for Constrained Reinforcement Learning

Q3: What is the reasoning behind baselines other than CURROT?

DEFAULT allows us to understand whether a curriculum learning approach has benefits in terms of safety during training or optimality at test time.

CURROT4Cost replaces CURROT's performance constraint with a cost constraint. Therefore, we evaluate whether the advantages of SCG can be obtained via a curriculum learning approach that only cares about safety.

NaiveSafeCURROT penalizes the reward with cost instead of return in the performance constraint of CURROT. NaiveSafeCURROT enables us to investigate whether a naive attempt at modifying CURROT to be safe is sufficient.

SPDL is also an interpolation-based curriculum learning approach that poses curriculum generation as a constrained optimization problem. Compared to CURROT, SPDL minimizes the KL divergence from the target context distribution, and its performance constraint is based on the expected return with respect to the subsequent context distribution. SPDL generates parametric context distributions and does not provide the robustness CURROT achieves via a performance constraint on individual constraints. Therefore, we can highlight why CURROT is a superior algorithm for building SCG.

PLR is a popular curriculum learning approach, yet it is unsuitable for continuous context spaces. It assumes that contexts are discrete and unidimensional and that the target distribution is a uniform context distribution.

ALP-GMM and GoalGAN are among the first automated curriculum generation approaches. They generate a curriculum as a sequence of distributions, similar to CURROT and SPDL, yet they do not interpolate them via Wasserstein distance or KL divergence. Similar to PLR, they assume uniform target context distributions.

Q4: What do the box plots indicate?

We thank the reviewer for their attention to detail. We accidentally commented on this sentence in the first submission. We added the following sentence to the caption of Figure 4:

Box plots show the minimum, the first quartile, the median, the third quartile, and the maximum, from bottom to top.

We would like to thank the reviewer for their thought-provoking comments and questions, which helped us improve our work in the rebuttal phase. Below, we respond to their comments regarding the weaknesses and then their questions.

W1: Lack of reasoning for design insights

We agree with the reviewer that our work could help other researchers by clarifying why we develop SCG on top of CURROT. To this aim, we added the sentences below to Section 4.1:

We investigate CURROT due to its favorable properties 1) It poses curriculum generation as a constrained optimization problem, allowing for a natural extension to CCRL (1) 2) Interpolating context distributions based on Wasserstein distance enables non-parametric distributions. 3) The performance constraint enforces a desired return level in any context, hence robustness.

Three novel algorithmic features differentiate SCG from CURROT:

• Cost constraint in the curriculum update in Eq. (4) directly accounts for safety in any context.
• Safety-prioritization via generating source distributions over safe contexts in Eq. (5) enables focusing on low-cost contexts early during training to avoid high constraint violation regret.
• Annealing mechanism to tune the prioritization ratio of safe or performant contexts in Eq. (5) and (6) allows for a smooth transition between safety and performance prioritization phases.

We also made changes in Section 5 to highlight the motivation behind these algorithmic differences.

W2: Lack of explanation of assumptions and motivations

We address this section in the corresponding questions.

Q1: What are the motivations behind the environments of interest?

We study these environments for the following reasons:

• Highlighting the misalignment problem between curriculum learning and constrained RL: In all domains, there is a trade-off between unsafe yet short (high return and high cost) paths over hazards and safe yet long (low return and low cost) paths over clear zones.
• Safety-Maze is a cost-constrained version of Maze, a domain studied in the paper that developed CURROT. We choose Maze because it is a simple domain, yet the cost modification to present the misalignment problem makes CURROT an unsafe curriculum learning approach.
• Safety-Goal and Safety-Maze are navigation tasks with realistic sensory observations in Safety-Gymnasium, a framework extensively used for constrained RL.

The second paragraph in Section 6 explains the challenges introduced in Safety Push, such as the need for multi-modal context distributions or the dead-end causing high constraint violation regret.

In addition, these environments differ from each other in terms of the following aspects:

• Dimensionality of state space: 2D (coordinates) in Safety-Maze, whereas 72D and 88D (LIDAR sensor outputs) in Safety-Goal and Safety-Push,
• Complexity of dynamics: Safety-Maze has easy dynamics, yet the cars in Safety-Goal and Safety-Push need to maneuver around hazards and pillars,
• Types of reward functions: Safety-Maze has a sparse reward function (1 for reaching the goal and zero for all other transitions), whereas Safety-Goal and Safety-Push utilize the displacement-based reward functions in Safety-Gymnasium.
• Objects to interact with: In Safety-Push, the car must learn to attach its front end to the box at the right angle and then push the box around pillars to the goal.

Q2: Does DEFAULT not generate a curriculum?

Yes, DEFAULT directly draws context from the target context distribution and runs a constrained RL algorithm, specifically PPO-Lagrangian, without generating a curriculum. We made a minor clarification in Section 6 to address this question.

We would like to thank the reviewer for their fair criticism and helpful questions that allowed us to improve the presentation of our work.

Below, we address the reviewer's comments regarding weaknesses first, then their questions:

W1: Limited novelty.

SCG not only adds a cost constraint to CURROT but also has two other features to achieve safer training and optimality at test time, as our ablation study in Section 6.3 indicates.

SCG's three novel features built on top of CURROT:

• Cost constraint in the curriculum update in Eq. (4).
• Safety-prioritization via generating source distributions over safe contexts in Eq. (5).
• Annealing mechanism to tune prioritization ratio of safe or performant contexts in Eq. (5) and (6).

Our ablation results in Figure 6 showcase that all these features result in a well-balanced, safe curriculum learning approach. Simply adding the cost constraint without safety-prioritization (SCG-NoPS and SCG-NoPPPS) may make the training safer than CURROT (see the bottom row of Figure 4a) but does not achieve optimality, as SCG-NoPS fails to yield zero cost in multiple runs and SCG-NoPPPS cannot achieve 100% success. Furthermore, not having annealing (SCG-NoAnn) causes higher CV regret and a lower success rate than SCG with all three features (see Figures 6a and 6c).

W2: Empirical limitations.

As indicated by the last sentence before Section 6.1, our empirical analysis involves a comparison against constrained RL algorithms that do not use curriculum learning. Appendix F.3. demonstrates an empirical comparison of SCG against PPO-Lagrangian, Constrained Policy Optimization (CPO), Projection-based CPO (PCPO), and First-Order Constrained Optimization in Policy Space (FOCOPS) in Safety-Push. Figure 14 evidences that, similar to DEFAULT, which uses PPO-Lagrangian without curriculum generation, CPO, PCPO, and FOCOPS cause high constraint violation regret during training and fail to yield optimal policies in multiple training runs.

Following the suggestion of the reviewer about sensitivity analysis to parameters of SCG, our general response includes an ablation study on three parameters:

• Target Gaussian mixture model weight ratio $\alpha$, i.e, the target for annealing $\alpha_k$,
• Number of annealing steps $K_{ann}$,
• Cost threshold $\tilde{D}$ in SCG update

Q1: Cost annealing in the Naive approach

No, there is no annealing in NaiveSafeCURROT because NaiveSafeCURROT does not have separate safety and performance prioritization phases. NaiveSafeCURROT extends CURROT by penalizing rewards with costs.

Q2: How does the proposed approach compare to other safe-RL algorithms in these domains?

As our response to W2 indicates, Appendix F.3 compares SCG against constrained RL algorithms. In summary, without curriculum generation, CPO, PCPO and FOCOPS cause high constraint violation regret during training and fail to yield optimal policies in multiple training runs.

In our rebuttal, we addressed the reviewer's comments on

• novelty by describing the algorithmic contributions of SCG and
• empirical limitations via a new ablation study and clarifying that we have already compared SCG with other safe-RL baselines.

We hope we addressed all their comments/questions sufficiently well and would appreciate it if the reviewer could point out any further comments/questions. Otherwise, we kindly ask them to consider revising their score accordingly.

Q1: A proper baseline should use a goal that cares about safety and performance. The score can be a weighted sum of both scores.

One of the baselines we evaluate, NaiveSafeCURROT, already modifies CURROT by combining performance and safety into a single reward function to generate curricula, as explained in the paragraph before Section 6.1. Our experiments indicate that SCG reaches higher success rates than NaiveSafeCURROT in all environments, whereas NaiveSafeCURROT fails to yield optimal policies in multiple runs in all environments. At the same time, SCG yields low constraint violation. Therefore, we conclude that treating reward and cost separately results in better performance and safer training.

Furthermore, it is already commonly accepted in the safe RL literature that simply combining the costs and rewards into a single reward function does not lead to safe behavior in many cases [2,3,4].

Q2: Have you thought about the possibility of making more mistakes first and learning those mistakes can avoid future mistakes

The ablation study in Section 6.3 responds to this question by evaluating SCG-NoPS, SCG without a safety prioritization phase. SCG-NoPS skips Phase 1 and directly looks for performant contexts at the expense of collecting cost, violating the cost constraint. Figure 6 indicates that SCG-NoPS not only yields higher CV regret but also fails to yield safe policies (zero cost in target contexts) in multiple runs.

• References
  1. Klink, P., Yang, H., D’Eramo, C., Peters, J., & Pajarinen, J. (2022, June). Curriculum reinforcement learning via constrained optimal transport. In International Conference on Machine Learning (pp. 11341-11358). PMLR.
  2. Ray, A., Achiam, J., and Amodei, D. (2019). Benchmarking safe exploration in deep reinforcement learning.
  3. Roy, J., Girgis, R., Romoff, J., Bacon, P., and Pal, C. J. (2022). Direct behavior specification via constrained reinforcement learning. ICML, 18828–18843.
  4. Kamran, D., Simão, T. D., Yang, Q., Ponnambalam, C. T., Fischer, J., Spaan, M. T. J., and Lauer, M. (2022). A modern perspective on safe automated driving for different traffic dynamics using constrained reinforcement learning. ITSC, 4017–4023.

We thank the reviewer for taking the time to write a critique of our work. Below, we address their comments and questions.

W1: 'Simple' environments and lack of comparison in previously studied domains

We would like to point out that the environments were carefully chosen to be challenging for constrained RL agents. Notice, for instance, that the Default approach, which does not use a curriculum, often fails to solve the underlying task (see Fig 4.c in the paper).

Furthermore, the tasks considered have a clear misalignment between the reward and costs. They force the agents to find a tradeoff between the reward and cost to satisfy the constraints, such that simply maximizing the reward does not return a policy that satisfies the constraints. From this perspective, the Bipedal Walker Stump is not interesting for our work, as the cost is already associated with falling over, and to maximize the return, the agent should avoid falling over anyway.

Furthermore, we would like to show that the environments studied are not 'simple', as they present varying challenges:

• Dimensionality of state space: 2D (coordinates) in Safety-Maze, whereas 72D and 88D (LIDAR sensor outputs) in Safety-Goal and Safety-Push,
• Complexity of dynamics: Safety-Maze has easy dynamics, yet the cars in Safety-Goal and Safety-Push need to maneuver around hazards and pillars,
• Types of reward functions: Safety-Maze has a sparse reward function (1 for reaching the goal, and zero for all other transitions), whereas Safety-Goal and Safety-Push utilize the displacement-based reward functions in Safety-Gymnasium.
• Objects to interact with: In Safety-Push, the car needs to learn that it needs to attach its front end to the box in the right angle and then push the box around pillars to the goal.

We do not study Point Mass [1], because we already modified (Safety-Maze) another environment in [1] called Maze, which has similar dynamics (2D observation and action spaces) and objectives (pushing a point mass to a goal). Regarding Bipedal Walker Stump Tracks, we argue that this environment is not necessarily more challenging than Safety-Goal and Safety-Push, as the observation space is only 24-dimensional.

W2: 'Trivial' method

We strongly disagree with this comment. While SCG is based on a similar logic from CURROT, it has three novel features to achieve safer training and optimality at test time:

1. Cost constraint in the curriculum update in Eq. (4).
2. Safety-prioritization via generating source distributions over safe contexts in Eq. (5).
3. Annealing coefficient to tune prioritization ratio of safe or performant contexts in Eq. (5) and (6).

Our ablation results in Figure 6 showcase that all these features result in a well-balanced, safe curriculum learning approach. Simply adding the cost constraint without safety-prioritization phase (SCG-NoPS and SCG-NoPPPS) may make the training safer than CURROT (see the bottom row of Figure 4a), but does not achieve optimality, as SCG-NoPS fails to yield zero cost in multiple runs and SCG-NoPPPS cannot achieve 100% success. Furthermore, not having annealing (SCG-NoAnn) causes higher CV regret and a lower success rate than SCG with all three features (see Figures 6a and 6c).

W3: Many other curricular learning methods have different settings (?)

It is unclear to us why the reviewer considers this as a weakness. What settings does the reviewer refer to? We would like to ask the reviewer to clarify their comment for us to respond to them.

In our rebuttal, we have addressed the reviewer's concerns over the contributions of our work and the environments experimented with. Our global response provides an additional ablation study to strengthen our claims and describes minor clarifications/changes in our submission to address the reviewer's concerns and improve the presentation.

We hope our rebuttal and the new version of our submission have sufficiently addressed the reviewer's comments and questions. We are happy to address any further comments/questions the reviewer has during the discussion phase. If not, we kindly request the reviewer to consider revising their score, as one of the reviewers changed their score towards acceptance upon reading our rebuttal.

W5: Lines 188 and 193 do not completely describe the objective.

We want to thank the reviewer for this helpful suggestion. We made a minor change in the problem statement section to mention the misalignment phenomenon before going into detail in Section 4.2.

Q1: What is the main limitation to applying this approach to Brax environments?

Our response to the reviewer's comments explains why applying CURROT-like algorithms to Brax environments may not be a major limitation. We also describe the reasoning behind choosing our environments of interest.

Nevertheless, we want to acknowledge that evaluation in complex robotics domains is a common missing piece in curriculum learning literature. We are working on addressing this evaluation gap in our current projects.

• References
  1. Klink, P., Yang, H., D’Eramo, C., Peters, J., & Pajarinen, J. (2022, June). Curriculum reinforcement learning via constrained optimal transport. In International Conference on Machine Learning (pp. 11341-11358). PMLR.
  2. Huang, P., Xu, M., Zhu, J., Shi, L., Fang, F., & Zhao, D. (2022). Curriculum reinforcement learning using optimal transport via gradual domain adaptation. Advances in Neural Information Processing Systems, 35, 10656-10670.
  3. Ji, J., Zhang, B., Zhou, J., Pan, X., Huang, W., Sun, R., ... & Yang, Y. (2023). Safety gymnasium: A unified safe reinforcement learning benchmark. Advances in Neural Information Processing Systems, 36.
  4. Feydy, J. and Roussillon, P. Geomloss, 2019. URL https://www.kernel-operations.io/ geomloss/index.html.
  5. Bonneel, N., Rabin, J., Peyré, G., & Pfister, H. (2015). Sliced and radon wasserstein barycenters of measures. Journal of Mathematical Imaging and Vision, 51, 22-45.
  6. Kolouri, S., Nadjahi, K., Simsekli, U., Badeau, R., & Rohde, G. (2019). Generalized sliced wasserstein distances. Advances in neural information processing systems, 32.

We would like to thank the reviewer for their positive comments, valuable criticism, and thought-provoking questions that helped us improve the presentation of our work.

We first respond to the reviewer's comments and then their question.

W1: Lack of a demonstration of scalability

Indeed, we study low-dimensional context spaces, specifically three-dimensional contexts, that determine the position and tolerance of goals. As the reviewer indicates, this is a common point of interest in the existing automated curriculum generation work, such as CURROT[1] and GRADIENT[2]. The main reason behind this is that off-the-shelf RL algorithms can fail even in multi-task settings where contexts are low-dimensional. An interesting open question in curriculum learning is how to address a high-dimensional context space, e.g., a space comprised of goal images [2].

We consider the scalability of curriculum generation to higher-dimensional context spaces orthogonal to the safety curriculum generation. We believe that techniques such as SCG will benefit from novel curriculum generation methods that scale to larger context spaces.

We also want to note that Safety-Goal and Safety-Push are not two-dimensional navigation tasks. The agent observes 72D and 88D observations comprised of LIDAR outputs for surrounding objects.

W2: Gains on both safety and performance are not as dramatic.

Could the reviewer clarify what they mean by 'dramatic' if possible?

Table 1 in Section 6.1. highlights that SCG is the only approach that yields zero cost in target contexts (Fig. 4b), hence satisfies the cost constraint in the CCRL objective while achieving the highest success rate in all domains (Fig. 4c). Simultaneously, SCG achieves the lowest constraint violation regret (Fig. 4a) among approaches that achieve zero cost and highest success. These results evidence that failure to address the misalignment phenomenon, regardless of the domain complexity, causes suboptimal policies or optimal policies at the expense of high constraint violation regret.

W2: Applicability to robotics domains

We agree that studying domains consisting of complex rigid bodies in Brax would further strengthen our empirical analysis. However, we would like to highlight that the studied domains in this paper are not necessarily simple and unrealistic. They differ in terms of the following aspects:

• Dimensionality of observation spaces: 2D (coordinates) in Safety-Maze, whereas 72D and 88D (LIDAR sensor outputs) in Safety-Goal and Safety-Push,
• Complexity of dynamics: Safety-Maze has easy dynamics, yet the cars in Safety-Goal and Safety-Push need to maneuver around hazards and pillars,
• Types of reward functions: Safety-Maze has a sparse reward function (1 for reaching the goal and zero for all other transitions), whereas Safety-Goal and Safety-Push utilize the displacement-based reward functions in Safety-Gymnasium.
• Objects to interact with: In Safety-Push, the car must learn to attach its front end to the box at the right angle and then push the box around pillars to the goal.

We study Safety-Maze to showcase that a simple modification to an existing domain in curriculum learning can cause the misalignment phenomenon. As the domain complexity increases in Safety-Goal and Safety-Push, existing baselines and naive attempts to make them safe fail to yield optimal policies and provide safety during training. Safety-Goal and Safety-Push are navigation tasks with realistic sensory observations in Safety-Gymnasium [3], a framework extensively used for constrained RL.

Although some rigid body domains in Brax have higher dimensional observation and action spaces, this does not necessarily mean that CURROT-like approaches cannot scale to them, as observation/action spaces differ from context spaces. In case one is interested in a high dimensional context space, where, for example, goals are indicated by positions/angles of tens of joints, then this would indeed increase the computational burden of the curriculum update in Eq. (4). Nevertheless, CURROT-like approaches can still address such settings. In Appendix A, the CURROT paper [1] describes how the library they utilize in their implementation can solve the optimal transport problem for sets of hundreds of thousands of samples in seconds on high-end GPUs [4]. More specifically, approximate solutions lower the complexity $\mathcal{O}(n^3)$ to $\mathcal{O}(Mn\log n)$ where $n$ is the dimensionality of a context space and $M\ll n$ [5,6].

W4: Particle colors in Figures 3 and 5c (right) are hard to interpret.

In our global response, we have provided new curriculum progression plots that use varying dimensions instead of varying colors for goal tolerances, as requested by the reviewer.

We thank the reviewer for initiating a discussion on the scalability of our proposed approach and its applicability to robotics domains, as well as for helping us improve the presentation of our work. As the end of the discussion period approaches, we would like to ask the reviewer if they have any remaining questions.

I thank the authors for their thoughtful rebuttal, and clarifying figures. I agree with most of the response and do not have any remaining questions. Though I believe this paper to be correct, I find it difficult to recommend it more strongly without an empirical demonstration of scalability. Thus I will, unfortunately, be leaving my score unchanged.