Thank you for valuable questions and commentaries. We provide detailed answers for them below.

Weaknesses:

• Comparison ITES with Safe HIRO and IAHRL.

Comparison ITES and Safe HIRO:

The Safe HIRO algorithm consists of two modules: a low-level learnable policy and a high-level HIRO policy. To ensure safety, Safe HIRO employs a Safety Layer, which substitutes each greedy action of the low-level policy with a safe action. The first distinction between Safe HIRO and ITES is that Safe HIRO only focuses on the safety of the controller's actions, without considering the safety of the high-level goal, which can result in the agent entering unsafe areas. In contrast, ITES addresses safety concerns at both levels, ensuring that we always remain in safe regions. The second distinction is that ITES utilizes a world model, which prevents the agent from reaching dangerous states and allows for the maximization of safety in imagination. This approach enhances the sample efficiency of the algorithm compared to Safe HIRO.

Comparison ITES and IAHRL:

The IAHRL algorithm consists of a high-level policy (learned via the RL algorithm SAC) and a set of low-level controllers (each of which is a predefined Frenet planner). In contrast, ITES employs a single learnable low-level policy that is conditioned on arbitrary goals within the environment.The imagination-based safety in ITES relies on a learnable world model, which is continuously updated with newly acquired experience and is applicable to any task. In IAHRL, safety is hardcoded into the planner, which is specifically designed for autonomous driving tasks In contrast, ITES does not have such limitations and can be utilized across a variety of environments. Additionally, subgoal safety in ITES is defined using a learnable Cost Model, which incorporates a safety component directly into the loss function of the high-level policy. In IAHRL, safety is implemented as a component of the reward, this leads to the well-known problem of reward engineering. While ITES addresses a more general formulation of CMDP, where the issue of reward engineering does not arise.

• We completely agree that the quality and confidence of the world model are crucial for safety-critical tasks. We recognized this from the outset and designed ITES with this consideration in mind. To minimize model uncertainty in ITES, we employ a classic strategy for reducing world model errors—learning an ensemble of world models. Additionally, our cost model is trained as a probabilistic classifier, providing the probability that a given state is safe or unsafe.

Questions:

• The safe region is implicitly defined by our learnable cost model: ITES learns this cost model from data generated through interactions with the environment. As the cost model is updated, the agent's understanding of the safe region is also refined. We do not use any fixed threshold to distinguish between two classes of states. Instead, ITES aims to minimize this probability to zero as part of the optimization process (we acknowledge in the conclusion section that this is one of ITES's limitations).
• The distinction between ITES and Safe HIRO, as well as IAHRL, is clearly outlined above (in the first point of the responses to the weaknesses).
• We consider the transfer of proposed ITES to real-world robotic applications as our future research. About unexpected changes: as our proposed algorithm is learnable it will adapt to changes in an environment, however to make possible real-world applications we should take into account that during the adaptation the agent policies become more unsafe.

Thank you for your valuable feedback and constructive suggestions.

Weaknesses:

• You are correct that in the PointGoal1 environment, our method exhibits a greater trade-off towards performance rather than safety. We propose that in this environment, it is impossible to optimize performance without sacrificing safety. To support this hypothesis, we investigated an additional baseline TD3LAG which demonstrates higher performance (15.75) compared to CUP (reward = 6.14 ± 6.97) and FOCOPS (reward = 5.23 ± 10.76), albeit at a significant cost to safety (cost = 56.22 ± 0.85) when compared to our method ITES (cost = 40.46 ± 0.12). Although our method generally exhibits lower safety than two out of three baselines in the PointGoal1 environment, it is more reliable in terms of variance: CUP (cost_std = 89.56), FOCOPS (cost_std = 93.51), TD3LAG (cost_std = 9.45), ITES (cost_std = 1.04). The substantial variance observed in CUP and FOCOPS indicates that there are seeds (random conditions) under which they violate safety constraints excessively, whereas our algorithm displays a more stable safety. To ensure that our method provides a favorable trade-off towards safety in more safety-critical environments, we recommend utilizing hyperparameters that address safety specifically, the “Subgoal Safety Coefficient” and “Controller Safety Coefficient”, as outlined in Appendix Table 4. | Env | Method | Final Reward | Final Cost | | ------------------- | ------------- | ---------------- | ---------------- | | PointGoal1| ITES | $21.42 \pm 1.14$| $40.46 \pm 0.12$ | | PointGoal1 | CUP | $14.42 \pm 6.74$ | $19.02 \pm 20.08$ | | PointGoal1 | FOCOPS | $14.97 \pm 9.01$ | $33.72 \pm 42.24$ | | PointGoal1 | TD3LAG | $15.75 \pm 0.39$ | $56.22 \pm 0.85$ |

• Thank you for your suggestion to improve the generalizability of our algorithm. To expand the range of complex environments, we have added SafeAntMaze W shape, an additional map for the SafeAntMaze environment that requires more steps to solve the task, and we have tested it with (ITES, HRAC, HRACLAG, TD3LAG). We have also introduced the SafetyGymPointGoal sparse environment, which consists of the same tasks as PointGoal1, but where a reward of +1 is granted only for reaching the goal, significantly increasing the difficulty of learning. In both of the added environments, model-free single-policy methods are unable to solve the tasks, highlighting the need for a hierarchical approach. | Env | Method | Final Reward | Final Cost | | ------------------- | ------------- | ---------------- | ---------------- | | PointGoal sparse | ITES | 8.64 ± 0.38| 33.455 ± 1.47 | | PointGoal sparse | MBPPOL | 6.15 ± 0.15 | 34.1 ± 3.6 | | PointGoal sparse | SAC-L| 0.675 ±0.08 | 64.8 ± 3.8 | | PointGoal sparse | TD3-L | 0.07 ± 0.006 | 64.7 ± 1.91 |

• Yes, you are correct that ITES demonstrates an advantage in terms of reward but performs worse in terms of safety in the SafetyGymPointGoal1 and SafetyGymCarGoal1 environments due to the trade-off favoring performance. We address this issue in the first weakness discussion.

• Yes, you are correct that ITES complicates the approach because it requires the training of an additional high-level policy. However, our results across all benchmarks demonstrate that ITES outperforms all single-policy methods in terms of performance and exhibits less variance in safety, as shown in Table 1 and Table 2. Moreover, when considering more complex environments such as AntMazeCshape, AntMazeWshape, and SafetyGymPointGoal1, our method does not compromise on safety.

Questions:

• We conducted additional experiments in the SafeAntMazeWshape and PointGoal sparse environments.
• In our experiments we obtained that ITES sacrifices some safety in short-horizon (SafetyGym) tasks in favor of performance. We provided results for the case where the agent have the highest performance while minimazing cost violations. During our research experiments we observed that increasing the role of safety reduces the performance, but even there it is not zero. Did we understand your question correctly?
• Since ITES outperforms single-policy algorithms in simpler environments, we interpret this question as whether we can improve the performance or safety of our method by modifying its various components. One possible modification is to change the RL policies used at the high and low levels. Currently, TD3, an off-policy method, is employed because it is more sample-efficient than on-policy approaches, so there is little justification for replacing it with on-policy model-free methods. We could also consider replacing the hierarchical policy with the HIGL algorithm, which may be a potential direction for future work, as HIGL demonstrates better performance compared to HRAC. Additionally, incorporating an MPC controller as a low-level policy for planning in imagination, as implemented in the MBRCE approach, could enhance the safety of our method.

We sincerely thank you for your valuable commentaries and questions. We updated the article and uploaded the revised version. Considering the feedback from all the reviewers, we conducted additional experiments on SafeAntMazeWshape (ITES, HRAC, HRAC-L, TD3LAG), PointGoal sparse (ITES, TD3LAG, MBPPOL, SAC-L), PointGoal1 (TD3LAG), and CarGoal1 (TD3LAG).

Weaknesses:

• In our work we mostly concentrated on long-horizon tasks where it is challenging to flat algorithms to find any solution. Generally, the model-based approach improves by leveraging predictive dynamics to anticipate and mitigate risky actions, but, on its own, it struggles with scalability and generalization to such complex tasks. The hierarchical structure complements this by enabling task decomposition, which simplifies decision-making and allows the agent to operate efficiently across varying temporal and spatial resolutions.

We present two types of experiments.

1. The first demonstrates that model-based safety enhances safety in Hierarchical Reinforcement Learning (HRL). In ITES, the model-based component plays a critical role by evaluating the controller policy using imagined trajectories, specifically optimizing safety when achieving subgoals generated by the HRL component(high level policy). In Figure 7 we provide such experiments where HRAC-SafeSubgoals = HRL and ITES = HRL + model-based safety.
2. The second experiment shows that combining HRL and safety improves safety in model-based approaches. Model-based safety alone is insufficient for safely solving the task because it optimizes safety based on imagined scenarios over a limited horizon (10 steps ahead for SafetyGym, 15 steps for AntMaze). As the distance increases, the model of the world diverges. However, by generating subgoals at a short distance, Model-based safety effectively minimizes safety risks related to those subgoals. For experiments see Appendix A.

• In our testing benchmarks, we primarily rely on existing state-of-the-art (SOTA) benchmarks to evaluate two key properties: safety (using the SafetyGym benchmark) and hierarchical capabilities (through long-horizon tasks, such as variations of Ant Mazes). For the future development of ITES, we will focus on a broader range of tasks, particularly in robotic manipulation. However, we would like to highlight that most existing safe reinforcement learning algorithms (MBPPOL, MBRCE) have been evaluated on only a limited subset of tasks from the SafetyGym benchmark.

• We have added a link to the implementation of our algorithm. https://anonymous.4open.science/r/ITES-677D

Questions:

• Hierarchical structure of ITES is applicable for other tasks, but we should define the mapping function $\phi$ from states to goals manually for every specific task. The architecture of the hierarchy is not need to be redesigned.
• The cost model is a binary classifier that distinguishes safe and dangerous states. We define it as a MLP approximator, and use Binary Cross Entropy as a loss function for learning. In the text, we added an additional description of defining and learning the cost model (Section 3.2). The safety states are task specific, so they are defined for each task to take into account safety constraints - this is like an engineering task, then to find a desirable solution we need to define a reward function, but here we also need a cost function. Defining cost and reward functions is a challenging problem that mostly depends on the exact task including real world tasks.
• We would like to clarify your definition of "interval scale" in the context of subgoals. We interpret this to mean the number of low-level (primitive) controller steps that occur between subgoals identified by the high-level policy. In this framework, the interval scale is represented by a constant hyperparameter specific to each task. This hyperparameter is denoted as $k$ in the text, and its value is provided in Table 4 in the Appendix.
• Indeed, you are correct. In Table 2, the average value of FinalCost for our method is greater than that of the other approaches. While we do have a higher average rate of constraint violations, it is important to emphasize that our method demonstrates greater stability. Specifically, the standard deviations are 89 for CUP and 93 for FOCOPS in the CarGoal environment, whereas our method, ITES, has a standard deviation of 1.04 for the Final Cost metric. This suggests that, for certain seeds, CUP and FOCOPS may exhibit excessive violations of safety constraints, whereas our method shows reduced sensitivity to seed variations, thereby enhancing its reliability.
• Thank you for highlighting the typos; we have addressed them.

Thank you for valuable comments and constructive suggestions. We uploaded the updated version of the article. We introduced several additional benchmarks and conducted a set of new experiments on them. Also additional baseline algorithms were added. We improve the text (see highlighted zones in the text) according to your comments. Detailed description for each point follows below.

Weaknesses:

• To enhance the number of used benchmarks we added SafeAntMaze (W shape) and GoalPoint sparse. These tasks are more complex than existing ones. The former contains the enlarged maze with modification in walls. The latter is the SafetyGym Goal Point, but the reward is dense: only +1 for achieving the goal. It is also worth mentioning that, compared to many works in the field of Safe Reinforcement Learning MBPPOL[1], MBRCE[2], SAC-L[3], CUP[4], FOCOPS[5], which evaluate their performance exclusively in SafetyGym navigation environments, we also consider long-horizon SafeAntMaze environments in addition to SafetyGym navigation.

• To compare with other baseline algorithms we added SAC-L(lagrangian-based), MBPPOL(model-based). We compared our approach with MBPPOL in the PointGoal sparse environment and demonstrated that, with equivalent safety levels, ITES outperforms MBPPOL in terms of reward function. We did not compare MBPPOL in long-horizon AntMaze environments because its code cannot be reproduced in other environments or would require excessive time to run due to poor code quality.

• We combined all training logic into pseudocode and provided the corresponding algorithm in the Appendix.

• List of the hyperparameters was also added to the Appendix.

• Thank you for noticing the typo, we fixed it.

Questions:

• ITES has several limitations. The main limitation is the need to manually design the mapping function from state space to goal space for each task, which restricts its adaptability, particularly to visual input. For example, for 2D navigation tasks the goals are 2D coordinates, but for 3D navigation we should manually modify mapping to 3D coordinates.

• The simple discretization technique used by HRAC's Adjacency Network limits its scalability to high-dimensional spaces.

• ITES does not account for task-specific cost budgets, as it minimizes cost violations independently at each time step. This leads to minimization of the cost violations than the agent policy is already safe.

• We updated text to highlight these limitations in the Conclusion Section.

[1]Jayant, Ashish K., and Shalabh Bhatnagar. "Model-based safe deep reinforcement learning via a constrained proximal policy optimization algorithm." Advances in Neural Information Processing Systems 35 (2022): 24432-24445. [2]Liu, Zuxin, et al. "Constrained model-based reinforcement learning with robust cross-entropy method." arXiv preprint arXiv:2010.07968 (2020). [3] Yang, Qisong, et al. "WCSAC: Worst-case soft actor critic for safety-constrained reinforcement learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021. [4] Yang, Long, et al. "Cup: A conservative update policy algorithm for safe reinforcement learning." arXiv preprint arXiv:2202.07565 (2022). [5] Zhang, Yiming, Quan Vuong, and Keith Ross. "First order constrained optimization in policy space." Advances in Neural Information Processing Systems 33 (2020): 15338-15349.]