HASARD: A Benchmark for Vision-Based Safe Reinforcement Learning in Embodied Agents

I have not seen the author's response. I concur with the other reviewers that VisDoom lacks a detailed simulation of real-world physics.

W1 & Q1. Unpragmatic Constraints

We acknowledge the significance of safety and accurate motor control in robotics and control systems, which has been a central focus within the safe RL community. However, as the scope of safety in RL extends to broader domains beyond complex robotic motor control, including autonomous navigation [1], healthcare [2], LM safety [3], and energy systems management [4], there is a need for safe RL methods that could generalize across this broad domain spectrum. Such methods should be capable of addressing a wide range of safety constraints, thereby expanding the applicability and impact of safe RL to yet unexplored domains.

To support the development of such methods, we require different environments and benchmarks that facilitate this, rather than all of them narrowly tailoring to individual, pragmatic applications. Relying solely on robotic simulation suites like Safety-Gymnasium as the gold standard narrows the scope. The community could benefit from embracing a broader spectrum of meaningful environments that challenge diverse competencies and encourage generalizable solutions.

HASARD does not directly address any of these specific application domains, nor does it attempt to cover the entire range of safety aspects, but it does extend beyond conventional 2D environments and widely used robotic control tasks. Instead of focusing on realistic physics or lifelike simulations, HASARD emphasizes fostering decision-making in visually complex and challenging settings, offering a fresh perspective for advancing the field.

Not all constraints in HASARD are entirely disconnected from real-world settings. Although ViZDoom does not facilitate the accurate motor control required to match the complexity of the real world, it simulates the challenges of decision-making in visually intricate environments, such as perceiving depth.

In Precipice Plunge, the agent has to determine whether stepping or jumping down to a lower platform is safe by assessing whether it's close enough to avoid fall damage. This parallels the challenges faced by robots navigating complex terrains.

In Armament Burden, the agent must learn how each item contributes to its carrying load and balance this against how much weight it can carry. It should plan a good path to maximize its time efficiency. It must also factor in the distance to the delivery zone, evaluating whether the potential reward for delivering an item justifies temporarily exceeding its carrying capacity. These characteristics could be tied to logistics or supply chain optimization, such as where a robot needs to manage the warehouse inventory.

In Collateral Damage and Detonator's Dilemma, the agent must anticipate the future positions of other entities by analyzing their speed and past trajectories. This mirrors a similar competency in autonomous driving, where safety hinges on accurately predicting the movements of nearby vehicles and pedestrians.

The focus of safety here is less about precise control problems and more about strategic decision-making. While we acknowledge that these game-based environments are detached from reality, they effectively simulate high-level challenges and complexities. Recent studies in safe RL continue to utilize simplified, unrealistic environments for experimental evaluation, such as grid worlds [5, 6, 7], to investigate some very intricate aspects of their method. These simplified settings remain relevant due to their interpretability, which is often lacking in more realistic environments. Additionally, they are not directly tied to pragmatic safety constraints. In contrast, ultra-realistic and highly complex environments make it challenging to isolate specific factors.

[1] Nehme, Ghadi, and Tejas Y. Deo. "Safe Navigation: Training Autonomous Vehicles using Deep Reinforcement Learning in CARLA." arXiv preprint arXiv:2311.10735 (2023).

[2] Cao, Junyu, Esmaeil Keyvanshokooh, and Tian Liu. "Safe reinforcement learning with contextual information: Theory and application to personalized comorbidity management." Available at SSRN 4583667 (2023).

[3] Wachi, Akifumi, et al. "Stepwise alignment for constrained language model policy optimization." arXiv preprint arXiv:2404.11049 (2024).

[4] Huo, Xiang, et al. "Optimal Management of Grid-Interactive Efficient Buildings via Safe Reinforcement Learning." arXiv preprint arXiv:2409.08132 (2024).

[5] Garcıa, Javier, and Fernando Fernández. "A comprehensive survey on safe reinforcement learning." Journal of Machine Learning Research 16.1 (2015): 1437-1480.

[6] Wachi, Akifumi, et al. "Safe exploration in reinforcement learning: A generalized formulation and algorithms." Advances in Neural Information Processing Systems 36 (2024).

[7] Den Hengst, Floris, et al. "Planning for potential: efficient safe reinforcement learning." Machine Learning 111.6 (2022): 2255-2274.

W3 & Q3. Only PPO variations

It just so happens that the best versions of these popular safe RL methods work best with PPO. Most of the Safe RL algorithms we evaluated are not constrained to a specific base RL algorithm, and neither is the benchmark. The PPO variants are known to have the best performance. We also included TRPO implementations of some safe RL methods. We present the results of Level 1 in the following table. The results satisfying the safety constraint are depicted in cursive, and the one of these with the highest reward is depicted in bold.

We acknowledge the concern regarding the focus on PPO variants in our evaluations and would like to clarify that the benchmark itself is not tied to any specific type of algorithm. Equally, the Safe RL methods we selected are inherently independent of any particular base RL algorithm. However, in the original papers, the authors frequently utilized PPO as a base for their model-free methods. We therefore only included the PPO-based version not only because those have achieved higher results on prior benchmarks, but also not to compare apples with oranges, i.e., not to conflate differences in performance due to the choice of the base RL algorithm.

To provide additional context, we also included TRPO-based implementations for some Safe RL methods. In the following table, we present the results for Level 1. Results that meet the safety constraint are denoted in italic, and the highest reward safe result is depicted in bold.

Similar to prior findings, the TRPO versions of these methods consistently perform noticeably worse than their PPO counterparts. Also note, that evaluating all possible Safe RL methods is beyond the scope of our work.

Q4. 1-1 Comparisons with Prior Benchmarks?

Could the reviewer clarify what specific features or real-world benchmarks they are referring to? It is not entirely clear to us what exactly is being requested. We've included a performance comparison with Safety-Gymnasium in Appendix C of the paper. Among prior Safe RL benchmarks, Safety-Gymnasium stands out as both the most widely adopted in recent research and the most complex and similar to our work. Please let us know what other features or benchmarks we should compare.

W2 & Q2. Incremental to ViZDoom.

While HASARD builds upon the ViZDoom platform, it is far from a mere incremental extension of the original ViZDoom environments. Not only do they lack the means of evaluating safe RL, but are static and overly simplistic in terms of variety, visuals, map layout, and objectives, focusing on simple navigation to a target location (my_way_home, deadly_corridor, health_gathering) or turning/sidestepping and shooting enemies (defend_the_center, defend_the_line, predict_position). Additionally, all the original ViZDoom scenarios are restricted to flat, 2D surfaces, limiting their complexity and challenge. In contrast, we carefully designed the HASARD environments with more complex terrains and visuals, also introducing a notion of cost and exhibiting reward-cost trade-offs. We've incorporated novel dynamic elements in each environment: 1) changing and moving platforms in Volcanic Venture, 2) various items spawns and random pitfall locations in Armament Burden, 3) units with sporadic behaviour and random entity and object spawns in Collateral Damage and Detonator's Dilemma, 4) random layout and moving platforms in Precipice Plunge, and 5) dynamic lighting and item spawns in Remedy Rush. Additionally, HASARD introduces difficulty levels, which are absent in the original ViZDoom scenarios. The levels are not merely achieved by tweaking parameters but introduce novel elements.

We agree with the reviewer that advancing towards open-world generalization and realism is crucial for the field. However, to our knowledge, the majority of recent works in the field continue to rely on platforms like Safety-Gymnasium [1, 2, 3, 4, 5], MetaDrive [3], Safe Control Gym [6], MuJoCo [5, 6], or RoboSuite [6], which are far removed from realistic applications. We would appreciate it if the reviewer could provide references to recent works that utilize real-world settings or highly realistic environments for Safe RL or benchmark their systems on real-world systems.

Using ultra-realistic simulation environments for research poses challenges of computational overhead. Incorporating highly detailed physics and advanced visual rendering substantially increases the cost and time required for simulations. HASARD is intentionally designed to strike a balance between realism and computational efficiency, making it accessible to low-budget research settings. This approach broadens access to vision-based Safe RL research, enabling not only more researchers to engage with the field but also to expand the scope of evaluation from robotic motor control. Note, that ViZDoom still remains widely adopted in recent research [7, 8, 9, 10].

[1] Zhao, Weiye, et al. "Guard: A safe reinforcement learning benchmark." arXiv preprint arXiv:2305.13681 (2023).

[2] Hoang, Huy, Tien Mai, and Pradeep Varakantham. "Imitate the good and avoid the bad: An incremental approach to safe reinforcement learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 11. 2024.

[3] Huang, Weidong, et al. "Safe dreamerv3: Safe reinforcement learning with world models." arXiv preprint arXiv:2307.07176 (2023).

[4] Zhao, Weiye, et al. "Implicit Safe Set Algorithm for Provably Safe Reinforcement Learning." arXiv preprint arXiv:2405.02754 (2024).

[5] Li, Zeyang, et al. "Safe reinforcement learning with dual robustness." IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).

[6] Gu, Shangding, et al. "A Review of Safe Reinforcement Learning: Methods, Theories and Applications." IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).

[7] Park, Junseok, et al. "Unveiling the Significance of Toddler-Inspired Reward Transition in Goal-Oriented Reinforcement Learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 1. 2024.

[8] Kim, Seung Wook, et al. "Neuralfield-ldm: Scene generation with hierarchical latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

[9] Zhai, Yunpeng, et al. "Stabilizing Visual Reinforcement Learning via Asymmetric Interactive Cooperation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[10] Valevski, Dani, et al. "Diffusion models are real-time game engines." arXiv preprint arXiv:2408.14837 (2024).

Q2. Why ViZDoom?

The core advantages of ViZDoom lie in its speed, lightweight design, cross-platform compatibility, and ease of configuration. Although it may lack realism, it effectively replicates the notion of navigation and interaction in a 3D realm, which is directly tied to real-world problems. ViZDoom remains a widely adopted platform in the RL community and continues to be used [1, 2, 3, 4]. Indeed, as the reviewer has pointed out, it is useful for the platform to be highly customizable for the creation of more diverse tasks. In Appendix E, we have listed several promising extensions for HASARD. Apart from increasing task complexity through configurable parameters, one could combine challenging elements across tasks. The rationale behind selecting ViZDoom as the foundation for HASARD is further elaborated in Appendix F.

Open-World In the context of the tasks we designed, the open-world aspect does not significantly impact the Safe RL challenges. Whether the engine procedurally generates more content as the agent ventures further or whether the agent eventually hits a wall has only a marginal effect on task complexity. As demonstrated and discussed earlier, the higher levels of HASARD or the full action space setting already provide a sufficient challenge to current methods. Conversely, having a physically bounded environment enables a more detailed analysis of the agent's movement patterns, allowing us to draw connections between its behavior and the training process or the specific method employed, as examined in Section 5.4. Moreover, tasks with a more bounded scope enhance reproducibility and provide more fair ground for comparison.

Open-world environments like MineDojo are computationally more expensive to run, making them less suitable for limited resources. ViZDoom, on the other hand, uses rendering with sprites and precomputed lighting, making it computationally inexpensive, lightweight, and accessible, while open-world environments often use dynamic lighting, shadows, and voxel-based rendering, adding a computational overhead.

MineDojo We agree with the reviewer that a Minecraft-based benchmark holds significant potential for evaluating Safe RL and could challenge similar competencies that HASARD aims to address. Minecraft offers great variety in terms of items, entities, interactions, and possible objectives. A key advantage of MineDojo lies in its extensive knowledge base, built from gameplay videos and the Minecraft Wiki, which could be effectively leveraged for offline Safe RL methods.

However, for online learning, the simulation speed might pose challenges. Although neither DOOM nor Minecraft were created for RL experiments, Vanilla Minecraft has notable downsides for RL. Its Java-based engine introduces layers of abstraction, memory management overhead, and rendering complexity, which negatively impact performance. Additionally, while ViZDoom allows efficient parallel execution of multiple instances, Minecraft’s Java implementation with its higher memory usage and single-threaded limitations—makes scaling less efficient. Nonetheless, we recognize a Minecraft-based Safe RL benchmark as a promising avenue for future work.

Q3. How is P3O Implemented?

We apologize for any confusion. To clarify, we used the Sample-Factory [5] framework for algorithm benchmarking, not Omnisafe. As for P3O, whether it is generally an effective method due to its algorithmic details falls outside the scope of our work. Our focus is on evaluating widely known popular algorithms within our proposed benchmark rather than assessing the intrinsic effectiveness of individual algorithms like P3O.

[1] Park, Junseok, et al. "Unveiling the Significance of Toddler-Inspired Reward Transition in Goal-Oriented Reinforcement Learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 1. 2024.

[2] Kim, Seung Wook, et al. "Neuralfield-ldm: Scene generation with hierarchical latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

[3] Zhai, Yunpeng, et al. "Stabilizing Visual Reinforcement Learning via Asymmetric Interactive Cooperation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[4] Valevski, Dani, et al. "Diffusion models are real-time game engines." arXiv preprint arXiv:2408.14837 (2024).

[5] Petrenko, Aleksei, et al. "Sample factory: Egocentric 3d control from pixels at 100000 fps with asynchronous reinforcement learning." International Conference on Machine Learning. PMLR, 2020.

Q1. Which Tasks Require Memory and Planning?

In Armament Burden, the agent needs to memorize which items it has already collected to estimate how many more, and of what type, it is reasonable to gather before returning to the delivery zone. There is no direct indicator of the carried weight from a raw observation. The only feedback is the slowdown the agent experiences once the capacity is exceeded. When setting off from the starting zone, the agent should plan out a route of which items in which order to obtain to maximize its reward in a limited time. The weapon types and their locations are random each time, presenting a unique setting at every start. Due to partial observability, especially in higher levels, where the entire environment and obtainable items are not visible from the starting zone, it is advantageous for the agent to memorize the locations of items it has seen but not obtained in preceding runs during the same episode.

In Collateral Damage, the movement and velocity of enemy and neutral units can not be inferred from a single frame, necessitating a capacity for short-term memory. Each entity type exhibits different movement patterns that can be anticipated to some extent. By accurately perceiving depth and predicting future positions of the entities, the agent can plan its actions and time its rocket launches effectively to hit targets while avoiding collateral damage to neutral units.

Similar to predicting the sporadic movements of units in Collateral Damage, the agent must anticipate unit trajectories in Detonator's Dilemma, accompanied by a greater variety in unit behaviors and a more complex terrain. With barrels scattered across the map and both neutral units and barrels respawning after elimination, the agent benefits from maintaining a memory of prior detonations and unit locations to optimize its strategy. For instance, if the agent observes that its immediate area has numerous barrels but is heavily crowded with neutral units, it might strategically navigate to a less populated area to detonate barrels there. By retaining this memory, the agent can later return to the original location when conditions are more favorable.

We will further expand the environment descriptions in Appendix A to incorporate these details. If the reviewer feels that our claims regarding memory and planning capabilities are overstated or artificial, we are open to revising the emphasis.

W1. Outdated Baselines

We appreciate the reviewer’s concern regarding the use of recent baselines. The baseline methods we selected were chosen for their robustness and proven track record across a variety of prior benchmarks. While these methods may not represent the absolute latest developments, they are well-tested and reliable for evaluating the complexity and nuances of HASARD. That said, we acknowledge that integrating more recent vision-based safe RL methods could enhance the relevance of HASARD as a benchmark.

While Lambda, Safe SLAC, and SafeDreamer have public author-provided implementations, they are not yet integrated into widely-used Safe RL libraries such as SSA [1], FSRL [2], SafePO [3], and OmniSafe [4], which could streamline their adoption. Adapting them to new environments like HASARD requires substantial effort in terms of implementation adjustments.

Additionally, we encountered compatibility issues during our attempts to incorporate these methods. For instance, SafeDreamer relies on JAX [5], which is well-optimized for environments natively written in JAX but presents challenges when integrating with non-native environments. Similarly, neither Lambda nor Safe SLAC could be installed directly using the dependencies listed in their respective repositories. Despite extensive efforts and manual adjustments, we were unable to get these methods to train effectively.

[1] Ray, Alex, Joshua Achiam, and Dario Amodei. "Benchmarking safe exploration in deep reinforcement learning." arXiv preprint arXiv:1910.01708 7.1 (2019): 2.

[2] Liu, Zuxin, et al. "Datasets and benchmarks for offline safe reinforcement learning." arXiv preprint arXiv:2306.09303 (2023).

[3] https://github.com/PKU-Alignment/Safe-Policy-Optimization

[4] Ji, Jiaming, et al. "Omnisafe: An infrastructure for accelerating safe reinforcement learning research." Journal of Machine Learning Research 25.285 (2024): 1-6.

[5] https://github.com/PKU-Alignment/SafeDreamer

W4. Only Navigation Tasks

Indeed, most tasks in popular benchmarks such as Safety-Gymnasium (1st-person POV) [1], BulletSafetyGym (proprioceptive) [2], and MetaDrive (3rd-person POV) [3] primarily focus on the agent navigating toward a goal location while avoiding hazardous objects. However, they are comparatively simpler to solve. For example, many Safe RL methods that rely on a cost critic, only require it to learn to assign low values to observations where hazards are prominently visible in the center of the screen, and higher values to observations where hazards are either absent or located at the edges. This allows the critic to provide accurate value estimates based on a single observation, without requiring any history or further context.

In contrast, the scenarios in HASARD require a deeper level of perception and decision-making. Agents must comprehend spatial relationships, accurately perceive depth, and predict the movement and future locations of entities. Only the Remedy Rush task has a navigation objective similar to the prior benchmarks. However, it features more complex visual surroundings, a greater variety of items to collect, and objects to avoid. In contrast to Safety-Gymnasium’s visually simplistic environments, with distinct, single-palette cylindrical goals and hazard zones that are easy to differentiate, the collectibles and hazardous objects in Remedy Rush are more difficult to identify.

The other 5 tasks each have their unique nature and distinct objective. Volcanic Venture could be considered most similar, as the agent does have to avoid certain zones when navigating the area. However, the platforms, the agent should stay on, periodically switch locations, are of different height and in constant motion. These attributes far extend the static Safety-Gymnasium environments.

Armament Burden introduces another layer of partial observability on top of the conventional limited field of view with egocentric perception. The agent has a carrying load that is not in any way observable from the pixels on the screen. For instance, acquiring a weapon with an empty inventory incurs no immediate cost. However, if the agent has already picked up other items, this could exceed its carrying capacity.

Similarly, in Detonator's Dilemma, detonating a barrel near a neutral unit with high health points might result in minimal consequences, whereas doing so near units with low HP could incur high cost. The agent should not only accurately perceive the distance between the neutral units and the barrels, but also the barrels themselves, to prevent unintended chain explosions.

The core difficulty in Collateral Damage lies in accurately firing the weapon, as the projectile takes time to reach its destination. Both neutral and enemy units are in constant motion, making it difficult to predict their future positions in relation to the projectile. The agent must also balance risks and rewards, considering whether targeting a cluster of enemies near neutral units is worth the cost within the allowable budget.

In Precipice Plunge, the central challenge lies in depth perception, as the agent must find a safe path to carefully descend down the cave and avoid fall damage. This type of task is explored in any of the prior benchmarks. As the difficulty level increases, the randomized terrain becomes steeper and some blocks are in constant vertical movement. Moreover, the cave becomes progressively darker as the agent ventures deeper.

The answer to Q1 will further outline the challenges of HASARD environments. Moreover, in Appendix A of the paper, we have described the characteristics and complexities of each environment.

[1] Ji, Jiaming, et al. "Safety gymnasium: A unified safe reinforcement learning benchmark." Advances in Neural Information Processing Systems 36 (2023).

[2] Gronauer, Sven. "Bullet-safety-gym: A framework for constrained reinforcement learning." (2022).

[3] Li, Quanyi, et al. "Metadrive: Composing diverse driving scenarios for generalizable reinforcement learning." IEEE transactions on pattern analysis and machine intelligence 45.3 (2022): 3461-3475.

W3. Task Complexity

Focus
While we acknowledge that environment suites like Safety-Gymnasium already offer embodied perception environments with more sophisticated physics simulation, our aim is to expand the scope beyond basic navigation and continuous robotic motor control tasks. Rather than competing directly with suites like Safety-Gymnasium, HASARD serves as a complementary benchmark. While safe behavior in realistic physics and lifelike environments is undoubtedly an important direction in the field, HASARD emphasizes fostering higher-order comprehension and decision-making.

The reviewer has raised an important point about the sacrifice of realism - indeed there is a trade-off between the computational efficiency and simulation complexity. HASARD caters for complexity not in terms of the already widely-explored complex continuous control problems, but the ability to perceive, reason, and plan. all the while being magnitudes faster than the complex physics-based simulation environments such as Safety-Gymnasium. This allows researchers to explore other aspects of safe RL at a fraction of the computational cost, especially with the sample-hungry model-free approaches.

Action Space
Regarding the action space, we've introduced 2 settings. We conducted our main experiments using the simplified version of the action space (ranging up to 54 multi-discrete actions, depending on the task). This was done to demonstrate the (1) solvability of the tasks, (2) various reward-cost trade-offs, (3) learning stage analysis, and (4) complexity across difficulty levels. However, the core setting of HASARD is meant to incorporate the full action space of DOOM, featuring 864 multi-discrete actions combined with 2 continuous actions. This significantly raises the difficulty even in Level 1, as we showed in Appendix B.1.

It is arguable whether the continuous action spaces for Safety-Gymnasium agents are more challenging. The Point, Car, and Racecar agents operate with 2 continuous actions (steering and acceleration), Ant uses 8 continuous joint torques, and Doggo has 12 continuous joint torques. Even if this were the case, HASARD does not directly aim to compete with such settings, instead targeting other domains than continuous motor control. The simplified action setting in HASARD is meant to streamline development by enabling faster analysis, quicker iteration, and immediate feedback. It allows researchers to test and refine methods efficiently, focusing on task dynamics without the computational overhead of the full-action space.

Hard Constraints
We appreciate the reviewer for addressing this. We wish to clarify that introducing hard constraints is not a novel contribution of our work, nor is it a central focus of this paper, as similar setups can be implemented in other libraries with little ease. Do note, however, that our modifications vary between tasks and go beyond only lowering the cost threshold to zero. For example, in Armament Burden, exceeding the carrying capacity results in the loss of all previously collected weapons. The aim of the hard constraint setting is to provide a wider variety of evaluation settings, due to various safety formulations. We will reduce the emphasis and claims regarding the hard constraints in the writing.

The responses to the following weaknesses and questions will further describe the challenges of the HASARD scenarios.

W2. Solvability by Existing Algorithms

Are the tasks solved?
The reviewer has rightfully pointed out the lack of a performance upper bound, without which we cannot determine whether the tasks are already solved. Regular PPO that neglects costs could set a reasonable upper bound for reward, but we are really interested in the highest reward obtainable while adhering to the safety constraints. In our attempt to address this issue, we have included a human baseline, similar to the Atari benchmark. A human player was evaluated over 10 episodes on Level 3 of each task, recording both reward and cost. In the following table, we compare the human baseline to the Level 3 main results in the paper obtained with the simplified action space. As noted in Appendix B.1, this had more favorable outcomes than with agents attempting to leverage the full action space. The human player, however, had access to the more complicated original full action space.

The results indicate that none of the safe RL baselines have achieved human-level performance, suggesting that the tasks are still far from being solved. It is worth noting that the human baseline is suboptimal, leaving ample room for improvement. The sole purpose of this baseline is to show the potential for improved performance, considering also that the human player has privileged knowledge of how the game works. The RL agent has to learn this from pixel observations.

Why are baselines incapable?
In many instances, the human player reached a higher score not due to more precise control, but through strategies that none of the evaluated baselines managed to learn. After all, while the RL method can consistently output precise actions for each frame, human outputs are inherently noisier.

In Armament Burden, the agent has the option to discard all weapons. This strategy can be used to get rid of the very heavy weapons far from the starting zone, since carrying them back can be very costly. None of our evaluated baselines managed to learn this behavior. Instead, they simply avoided picking up heavy weapons altogether, as discarding weapons does not provide an immediate or obvious reward. Level 3 introduces decoy items that add to the carrying load but do not grant any reward when delivered. None of the methods were able to figure out how to avoid these items.

In Precipice Plunge, the agent has the option to restart the episode, if it decides that no safe actions are possible. This becomes very relevant in Level 3, as the height differences to all surrounding blocks might be too great to permit a safe leap. However, again none of the agents learned to utilize this. Instead, they either remained stuck on a block until the end of the episode or ended up making an unsafe action.

In Levels 2 and 3 of Remedy Rush, the environment becomes periodically dark after short time intervals, limiting the agent's ability to see the items it needs to collect or avoid. However, night vision goggles are located in the map, granting permanent vision when obtained. Despite this, none of the baselines developed a consistent strategy to seek out the goggles early in the episode before proceeding with their item collection.

Level 3 of Detonator's Dilemma allows the agent to push the barrels to safer locations with few or no units nearby before detonating them. However, since pushing barrels does not yield immediate rewards, the baseline methods failed to adopt this strategy. Instead, the agents tend to avoid going near the barrels, as detonating them while standing too close often results in the agent harming itself and incurring a cost.

Thanks for the reply. Considering that the contribution of this paper is incremental, I decided to increase my score. However, the author did not address my biggest concern and try to use the latest algorithm to run the proposed benchmark environment during the rebuttal period, so it's still unclear whether the current algorithm has solved the problem proposed by this benchmark.

W3 & Q2. ViZDoom Lacks Realism

While we acknowledge that engines like Isaac Gym offer advanced physics simulation, our aim is to expand the scope not only beyond conventional 2D environments (listed as a strength by the reviewer), but also continuous control robotic manipulation tasks. Rather than focusing on realistic physics or lifelike environments, our emphasis is on fostering decision-making in visually intricate settings. Notably, many recent studies in safe RL continue to utilize simplified, unrealistic environments for experimental evaluation [2, 3], indicating their ongoing relevance. ViZDoom, in particular, remains widely adopted in recent research [4, 5, 6, 7]. Appendix F of the paper provides further rationale for our choice of ViZDoom.

Figure 18b in Appendix G depicts the learning curves for Level 3 tasks. Training has not fully converged across multiple methods even after 5e8 environment iterations, underscoring the complexity of these environments and the high iteration count needed to approach peak performance. Despite the complexity of the high-dimensional pixel-based inputs, the framework was able to simulate 500M frames and perform 150K policy updates in a training time of ~3 hours on a single GPU. This complexity doesn’t stem from advanced physics or intricate motor control but instead from demands like precise navigation, spatial awareness, depth perception, and predicting the movement of other entities. Combining these challenges with accurate physics and continuous motor control would make the problem unfeasible to solve within a reasonable timeframe on standard hardware. Built on top of ViZDoom, HASARD therefore offers a balanced approach, blending fast simulation with complex tasks that remain meaningfully challenging.

W4. Inaccessible Video

We tried accessing the video hosted on emalm from multiple devices and did not encounter any issues. To ensure the reviewer can view the demo, we have also re-uploaded the video to the jumpshare platform. Please let us know if further adjustments are needed.

We thank the reviewer for their feedback! Please let us know if there is anything else we ought to address.

[1] Wachi, Akifumi, Xun Shen, and Yanan Sui. "A Survey of Constraint Formulations in Safe Reinforcement Learning." arXiv preprint arXiv:2402.02025 (2024).

[2] Wachi, Akifumi, et al. "Safe exploration in reinforcement learning: A generalized formulation and algorithms." Advances in Neural Information Processing Systems 36 (2024).

[3] Den Hengst, Floris, et al. "Planning for potential: efficient safe reinforcement learning." Machine Learning 111.6 (2022): 2255-2274.

[4] Park, Junseok, et al. "Unveiling the Significance of Toddler-Inspired Reward Transition in Goal-Oriented Reinforcement Learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 38. No. 1. 2024.

[5] Kim, Seung Wook, et al. "Neuralfield-ldm: Scene generation with hierarchical latent diffusion models." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

[6] Zhai, Yunpeng, et al. "Stabilizing Visual Reinforcement Learning via Asymmetric Interactive Cooperation." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[7] Valevski, Dani, et al. "Diffusion models are real-time game engines." arXiv preprint arXiv:2408.14837 (2024).

W1. Hard Constraints

This concern is well-founded. We wish to clarify that our introduction of hard constraints is not presented as a novel contribution, nor is it central to our paper, as similar setups can indeed be implemented in other libraries. However, it is worth noting that simply decreasing the cost threshold to 0 is not our only modification. Additional task-specific adjustments have been made; for example, harsher penalties apply upon incurring costs, often terminating the episode abruptly. In Armament Burden, for instance, the agent loses all acquired weapons if it exceeds carrying capacity.

Our primary focus here lies in the evaluation protocol introduced by HASARD, which emphasizes the integration and assessment of safety constraints in RL training. HASARD supports evaluations across diverse safety formulations [1], enabling fairer comparisons by aligning the safety problems that different algorithms are designed to address. If the reviewer yet nevertheless feels that the emphasis on hard constraints seems overstated or the claims appear unsubstantiated, we are open to adjust the writing to reflect this.

W2 & Q1. Different Safety Budgets

Understanding the trade-off between safety and reward is indeed crucial for evaluating how algorithms adapt to varying safety requirements. We addressed this in Appendix B.2 (referenced in Section 5.1), where we examined the effects of different cost budgets by testing PPOLag and PPOSauté on Level 1 of HASARD, with both higher and lower safety thresholds than the original setting. The results in Figure 14 nicely show that stricter safety bounds lead to lower rewards, with PPOSauté exhibiting greater sensitivity than PPOLag. We are open to incorporating additional safety thresholds and other baselines if the reviewer believes this would strengthen our work.

If the reviewer feels further details on these results would strengthen the main text, we are open to incorporating the key findings on these trade-offs directly into the main paper. This could potentially replace the hard constraints results section, which may have been overstated (as discussed above). We welcome and would appreciate the reviewer’s feedback on this matter.