Thank you to the authors for addressing all my questions, remarks, and concerns. I have raised the score to 8. The absence of a solid benchmark for robust reinforcement learning is one reason the field of deep robust reinforcement learning remains underexplored. I am confident this benchmark marks a meaningful step forward and will motivate the community to contribute and advance deep robust reinforcement learning.

We sincerely appreciate the reviewer’s insightful feedback and recognition of our work as a significant contribution to the robust reinforcement learning community. We are also grateful for the constructive suggestions, which will help us further improve the quality of this work.

Q1: M2TD3, a state-of-the-art baseline for robustness under model misspecification, is not cited. Its inclusion would strengthen the paper’s coverage of relevant baselines. While the benchmark is nearly exhaustive, baselines like RARL and M2TD3 are missing.

A1: Thanks for the reviewer’s valuable suggestions! We have added M2TD3 [1] and RARL [2] as important related studies in Appendix A. M2TD3, and RARL are certainly important advancements and baselines for robust RL in the face of model misspecification/shift.

As the reviewer noted, the primary goal of this benchmark is to provide a wide range of tasks for evaluating and proposing new algorithms, while haven't going beyond as a baseline benchmark. But it is a very interesting future direction to aggregate existing robust RL baselines in a benchmark for people's convenience to evaluate baselines and build new algorithms. M2TD3 and RARL will definitely be the important ones to be included.

[1] Tanabe, Takumi, et al. "Max-min off-policy actor-critic method focusing on worst-case robustness to model misspecification." Advances in Neural Information Processing Systems 35 (2022): 6967-6981.
[2] Lerrel Pinto, James Davidson, Rahul Sukthankar, and Abhinav Gupta. Robust adversarial reinforcement learning. In International Conference on Machine Learning, pp. 2817–2826. PMLR, 2017.

Q2: The explanation of adversarial disturbance via LLMs is interesting but could be more general. Instead of focusing on LLMs, the paper should emphasize the adversarial setup and consider an adversary such as two player Markov games with potential LLM integration as an example.

A2: We appreciate the reviewer’s insightful suggestions. The reviewer is absolutely correct: the adversarial disturbance mode for RL can be formulated as a two-player zero-sum game, where we currently use LLMs as an interesting example of the adversarial agent to disrupt the RL agent. We emphasize this mode and this game-theoretical view in Section 3.2 of the new version and refer to existing algorithms (such as M2TD3 [1]) that consider the adversarial disturbance mode. In the future, we will incorporate multiple existing algorithms (M2TD3, [2] for state-adversarial disturbance) to provide more options for the adversarial agent in the adversarial disturbance mode for a more comprehensive evaluation.

[1] Tanabe, Takumi, et al. "Max-min off-policy actor-critic method focusing on worst-case robustness to model misspecification." Advances in Neural Information Processing Systems 35 (2022): 6967-6981.
[2] Zhang, Huan, et al. "Robust deep reinforcement learning against adversarial perturbations on state observations." Advances in Neural Information Processing Systems 33 (2020): 21024-21037.

Q3: It is unclear how uncertainty sets can be built with the benchmark. Including examples in the appendix on constructing such sets, as proposed in the M2TD3 paper, would be beneficial.

A3: Thank you for the reviewer’s insightful comments. We have included more examples of constructing uncertainty sets in Appendix C.2 and E. Specifically, taking environment-disruptor for example, additional noise/attacks can be applied as external disturbance to workspace parameters to construct an uncertainty set, e.g., instead of a fixed wind and robot gravity, we can put random shift on them, let the wind speed follow a uniform distribution $U(0.8, 1.2)$, while robot gravity vary uniformly within $U(9.81, 19.82)$. Moreover, we can add disturbance to the robot internal physical dynamics, such as the torso length, which can be expressed as the original length plus $0.1\sin(0.2 \cdot \text{iteration number})$, and the foot length, which follows a similar perturbation.

The parameter perturbations related to friction and mass are far from sufficient for evaluating the robustness of RL algorithms. Furthermore, it is very easy for RRLS to generate these two types of parameter perturbations, and I do not believe that simple testing on 6 MuJoCo tasks can be considered a benchmark.

We appreciate the insightful discussions among the (public) reviewers regarding the related work RRLS. RRLS is certainly an important recent advancement in the robust RL literature that we highlight in both the introduction and related works in our initial paper. A detailed response to the reviewer will be provided shortly, along with responses for all other reviewers.

[1] Zouitine, A., Bertoin, D., Clavier, P., Geist, M., & Rachelson, E. (2024). RRLS: Robust Reinforcement Learning Suite. arXiv preprint arXiv:2406.08406.

Q5: The discussion about the limitation of the benchmark is missing.

Thanks for this important question. We listed the current limitations along with future directions as below:

• More tasks for broader areas. The current version of Robust-Gymnasium primarily focuses on diverse robotics and control tasks (over 60). For future plans, It will be very meaningful to include broader domains (adapt its corresponding existing benchmarks or works into robust RL tasks): such as semiconductor manufacturing [1], autonomous driving [2], drones [3], and sustainability [4]. This will allow us to foster robust algorithms in a broader range of real-world applications.
• More tutorials for user-friendly. As the reviewer suggested, one limitation of the initial implementation is the lack of detailed tutorials. So we make new tutorials: https://robust-gymnasium-rl.readthedocs.io to ensure this open-source tool enables flexible construction of diverse tasks, facilitating the evaluation and development of robust RL algorithms. We will keep evolving the tutorials as users request new requirements and demands.
• Evaluating more tasks and baseline algorithms. We primarily conduct experiments on selected representative tasks with corresponding baselines to cover as many kinds of tasks as we can. However, running baselines on all tasks would provide the most comprehensive evaluation, while the computational cost of such an approach is prohibitively high. Moving forward, we will continue evaluating more tasks and hope to get user feedback to accomplish the entire evaluation more efficiently together.

[1] Zheng, Su, et al. "Lithobench: Benchmarking ai computational lithography for semiconductor manufacturing." Advances in Neural Information Processing Systems 36 (2024).
[2] Dosovitskiy, Alexey, et al. "CARLA: An open urban driving simulator." Conference on robot learning. PMLR, 2017.
[3] Panerati, Jacopo, et al. "Learning to fly—a gym environment with pybullet physics for reinforcement learning of multi-agent quadcopter control." 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021.
[4] Yeh, Christopher, et al. "SustainGym: A Benchmark Suite of Reinforcement Learning for Sustainability Applications." Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. PMLR. 2023.

We deeply appreciate the reviewer's thoughtful comments, particularly the reviewer's recognition of our work's comprehensive overview and task design.

Q1: This paper made an effort in transforming diverse RL tasks into robust RL tasks where environmental perturbations are considered. While there are existing benchmarks ([1], [2], [3], [4]) that allow to add disturbances to RL tasks to test the robustness of RL algorithms.

A1: Thanks to the reviewer for raising this question --- really helpful for a thorough differentiation from this paper to prior works. We added a new section in Appendix A for the differentiation: RL works involving tasks for robust evaluation.

For a brief answer: There exists a lot of great prior works related to robust evaluation in RL. This work (Robust Gymnasium) fills the gaps for comprehensive robust evaluation of RL: 1) support a large number of tasks (over 60) for robust evaluation; 2) support various disruption types that may hinder robustness --- support potential uncertainty over various stages of the agent-environment interaction, over the observed state, observed reward, action, and the environment (transition kernel).

• Comparison to RL benchmarks designed for robustness evaluations. To the best of our knowledge, RRLS is the only existing benchmark designed specifically for robustness evaluations. Here is the table for comparisons between this work, RRLS, and the other three related works that the reviewer suggested. [1] and [3] focus specifically on safe RL, and [4] focuses on natural signal environments.

[1] https://github.com/utiasDSL/safe-control-gym
[2] RRLS: Robust Reinforcement Learning Suite
[3] Datasets and benchmarks for offline safe reinforcement learning
[4] Natural Environment Benchmarks for Reinforcement Learning

• Comparisons to other works involving robust evaluation RL tasks

Despite recent advancements, prior works involving robust evaluations of RL typically support a few robust evaluation tasks associated with only one disruption type.

Specifically, there exists a lot of benchmarks for different RL problems, such as benchmarks for standard RL [5,10], safe RL, multi-agent RL, offline RL, and etc. These benchmarks either don't have robust evaluation tasks, or only have a narrow range of tasks for robust evaluation (since robust evaluation is not their primary goal), such as [5] support 5 tasks with robust evaluations in control Besides, there are many existing robust RL works that involve tasks for robust evaluations, while they often evaluate only a few tasks in specific domains, such as 8 tasks for robotics and control [10], 9 robust RL tasks in StateAdvRL [6], 5 robust RL tasks in RARL [7], a 3D bin-packing task [11], etc. Since their primary goal is to design robust RL algorithms, but not a platform to evaluate the algorithms.

[5] Duan, Yan, et al. "Benchmarking deep reinforcement learning for continuous control." International conference on machine learning. PMLR, 2016.
[6] Zhang, Huan, et al. "Robust deep reinforcement learning against adversarial perturbations on state observations." Advances in Neural Information Processing Systems 33 (2020): 21024-21037.
[7] Pinto, Lerrel, et al. "Robust adversarial reinforcement learning." ICML, 2017.
[8] Zouitine, A., Bertoin, D., Clavier, P., Geist, M., & Rachelson, E. (2024). RRLS: Robust Reinforcement Learning Suite. arXiv:2406.08406.
[9] Towers, Mark, et al. "Gymnasium: A standard interface for reinforcement learning environments." arXiv (2024).
[10] Ding, Wenhao, et al. "Seeing is not believing: Robust reinforcement learning against spurious correlation." NeurIPS 2024.
[11] Pan, Yuxin, Yize Chen, and Fangzhen Lin. "Adjustable robust reinforcement learning for online 3d bin packing." NeurIPS 2023.

Thank the authors for solving my some concerns. I increased my score in response.

I appreciate the effort of introducing environmental perturbations into existing RL benchmarks. However, in my view, this contribution is not sufficient to present in a top conference like ICLR.

I won't stop it for being accepted if other reviewers champion it.

We sincerely appreciate the reviewer’s thoughtful review and recognition of our work’s motivation, benchmark, and experimental contributions. We are especially grateful for the valuable and constructive suggestions to enhance the quality of this work.

Q1: Improving the Presentation

• Re-organize Section 2.1, 2.2, 3.2. Need more examples, such as it is not clear to me what random noise on an environment disruptor means.
• It will be helpful to add additional information about the state-action space and how the different disruptors affect them for each environment. Add information about the action spaces for different environments such as “these environments all use joint control with action spaces according to the number of degrees and an additional grasp action”.

A1: The suggestions from the reviewer are very insightful and significantly improve the presentation of this work. As the reviewer suggested, we clarify and revise a lot in the updated version:

• More concise organization between original Sections 2.1, 2.2, and 3.2
  • We merge Section 2.1 and the first half of Section 2.2 into Section 2.1: introducing the functionality of three possible disruptors together. We merge the second half of Section 2.2 with Section 3.2 into Section 3.2: specify how those disruptor works: modes and frequencies.
  • We add more examples of adding noise to the environments in current Section 3.2: such as "The environment-disruptor introduces biases to dynamic parameters within a prescribed uncertainty set. For example, noise can be added to the torso length (Fig.4 (c)) to make it shift from $0.3$ to $0.5$."
• More details of the action-state space. We agree with the reviewer that more introduction of the robot model in tasks is important, which the reviewer can refer to Section 3.1 in the original/updated version. Such as for "Fetch" tasks, we include the details of the robot as "Fetch features a 7-degrees of freedom (DoF) Fetch Mobile Manipulator arm with a two-fingered parallel gripper [Plappert et al., 2018]". This implies that the action space is about to control this robot model. But since sometimes, there are various kinds of robot models (action and state spaces) in one set, where we can't specify each of them due to the limited space. However, we will include more in the Appendix and our tutorials to help users choose their preferred models.

Q2: Suggestions for the related works

• The related work section seems support the claim In L 73: “While numerous RL benchmarks exist, including a recent one focused on robustness to environment shifts (Zouitine et al., 2024), none are specifically designed for comprehensively evaluating robust RL algorithms.” I believe a more thorough differentiation from this paper would benefit the presented manuscript.
• I appreciate the additional section on robust benchmarks in Appendix A. In general for benchmark papers, I find it beneficial to demonstrate the novelty of the benchmark but providing citations to benchmarks that are related to demonstrate that there is a gap in the existing literature. Here is a non-exhaustive list of possibly relevant recent benchmarks that might be of use as a starting point [1-11]. There are older benchmarks too such as ALE and DM Control for which I recommend standard citations. Such a differentiation does obviously not have to happen in the main text

A2: We appreciate the reviewer's thoughtful suggestions for the related works. It significantly benefits the presented manuscript.

• We added a new section in Appendix A for the differentiation: RL works involving tasks for robust evaluation. We provide a more comprehensive comparison to existing RL works and benchmarks that involve tasks for robust evaluations, as a new section in the related works. The reviewer can refer to the general response or Appendix A.
• We added more related benchmarks, including [1-11]. We include more through differentiation between this work and other related benchmarks as the reviewer suggested, adding [1-11], ALE, DM Control, and more in the updated version. Please refer to the first section paragraph in the Related Work (Appendix A).

Dear authors, I appreciate your extensive response to my questions and many of my unclarities have been resolved. That being said, I have come to the conclusion that I do not believe that this paper is in a state that is publication ready and I will not raise my score. I do maintain that this paper can be a useful contribution to the community and encourage the authors to continue their valuable work.

First I would like to highlight that I appreciate the consolidation changes to previous sections for readability. These make reading significantly easier.

I also appreciate the addition of a second related work section which highlights the gap in the broader literature.

Thank you for the elaboration on the LLM usage. I’m somewhat neutral to this point and will drop it in my consideration. I believe that it’s odd that we will ask researchers to run a proprietary LLM to do their experiments but given the development in other fields this might be impossible to avoid.

My remaining concerns I will outline here:

Standardized Usage There seems to be some confusion about how the term benchmark is used in our discussion. Thus, I will clarify what I understand by benchmark. This is opposed to a general code wrapper that can be used to implement a benchmark. I think a reasonable definition of a benchmark is provided in [1]:

``A benchmark is a software library or dataset together with community resources and guidelines specifying its standardized usage. Any benchmark comes with predefined, rankable performance metrics meant to enable algorithm comparison.''

I do not believe that the current manuscript makes it clear what the standardized usage is. And the more I think about it, the harder I find it to imagine it is possible to determine this usage without providing at least one experiment per task. I appreciate the addition of Appendix E but it seems that that Appendix re-iterates the values stated in the experiments. And I don’t find any reason why these should be the standard. For instance, given that there are only 3 seeds the value of 0.1 in Figure 5 seems to basically have no impact PPO. I believe that this goes in hand with the public comment on the other review.

I think a good example for this is the meta-world paper [2] section 4.3. It provides the evaluation protocol and how the authors suggest one use the benchmark. This includes for instance how various levels of difficulty can be achieved. In the presented manuscript, this information is not clear to me. They then proceed by running experiments to validate that these are good choices.

Metrics I appreciate the additional information on metrics but similar to the previous paragraph, it is not clear to me if I should measure these since they have not been validated and proven to be useful in at least a few experiments.

Experiments on all tasks Determining the correct settings for the benchmark in a benchmark like this requires running experiments on all tasks. I understand that it is computationally expensive but the manuscript is supposed to be a scientific publication of a benchmark with 60 tasks. Thus, I would expect there to be experiments on 60 tasks. Again, I will use meta-world [2] as a reference which has an experiment on all their 50 newly-designed tasks demonstrating that all tasks are learnable. Similarly, an experiment validating the choices of parameters here I believe is crucial. I’m not asking for all baselines but to determine whether 0.1 is a reasonable value for Gaussian noise, one needs to look at more than one situation.

The rebuttal argues that these tasks are representative but it is unclear what representative means. For instance, [1] highlights that hopper is not representative of itself under a different framework. The rebuttal also states that the authors will add more experiments in the future. However, I believe that this should be included in the present manuscript and not a future publication.

Specification of state space Given that the benchmark is about state space disturbances, I believe it should be clear in the manuscript what the state spaces are. More precisely, if I wanted to answer the question whether the same noise parameters have a larger impact on a larger state space (as suggested in A13) I would first have to go to another paper to figure out the state space. I understand that space is limited but this could go to the Appendix with a pointer in section 3.

Summary The rebuttal reads to me that experiments are still being run for future work and that not all features such as environment disruptions are implemented. Computational cost can not become an argument for not running experiments on a scientific publication. If the cost is too high, it might make sense to consider fewer tasks. All this indicates to me that some more time before publication is needed. Given my other concerns above, I will maintain my overall score, increase my presentation and reduce my soundness score.

Q4: Specification of state space Given that the benchmark is about state space disturbances, I believe it should be clear in the manuscript what the state spaces are. More precisely, if I wanted to answer the question whether the same noise parameters have a larger impact on a larger state space (as suggested in A13) I would first have to go to another paper to figure out the state space. I understand that space is limited but this could go to the Appendix with a pointer in section 3.

A4: Thank you for the insightful suggestion. The state space in our benchmark includes both the robot's state and relevant environmental information, as is standard in RL settings. For example, in Mujoco tasks such as Ant, the state space represents the robot's joint positions, velocities, and other dynamic properties.

We agree that a detailed specification of state spaces is essential for clarity, especially for understanding the impact of noise parameters on larger state spaces, as suggested in A13. To address this, we will provide a detailed introduction of the state spaces for each task in our tutorial.

Q3: Experiments on all tasks Determining the correct settings for the benchmark in a benchmark like this requires running experiments on all tasks. I understand that it is computationally expensive but the manuscript is supposed to be a scientific publication of a benchmark with 60 tasks. Thus, I would expect there to be experiments on 60 tasks, ... ,The rebuttal also states that the authors will add more experiments in the future. However, I believe that this should be included in the present manuscript and not a future publication.

A3: Thank you for your insightful comments. We fully agree with the reviewer that all newly proposed tasks are required to be tested to confirm their usefulness.

• The learnable of all over 60 task bases were verified. We appreciate the reviewer raising the learnability question for these task bases. The goal of this benchmark is to introduce an additional disruption module for constructing diverse, robust RL tasks using different robot models and task types (e.g., grasp manipulation, dexterous hands, and locomotion). To achieve this, we collected tasks from various existing standard RL, safe RL, and multi-agent RL benchmarks, forming a diverse set of task bases. Thanks to prior works, the learnability of these 60+ tasks has been widely evaluated in the RL literature. When it comes to Meta-World [2], a solid benchmark for meta-learning, introduced 50 new robot manipulation tasks with structural similarity, enabling the construction of subsequent meta-learning tasks in Section 4.3. As those 50 tasks are new, an exhaustive evaluation of these 50 tasks is both reasonable and valuable.

• We follow the approach of Meta-World [2], which demonstrates meta-learning tasks using representative examples. Specifically, we adopt the evaluation process outlined in Meta-World [2], a strong example of a modular benchmark. It is modular in that it constructs different meta-learning tasks using 50 task bases (a feature we greatly appreciate). Meta-World covers all four types of meta-RL tasks proposed in Section 4.3 by focusing on representative examples. For instance, in Meta-Learning 1 (ML1), which involves few-shot adaptation to goal variations within a task, ML1 tasks are constructed using three task bases (reach, push, pick-place) rather than all 50. This approach effectively demonstrates the reasonableness of such meta-RL tasks, as evaluating combinations of all 50 tasks is computationally infeasible.

  Similarly, we cover all proposed robust RL tasks by focusing on representative examples. We introduce six categories of robust RL tasks in a modular framework (task bases + a disruptor), with some overlap between categories. For each category, we select representative tasks to demonstrate their learnability and vulnerability to uncertainty.

  • Observation-disrupted RL problems: We evaluate robust RL tasks (task base + disruption on state/reward), e.g., evaluated in HalfCheetah with Gaussian noise (Figure 5 (a)), Multi-Agent HalfCheetah with Gaussian noise (Figures 8 (a) and $c$), Ant with Uniform noise (Figure 9 (a)).
  • Action-disrupted RL problems. We evaluate robust RL tasks (task base + disruption on agent's action sent to the environment), e.g., HalfCheetah with Gaussian noise (Figure 5 (b)), Multi-Agent HalfCheetah with Gaussian noise (Figure 8 (b)).
  • Environment-disrupted RL problems with the internal dynamic shift: We evaluate robust RL tasks (task base + disruption on robot models), e.g., evaluated in Ant with internal attack in Figure 6 (b), such as torse length.
  • Environment-disrupted RL problems with the external dynamic shift: We evaluate robust RL tasks (task base + disruption on external environment), e.g., evaluated in Ant with external attack in Figure 6 (a), such as wind.
  • Random-disrupted RL problems: We evaluate robust RL tasks (task base + random diruption on observation/state/environment), e.g., evaluated in HalfCheetah with Gaussian attack on reward in Figure 10 (a) and (b), SafetyWalker2d with Gaussian attack on cost in Figures 7 $c$ and (d).
  • Adversarial-disrupted RL problems: We evaluate robust RL tasks (task base + adversarial disruption on observation/state/environment), e.g., evaluated in Ant with an adversary LLM policy attack, in Figure 9 (a) and (b).

All the proposed robust RL tasks are covered, following a similar approach to Meta-World. Additionally, given the differences in task bases, we ensure that each kind of task is included in at least one type of robust RL task: single-agent RL (Sections 4.1–4.2), safe RL (Section 4.3), and multi-agent RL (Section 4.4).

Q1: Standardized Usage A reasonable definition of a benchmark is provided in [1]: ``A benchmark is a software library or dataset together with community resources and guidelines specifying its standardized usage. Any benchmark comes with predefined, rankable performance metrics meant to enable algorithm comparison.'' I do not believe that the current manuscript makes it clear what the standardized usage is. And the more I think about it, the harder I find it to imagine it is possible to determine this usage without providing at least one experiment per task. I appreciate the addition of Appendix E but it seems that that Appendix re-iterates the values stated in the experiments. And I don’t find any reason why these should be the standard. For instance, given that there are only 3 seeds the value of 0.1 in Figure 5 seems to basically have no impact PPO. I believe that this goes in hand with the public comment on the other review. I think a good example for this is the meta-world paper [2] section 4.3. It provides the evaluation protocol and how the authors suggest one use the benchmark. This includes for instance how various levels of difficulty can be achieved. In the presented manuscript, this information is not clear to me. They then proceed by running experiments to validate that these are good choices.

A1: Thank you for raising this important point and providing a thoughtful definition of a benchmark. To clarify the standardized usage of our benchmark, we propose the following framework for attack modes and evaluation settings. These align with the principles of benchmarking, including standardized performance metrics and evaluation protocols:

• Random Attack (Easy) -> Adversarial Attack (Hard).

  • Random Attack (Easy): Random noise, drawn from distributions such as Gaussian or uniform, is added to the nominal variables. This mode is applicable to all sources of perturbation and allows for testing robustness under stochastic disturbances, e.g., see Figure 5 (a) and (b). Adversarial Attack (Hard): An adversarial attacker selects perturbations to adversely degrade the agent’s performance. This mode can be applied to observation or action perturbations and represents the most challenging scenario, e.g., see Figure 9 (a) and (b).

• Low state-action dimensions)(Easy) -> High state-action dimensions (Hard).

  • As the state and action space dimensions increase, the tasks become significantly more challenging. The difficulty level of tasks typically progresses from Box2D, Mujoco tasks, robot manipulation, and safe tasks to multi-agent and humanoid tasks. For instance, the Humanoid task, with a 51-dimensional action space and a 151-dimensional state space, is substantially more challenging than the Mujoco Hopper task, which has a 3-dimensional action space and an 11-dimensional state space.

Regarding the concern about "3 seeds and the value of 0.1 in Figure 5 having no impact on PPO," we followed the seed settings in the original PPO paper by Schulman et al [3], which used 3 seeds. In Figure 5, under both in-training attack and post-training attack scenarios, PPO’s performance is disturbed by the random attacker and performs worse compared to the original PPO without any attacks, demonstrating the impact of these disturbances.

We acknowledge that the manuscript could better specify these standardized protocols and provide a clearer evaluation framework. We have included detailed usage protocols, similar to the evaluation methodology outlined in Section 4.3 of the meta-world paper [2], to ensure clarity and standardization, see Appendix D. Additionally, while Appendix E reaffirms the experimental settings, we agree that running experiments for all tasks and difficulty levels would provide further validation. However, due to computational constraints, we have prioritized representative tasks across various scenarios to balance feasibility with meaningful evaluation.

Thank you again for pointing out this critical aspect, and we will ensure the final manuscript includes these clarifications.

[1] Can we hop in general? A discussion of benchmark selection and design using the Hopper environment. Claas A Voelcker et al., Finding the Frame Workshop at RLC 2024.
[2] Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning. Tianhe Yu et al., CoRL 2019.
[3] Schulman, John, et al. "Proximal policy optimization algorithms." arXiv preprint arXiv:1707.06347 (2017).

We want to sincerely thank Reviewer SF50 for the constructive suggestions and active discussions, which have greatly helped refine and strengthen this paper, positioning it as a more qualified benchmark. We acknowledge that there is room for improvement and are actively continuing its development. We believe this initial version provides a solid starting point for researchers working on robust RL and serves as a valuable tool for gathering user feedback to guide the creation of a more tailored next generation.

We will do our best to address the reviewer’s concerns as outlined below and warmly welcome further discussions or additional questions.

Clarification: Public comment was on the related work RRLS, not this work

We want to clarify that the public comments were actually a response to "reviewer RCMy" about prior work RRLS, "The parameter perturbations related to friction and mass are far from sufficient for evaluating the robustness of RL algorithms. Furthermore, it is very easy for RRLS to generate these two types of parameter perturbations, and I do not believe that simple testing on 6 MuJoCo tasks can be considered a benchmark." To the best of our knowledge, RRLS is the only existing benchmark designed specifically for robustness evaluations before this work, which involves 6 tasks. Please refer to the general response for the detailed comparisons (table) between RRLS and this work.

Q2: Metrics I appreciate the additional information on metrics but similar to the previous paragraph, it is not clear to me if I should measure these since they have not been validated and proven to be useful in at least a few experiments.

A2: Thank you for the feedback on the metrics. We apologize for misunderstanding the reviewer's question previously. We do include standard metrics for evaluating the robustness of an RL algorithm in the initial manuscript, which is a standard metric for robust evaluation in RL literature.

The metric is "post-training performance":

• The evaluation process.: An algorithm will be trained in a standard non-disrupted environment (a task base) without awareness of any uncertainty in the future. After training, we fix the output policy of the algorithm and evaluate it in the testing environment with additional disruption. The disruption module during testing can be set in different types (Sec 2.1) and modes (Sec 3.2), with different attacking levels and frequenties that can be specified by the users' preferences.
• The evaluation of robust metrics. We will use the average performance of a batch of episodes during testing as the robust metrics for RL algorithms [1-2]. For instance, for a random attack for a state-disruptor, in our experiments, we apply a random Gaussian noise attack during testing and then collect the average return of 30 episodes as the performance metric of an algorithm. Since we have 3 seeds, we evaluate in 90 episodes in total for any evaluated tasks, that match the standard styles in prior arts ([1-2] use an average performance of 50 episodes in testing ). Besides the average performance, users can also use other performance metric during testing, for instance, the worst-case performance of 90 episodes, or other metrics [2].

[1] Zhang, Huan, et al. "Robust deep reinforcement learning against adversarial perturbations on state observations." Advances in Neural Information Processing Systems 33 (2020): 21024-21037.
[2] Zouitine, Adil, et al. "RRLS: Robust Reinforcement Learning Suite." arXiv preprint arXiv:2406.08406 (2024).

We thank the reviewer for the engaging discussion and for providing useful suggestions. We sincerely appreciate the insightful and valuable feedback. Below is our response:

Q1: Furthermore, I understand that you believe the only relevant measure of robustness is average performance. I maintain that other interesting measures exist and the benchmark would be more useful if these were considered since measuring additional signal is usually easy.

A1 Thank you for the reviewer’s valuable feedback. We agree with the reviewer that robustness can be evaluated through additional metrics, such as the worst-case performance [1], which we already included in the updated version.

[1] Zouitine, Adil, et al. "RRLS: Robust Reinforcement Learning Suite." arXiv preprint arXiv:2406.08406 (2024).

Q2: Statistical validity of PPO. As pointed out by my other references before, much has changed about statistical evaluation since PPO was first published. Also, I’m not saying you must have more seeds. I’m saying that the experiment conducted in Figure 5a is not statistically significant. The choice of statistical robustness should be made based on necessity to support a claim.

A2: Thank you for bringing this to our attention. We agree that different tasks require varying numbers of seeds to ensure statistical validity for drawing conclusions. While Figure 5(a) does exhibit high variance, it sufficiently supports our claim regarding the degradation of baseline performance when the agent's observed state is attacked. We will work on including more seeds in the experiments to minimize potential confusion.

Q3: Experimental evaluation. I understand that the task bases have been evaluated by other works but the presented manuscript does not suggest new task bases. It is suggesting tasks with state perturbations. These perturbations effectively create new tasks which the benchmark should evaluate. In the paper's terms, it provides a new set of state-perturbed MDPs that need to be evaluated. Whether or not these are still solvable is completely unclear from prior work’s evaluations.

A3: Thanks for the reviewer's insightful comments. Indeed, our benchmark introduces a unified framework that suggests tasks with perturbations on three different components of RL, on state, action, and the environment (not only state perturbation), along with different perturbation modes (e.g., random, adversarial, shifting environmental internal/external dynamics).

We did test all tasks in a simple manner and provided examples demonstrating how to use each of them (See the tutorials link's "examples" folder). These examples can render the task visual and are intended to help users understand which tasks are particularly challenging and how to effectively utilize the benchmark for their research. Due to the limited computation resources, we only conduct comprehensive experiments on parts of them, which still cover all three components (state, action, environment) and all perturbation modes. Specifically, for State/Action-disrupted MDPs, we provide both random and adversarial attacks: Random: Gaussian noise (e.g., 0.1) is added to state variables, as shown in Figure 5. Adversarial: Action perturbations are subjected to adversarial attacks, as demonstrated in Figure 9 (a) and (b). Additionally, we introduce Environment-disrupted MDPs, such as evaluating Ant with internal attacks, like torso length changes, as shown in Figure 6 (b). The small examples and thorough experiment results provide users with a clear understanding of how to approach these new tasks. We agree with the reviewer that more thorough experiments would be beneficial.

I again thank the reviewers for this thorough discussion round and their hard work. I believe the paper has made some good progress. However, it seems to me that this discussion is circling at this point and I do not believe my concerns have been addressed. I will summarize my points for the AC.

• The paper currently does not meet the benchmark definition that I have provided earlier as no clear and explicit evaluation protocols are provided. Describing the features of the benchmark is not equivalent to evaluation protocols for the end-user. Each benchmark should have standardized usage protocols that include which tasks to use and how to set the corresponding parameters for each task.
• The work introduces novel tasks that have not all been evaluated in the manuscript and it has not been demonstrated that tasks are still learnable or under which circumstances they are not. If the tasks have been evaluated, evidence for this should be included in the manuscript. It should not be needed that every user runs every task and visualizes it to figure out which tasks might make sense to use. As I have highlighted, representativeness of tasks is somewhat ill-defined.
• Computational cost is not an excuse to not conduct required scientific experiments. If the computational cost is too high, it might make sense to provide settings with fewer tasks. This could easily be achieved via evaluation protocols.

Thus, I maintain that the work has the potential to make a valuable benchmark contribution but for now I will keep my score and recommend rejection.

Dear Reviewer SF5o,

Thank you for your constructive feedback. We apologize if there was any misunderstanding in addressing your points. Below, we aim to clarify our responses:

1. Usage Protocols:
   We do provide usage protocols to guide users on how to use the benchmark and set corresponding parameters. These can be found in the following resources:

   • Source Code: Examples are available in the "examples" folder of our repository: https://anonymous.4open.science/r/Robust-Gymnasium-E532/.
   • Appendices: Detailed descriptions are included in Appendices C, D, and E of the latest version of our paper: https://openreview.net/pdf?id=2uQBSa2X4R#page=20.78.
   • Tutorials: Comprehensive tutorials are provided in our documentation: https://robust-gymnasium-rl.readthedocs.io.

2. Task Evaluation:
   While it is not feasible to run thorough experiments on all tasks due to high computational costs within our current resources, we have conducted small-scale tests on all tasks (e.g., using a limited number of episodes) to evaluate the validity of all tasks, and conducted thorough experiments on at least 15 representative tasks. These tasks were selected to cover diverse settings, such as based on attack modes and attack sources.

3. Representativeness of Tasks:
   We agree with the reviewer's points, representativeness can vary depending on the context. To address this, we categorized the tasks based on different attack modes and sources to ensure diverse and meaningful coverage. While we acknowledge the limitations of defining representativeness, this strategy aims to balance computational feasibility with scientific rigor.

4. Computational Cost:
   The reviewer is correct, running more experiments would definitely be useful for the benchmark. However, as mentioned, running experiments on all tasks would lead to high computational costs, which might not provide proportional scientific value. Instead, we have evaluated at least 15 tasks comprehensively to cover different settings, and we believe at least the evaluated tasks can be useful for the robust RL community and support further development in this field.

We appreciate your suggestions and hope our response can be useful in addressing your concerns. Thank you for your feedback and the opportunity to improve the quality of our work.

Best regards,

The Authors

Q5: Can the benchmark be used to evaluate the robustness of RL algorithms in partially observable environments?

A5: Thanks for the valuable suggestions. Yes, this benchmark already involves partially observable tasks and can be used to construct more partially observable tasks directly.

• Support two kinds of partially observable tasks.
  • In multi-agent RL tasks (MAMuJoCo), we can selectively apply attacks to the observations of part of the agents while leaving others unaffected. Regarding the observations of all agents as the 'observation', this will be a partially observable task. Such partially observed experiments were conducted and the reviewer can refer to Figure 13(a).
  • Random noise mode on observation in all single-agent tasks. One kind of tasks in this benchmark is to attack the observation with random noise following a fixed distribution, which is indeed a kind of partially observable tasks [1].
• More partially observable tasks can be directly constructed. The user can design their own distribution of noise to add on the observation --- constructing different partially observable tasks, such as masking out partial dimensions of the observations.

[1] Duan, Yan, et al. "Benchmarking deep reinforcement learning for continuous control." International conference on machine learning. PMLR, 2016.

Q6: Limitations and future works of the current implementation of Robust-Gymnasium.

A6:
Thanks for this important question. Future directions are listed below:

• More tasks for broader areas. The current version of Robust-Gymnasium primarily focuses on diverse robotics and control tasks (over 60). For future plans, it will be very meaningful to include broader domains (adapt its corresponding existing benchmarks or works into robust RL tasks): such as semiconductor manufacturing [1], autonomous driving [2], drones [3], and sustainability [4]. This will allow us to foster robust algorithms in a broader range of real-world applications.
• More tutorials for user-friendly. As the reviewer suggested, one limitation of the initial implementation is the lack of detailed tutorials. So we make new tutorials: https://robust-gymnasium-rl.readthedocs.io to ensure this open-source tool enables flexible construction of diverse tasks, facilitating the evaluation and development of robust RL algorithms. We will keep evolving the tutorials as users request new requirements and demands.
• Evaluating more tasks and baseline algorithms. We primarily conduct experiments on selected representative tasks with corresponding baselines to cover as many kinds of tasks as we can. However, running baselines on all tasks would provide the most comprehensive evaluation, while the computational cost of such an approach is prohibitively high. But we will continue evaluating more tasks and hope to get user feedback to accomplish the entire evaluation more efficiently together.

[1] Zheng, Su, et al. "Lithobench: Benchmarking ai computational lithography for semiconductor manufacturing." Advances in Neural Information Processing Systems 36 (2024).
[2] Dosovitskiy, Alexey, et al. "CARLA: An open urban driving simulator." Conference on robot learning. PMLR, 2017.
[3] Panerati, Jacopo, et al. "Learning to fly—a gym environment with pybullet physics for reinforcement learning of multi-agent quadcopter control." 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021.
[4] Yeh, Christopher, et al. "SustainGym: A Benchmark Suite of Reinforcement Learning for Sustainability Applications." Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track. PMLR. 2023.

We appreciate the reviewer’s recognition of the comprehensive and user-friendly nature of our benchmark, designed specifically for the robust evaluation of RL algorithms.

Q1: The variety of disruptions and the modular nature might make the benchmark complex to understand and use for some users.

A1: We appreciate the reviewer’s insightful comments on the potential complexity for users. As the reviewer suggested, we make new tutorials: https://robust-gymnasium-rl.readthedocs.io to ensure "the variety of disruptions and the modular nature" leads to flexible and friendly usability rather than hinder it:

• For disruption modules: 1) A complete example of selecting disruption types in "Section 2 in Quick Start"; 2) For each type of disruption (observation-, action-, and environment-), we provide separate examples and detailed documentation.
• For modular design: in "Quick Start"; 1) A complete example of constructing one task via modality; 2) Showing how to replace individual modules, such as task base, disruption mode/type, etc.

With the new tutorials, we believe the variety of disruptions can offer RL researchers more options to test and enhance the robustness of their algorithms, while modularity enables flexible task construction and easy code customization for users.

Q2: The effectiveness of some robust RL algorithms might rely on the quality and quantity of offline demonstration data.

A2: The reviewer's insights are totally correct. There is a line of works that focus on robust RL with offline data, e.g., [1]. The evaluation experiments of RL baselines in this work currently focus on using online data, since this is the most classical setting of RL. But the reviewer's question is inspiring since our Robust-Gymnaisum benchmark certainly provides a wide array of tasks, where users can generate offline demonstration data with diverse quality and quantity from and work on offline RL. The offline dataset can be obtained by training or using a behavior policy to run in a task and collect the data during or after training, where the quality and quantity of the offline dataset can be controlled by choosing a behavior policy or the phase to collect data.

[1] Panaganti, K., Xu, Z., Kalathil, D., & Ghavamzadeh, M. (2022). Robust reinforcement learning using offline data. Advances in neural information processing systems, 35, 32211-32224.

Q3: The performance of algorithms on the benchmark could be sensitive to hyperparameter tuning, which might not be straightforward.

A3: Thanks for raising this important question. To be fair, we have used the same hyperparameters and experimental settings across all algorithms for comparisons in our experiments. The reviewer is correct that hyperparameter tuning may influence each baseline and remains an open challenge in the field. We offer the benchmark to serve as a platform for boosting the hyperparameter tuning for new algorithms.

Q4: How does Robust-Gymnasium handle continuous action spaces and high-dimensional state spaces?

A4: Thanks for raising this point. Covering tasks with continuous action spaces and high-dimensional states is indeed a great feature of Robust-Gymnanism.

• Over 50 tasks are with continuous action spaces. Robust-Gymnasium primarily focuses on robotics and control applications, so all the tasks are with continuous action spaces, except for some in Gymnasium-Box2D. For implementation, Robust-Gymnasium supports continuous action spaces using the Box space API originated from Gym API, allowing defining and using real-valued actions within set intervals.
• Support high-dimensional state spaces. Robust Gymnasium also supports tasks with high-dimensional state spaces, such as the Mujoco Humanoid task (the state space are of 45 dimensions), and four tasks from HumanoidBench which features high-dimention (the state space are of 151 dimensions with two hands). For implementation and computation efficiency, Robust-Gymnasium incorporates vector data streamline processing to enable fast computation of high-dimensional vectors.

I am happy with the authors' detailed response. Thanks!

I decide to raise my score to 8.