MuJoCo Manipulus: A Robot Learning Benchmark for Generalizable Tool Manipulation

Thank you for your review of our work. Here, we address your concerns:

Pt. 1 “A significant limitation of the benchmark is that the tasks are based on floating control points, rather than simulating direct interaction between the robot and the tool. This abstraction reduces the realism of the tasks and poses challenges for sim-to-real transfer, which is critical in robotic manipulation.”

• Note, we provide the same response to this as for Reviewer P8eT, who raised a similar concern about our work:
• Although there are no robots, this does not mean our benchmark is incomplete, or that it does not support Sim2Real research efforts.
  • Direct control of tools is a common way to test tool manipulation, as demonstrated in related work such as https://robotic-tool-design.github.io/. The idea of using floating manipulators is not new either, as shown in the popular Adroit Manipulation Platform, and is a popular method for simulating 6-DoF motion without using an Inverse Kinematics library, such as mink, which can slow down simulation speed since it adds additional CPU computation requirements to the task.
  • Directly controlling a tool in our benchmark provides the user with (up to) 3 translational DoF and 3 rotational DoF, which added together, is on-par with the 6 DoF action space of a UR5E Robot Arm from the MuJoCo Menagerie. Assuming there is no grasping of the tool involved, and each tool’s canonical pose remains the same (as in our simulation), we can directly transfer the learned policies from our benchmark onto real robots.
  • In comparison to the above “robotic tool design” work, we provide more tools and tasks, and include dense reward functions for training RL policies. Additionally, the constrained action spaces we introduce in our work are important and carefully designed to address the constraints and safety requirements of each tool. Please see our website, mujoco-manipulus.github.io, which includes qualitative videos showing successful policy rollouts for each of our 16 tasks. There, you can see that the tool behavior exhibits natural motions and realistic physical simulation of all tools + objects. We believe this justifies the design of our constrained action spaces and reinforces the realistic behavior of tools in our benchmark.

Pt. 2 “Despite the claim of offering a "diverse set of challenging tool manipulation tasks," the benchmark consists of only 14 tasks, with a significant portion focused on pouring. This narrow focus could limit the utility of the benchmark for evaluating a broader range of tool manipulation skills.”

• We have updated our benchmark, which now includes 16 tool manipulation tasks and 14 tools from either MuJoCo or the TACO Dataset. More details about our tasks and tools are in our updated paper in Section 3.2. As mentioned in this section, the tools we provide can be broken down into 3 general categories: Kitchen Tools, Home Tools, and Sports Tools.
  • Our “Kitchen Tools” category includes models for Bowl, Mug, Knife, Pan, Pot, Plate, Spatula, Ladle, and Cup objects.
  • Our “Home Tools” category includes models for a Brush, Hammer, and Scooper.
  • Our “Sports Tools” category includes models for a Ping-Pong Paddle and Golf Club.
• We believe this is a strong representative set of tools to begin tool manipulation research with in a unified setting, and hope this addresses your specific concerns about whether our benchmark has diverse tool manipulation tasks.
• Re: The difficulty of our tasks, we refer you to our paper’s updated Section 4.2, where we discuss the challenges in our more difficult tool manipulation tasks.
• For additional tasks that are not included in our benchmark, as-is the case with other simulation benchmarks like ManiSkill, we will continue to update our benchmark with additional tasks. We will also open-source our benchmark, and encourage the community to make contributions with our task design guidelines to introduce new simulation tasks with our framework.

Pt. 3 Reviewer is concerned about whether ICLR is the most appropriate venue for this work

• ICLR supports a Datasets and Benchmarks “Primary Area” for paper submissions this year. We also highlight that robotic simulation works like ManiSkill2 have been accepted in prior ICLR conferences. Hence, we believe that our work has strong merit and relevance to ICLR.

Thank you for your review of our work. Here, we address your concerns:

Pt. 1 The novelty of our benchmark

• Please read through our detailed General Response to all reviewers, where we address this concern.

Pt. 2 “The content is incomplete, with multiple paragraphs repeating similar ideas. While the state space for different tasks is described, there is no mention of the action space, reward function, or terminal states.”

• Please read through our updated Sections 3, 4, and Appendix of our paper. These sections introduce new figures, results, and detailed discussions of {observation spaces, action spaces, environment resets, dense reward design, success conditions}.
• Regarding terminal states, all tasks are 200 episodic steps, with the exception of Ping Pong, which is 50 episodic steps due to its short nature, and there are no terminal states which cause the episode to end early. All results are from episodes that last 200 or 50 episodic steps. We are concerned with ensuring that tools remain in a “success” state during the entire episode, and for this reason, we do not introduce early episode termination in our environments.

Pt. 3 “There is no comprehensive comparison with previous benchmarks, leaving the unique contributions of this benchmark unclear.” / Pt. 4 “The number of objects or tools available for each task is limited, which restricts the potential for in-category object manipulation learning.”

• We added Table 1 + Appendix Section A.1 to comprehensively compare our work with existing works. In Sections 2 and 3 of our updated paper, we discuss the merits of our work over existing works too.
• Related to Pt. 4, {Section 3, Table 1, Appendix Section A.1} address the in-category object variants for each of our tasks, as well as the tool skills that can be learned in our benchmark compared to existing benchmarks. There are additional details we provide, such as how we have dense rewards for all our tasks, and a diverse set of tools compared to existing works.

Q1 Could different robot manipulators be added to the simulator? Most robotic arms are limited by their mechanical design and current motion planning algorithms, making it challenging to follow the trajectory of floating objects.

• Yes, we can add different robot manipulators to our simulator, and plan to do this in the near future.
• We repeat the following content from our response to Reviewer P8eT:
  • Although there are no robots, this does not mean our benchmark is incomplete, or that it does not support Sim2Real research efforts.
    • Direct control of tools is a common way to test tool manipulation, as demonstrated in related work such as https://robotic-tool-design.github.io/. The idea of using floating manipulators is not new either, as shown in the popular Adroit Manipulation Platform, and is a popular method for simulating 6-DoF motion without using an Inverse Kinematics library, such as mink, which can slow down simulation speed since it adds additional CPU computation requirements to the task.
    • Directly controlling a tool in our benchmark provides the user with (up to) 3 translational DoF and 3 rotational DoF, which added together, is on-par with the 6 DoF action space of a UR5E Robot Arm from the MuJoCo Menagerie. Assuming there is no grasping of the tool involved, and each tool’s canonical pose remains the same (as in our simulation), we can directly transfer the learned policies from our benchmark onto real robots.
    • In comparison to the above “robotic tool design” work, we provide more tools and tasks, and include dense reward functions for training RL policies. Additionally, the constrained action spaces we introduce in our work are important and carefully designed to address the constraints and safety requirements of each tool. Please see our website, mujoco-manipulus.github.io, which includes qualitative videos showing successful policy rollouts for each of our 16 tasks. There, you can see that the tool behavior exhibits natural motions and realistic physical simulation of all tools + objects. We believe this justifies the design of our constrained action spaces and reinforces the realistic behavior of tools in our benchmark.

Thank you for your review of our work. Here, we address each of the Weaknesses you mentioned:

Pt. 1 Although the authors motivated this benchmark for robot learning and manipulation research, this paper did not use realistic robot models or controllers. Instead, it used an idealized "point mass" robot with simple Cartesian action spaces. This design severely abstracts away the challenges of contact-rich robot manipulation problems in the real world. Compared to existing manipulation benchmarks, such as ManiSkill2 (Gu et al., 2023) and Robosuite (Zhu et al., 2020), which aimed to model robot dynamics faithfully, the simplified robot model could hinder the ability to deploy the learned models on physical robots and thus limit its impact in advancing robot learning research.

• We also discussed the reasoning behind our Floating Tool Control model with other reviewers who shared this concern, and provide the following response:
  • Although there are no robots, this does not mean our benchmark is incomplete, or that it does not support Sim2Real research efforts.
  • Direct control of tools is a common way to test tool manipulation, as demonstrated in related work such as https://robotic-tool-design.github.io/. The idea of using floating manipulators is not new either, as shown in the popular Adroit Manipulation Platform, and is a popular method for simulating 6-DoF motion without using an Inverse Kinematics library, such as mink, which can slow down simulation speed since it adds additional CPU computation requirements to the task.
  • Directly controlling a tool in our benchmark provides the user with (up to) 3 translational DoF and 3 rotational DoF, which added together, is on-par with the 6 DoF action space of a UR5E Robot Arm from the MuJoCo Menagerie. Assuming there is no grasping of the tool involved, and each tool’s canonical pose remains the same (as in our simulation), we can directly transfer the learned policies from our benchmark onto real robots.
  • In comparison to the above “robotic tool design” work, we provide more tools and tasks, and include dense reward functions for training RL policies. Additionally, the constrained action spaces we introduce in our work are important and carefully designed to address the constraints and safety requirements of each tool. Please see our website, mujoco-manipulus.github.io, which includes qualitative videos showing successful policy rollouts for each of our 16 tasks. There, you can see that the tool behavior exhibits natural motions and realistic physical simulation of all tools + objects. We believe this justifies the design of our constrained action spaces and reinforces the realistic behavior of tools in our benchmark.

Pt. 2 The authors motivated the benchmark as a "wider-scale tool manipulation benchmark with substantially more tasks." However, the current offering of the tasks is fairly limiting compared to several existing simulation frameworks for robot navigation and manipulation (Nasiriany et al., 2024; Szot et al., 2021; Li et al., 2022), which provides 100-1000 tasks in large-scale home environments. In contrast, the proposed benchmark only provides 14 simplified tasks in a restricted tabletop workspace.

• Robot Navigation and Embodied AI works, which the reviewer has mentioned, are focused on high-level planning for robots. Our benchmark is for low-level control of tools and robots, with an emphasis on RL algorithms for learning low-level control. Similar benchmarks to our work are included in Table 1 of our paper and discussed in Section 2 of our paper, and we provide a comparison our Benchmark’s Tool Use Skills compared to other benchmarks in Appendix Section A.1. We also include details of our low-level tool control tasks in Section 3 of our work.
• For manipulation benchmarks such as RLBench, and Embodied AI works that include Navigation and Mobile Manipulation, they provide no dense rewards for guiding RL policy learning. RL algorithms are typically applied to low-level control problems like those found in our work and related benchmarks such as ManiSkill and Robosuite. In the case of RLBench, while they provide many manipulation tasks, they only provide sparse rewards, which are hard to learn low-level control policies with using plain model-free RL algorithms. We’d like to mention that our benchmark jointly includes dense rewards and sparse rewards for all tasks.

Thank you for your review of our work. Here, we address your concerns:

Pt. 1 The size of the benchmark is relatively small

• The updated size of our benchmark is 16 tasks, and we provide 14 tools in our benchmark. For comparison to other benchmarks, please see Table 1 and Appendix Section A.1 in our updated paper. Our benchmark provides Dense Rewards for all tasks, a critical feature for guiding RL policy learning. Compared to existing benchmarks such as ManiSkill and Robosuite, which also provide Dense Rewards, we are on-par with the size of ManiSkill 3’s tabletop manipulation task suite (16 tasks) and larger than Robosuite (9 tasks) while offering more tool interaction options than both benchmarks.

Pt. 2 The complexity of these tasks is relatively low. The agent has direct control over the tools (non-dexterous) and requires only a few degrees of freedom to move them.

• We discuss our more challenging tool manipulation tasks in Section 4.2 of our updated paper, and in our General Response to all reviewers. We also discuss the reasoning behind our Floating Tool Control model with other reviewers who shared this concern, and provide the following response:
  • Although there are no robots, this does not mean our benchmark is incomplete, or that it does not support Sim2Real research efforts.
  • Direct control of tools is a common way to test tool manipulation, as demonstrated in related work such as https://robotic-tool-design.github.io/. The idea of using floating manipulators is not new either, as shown in the popular Adroit Manipulation Platform, and is a popular method for simulating 6-DoF motion without using an Inverse Kinematics library, such as mink, which can slow down simulation speed since it adds additional CPU computation requirements to the task.
  • Directly controlling a tool in our benchmark provides the user with (up to) 3 translational DoF and 3 rotational DoF, which added together, is on-par with the 6 DoF action space of a UR5E Robot Arm from the MuJoCo Menagerie. Assuming there is no grasping of the tool involved, and each tool’s canonical pose remains the same (as in our simulation), we can directly transfer the learned policies from our benchmark onto real robots.
  • In comparison to the above “robotic tool design” work, we provide more tools and tasks, and include dense reward functions for training RL policies. Additionally, the constrained action spaces we introduce in our work are important and carefully designed to address the constraints and safety requirements of each tool. Please see our website, mujoco-manipulus.github.io, which includes qualitative videos showing successful policy rollouts for each of our 16 tasks. There, you can see that the tool behavior exhibits natural motions and realistic physical simulation of all tools + objects. We believe this justifies the design of our constrained action spaces and reinforces the realistic behavior of tools in our benchmark.

Pt. 3 The realism is fairly limited, while at the same time the simulator uses Mujoco on the CPU. Most recent robotics benchmarks are moving toward either greater realism or very-high-throughput simulation on the GPU, and so this benchmark, which lacks both, feels somewhat behind the Pareto frontier.

• Please see Point #6, “Is Our Benchmark on the Pareto Frontier of Robotics?”, in our General Response to all reviewers. We believe this is an important topic to mention to all reviewers, and thank the reviewer for bringing this concern to our attention.

Pt. 4 It's not clear how challenging this benchmark is. The authors run PPO and SAC on these tasks and show that they struggle to solve them out of the box. In many robotics applications though, a large degree of reward shaping or learning from demonstrations is frequently necessary to solve real world problems, and it's not clear why those approaches wouldn't work here as well.

• Please see our updated Figure 9 in our paper, which shows the Success Rate curves for all 16 of our tasks. In short, we applied 3 model-free RL baselines to our tasks, and were able to learn meaningful behaviors across all tasks with ≥ 1 baseline algorithm. For qualitative visualizations of the best-performing policies for each task, please visit mujoco-manipulus.github.io. The results we achieve are thanks to additional Reward Engineering, which we include extensive discussion of in our new Appendix Section A.4.

Thank you for your review of our work. Here, we address each of the Weaknesses you mentioned:

Pt. 1 The benchmark lacks completeness / Pt. 3 Is True Physical Simulation involved?

• We are the first simulation benchmark to present a complete packaging of 14 tools and 16 tool manipulation tasks, all with an easy-to-use Reinforcement Learning interface for robotics and RL practitioners.
• Although there are no robots, this does not mean our benchmark is incomplete, or that it does not support Sim2Real research efforts.
  • Direct control of tools is a common way to test tool manipulation, as demonstrated in related work such as https://robotic-tool-design.github.io/. The idea of using floating manipulators is not new either, as shown in the popular Adroit Manipulation Platform, and is a popular method for simulating 6-DoF motion without using an Inverse Kinematics library, such as mink, which can slow down simulation speed since it adds additional CPU computation requirements to the task.
  • Directly controlling a tool in our benchmark provides the user with (up to) 3 translational DoF and 3 rotational DoF, which added together, is on-par with the 6 DoF action space of a UR5E Robot Arm from the MuJoCo Menagerie. Assuming there is no grasping of the tool involved, and each tool’s canonical pose remains the same (as in our simulation), we can directly transfer the learned policies from our benchmark onto real robots.
  • In comparison to the above “robotic tool design” work, we provide more tools and tasks, and include dense reward functions for training RL policies. Additionally, the constrained action spaces we introduce in our work are important and carefully designed to address the constraints and safety requirements of each tool. Please see our website, mujoco-manipulus.github.io, which includes qualitative videos showing successful policy rollouts for each of our 16 tasks. There, you can see that the tool behavior exhibits natural motions and realistic physical simulation of all tools + objects. We believe this justifies the design of our constrained action spaces and reinforces the realistic behavior of tools in our benchmark.

Pt. 2 Novelty and Contribution are severely concerned / Pt. 4 Current Simulation Benchmarks can support these tasks by simply importing the relevant 3D objects

• Please see Figure 2 in our updated paper, and read through our renovated Section 3 of our paper for a detailed discussion of our benchmark’s design pipeline. In short, it is not as simple as importing the relevant 3D objects into the simulator. We additionally have added an Appendix with Reward Engineering details, and a new Table 1 + Appendix Section A.1 in our paper with a comparison of our benchmark to existing simulation benchmarks.
• We also hope that the General Response to all reviewers addresses your concerns about the novelty of our work, and why we chose MuJoCo instead of a different simulation backend (see Point #6 in our General Response to all reviewers).

Thank you for the clarification. I appreciate the visualization (the website) and the additional results. However, I still encourage the authors to integrate robots into the benchmark. Even the listed related works in this rebuttal have either a full Franka or a dexterous hand, not a floating point like this paper.

For policy learning, it's possible to use a 6DoF point to learn trajectory, but for a robotic benchmark paper, it must contain a robotic embodiment (at least an end-effector). The sim-to-real gap is not that trivial, from a robotic side, even a photo-realistic simulator with accurate models of robots can cause problems when transferring from sim to real.

Therefore, a possible way to improve this paper is to import robots, or to consider it as a dataset paper providing tool using demos. An exception to this is that you are trying to tackle super-hard problems, such as simulating soft object manipulation, where you can focus on the object itself, not the robotic side. But obviously, tools are rigid objects, the difficulties is in robotic grasping and manipulation. I am afraid that it cannot be accepted by robotics-related conferences before importing robots (at least EE) into the benchmark.

Please consider improving it with these suggestions.

5. Differentiating our Simulation Benchmark from Existing Works:

   • Our benchmark is a first-of-its-kind tool manipulation benchmark. This is a critical area in robotics since it enables robots to make use of external items to accomplish tasks that are difficult with native hardware.
   • In comparison to other benchmarks, such as Robosuite, Meta-World, and ManiSkill2, we provide our users with 16 tasks and 14 tools that are not included in these works, all in a unified setting.
   • Please see Table 1 in our paper, which provides a comparison of our benchmark with existing simulation benchmarks, by comparing # of Tasks, Availability of Dense Rewards, # of Tool Skills in each benchmark, and Simulation Engine backend for each benchmark. We additionally provide a list of tool skills in each simulation benchmark, and compare against ours, in our Appendix in Section A.1.
   • We hope this breakdown provides reviewers with a greater understanding of how our work sets a new bar for what can be achieved in simulation, and why our tool manipulation benchmark will enable greater research to be done in the areas of RL, Vision, and Robotics.

6. Is Our Benchmark on the Pareto Frontier of Robotics?

   • Speed of MuJoCo:

     • This point, raised by Reviewer eoGS, is important to address and we want to include this in our detailed general response to all reviewers. Unlike ManiSkill or Isaac Lab, which use NVIDIA PhysX to simulate physics, we use MuJoCo’s highly-optimized CPU backend. More specifically, we use MuJoCo 3.0+ in our benchmark, which we have plans to soon integrate with Brax and use the MuJoCo XLA (MJX) backend for GPU-simulated physics. At the time MuJoCo 3.0 was announced, the following results were published in this discussion thread on Github:

       Platform	Time to train PPO at 60M Steps
       MJX + Brax on TPU v5e-8	249 seconds
       MJX + Brax on Nvidia A100 GPU	718 seconds
       Nvidia Isaac Gym on A100 GPU	600 seconds

     • These results are from the following experiment “…Google’s Barkour robot [learned] to run using MJX and Brax’s PPO trainer. We train a policy that can be deployed on a real robot and compare the training time to the Barkour publication, which used Nvidia Isaac Gym.” With the MJX + Brax Barkour environment, and using a TPU, PPO trained for 60 Million Steps in 249 seconds (4 minutes and 9 seconds) vs Isaac Gym taking 600 seconds on an A100 GPU (10 minutes).

     • Since our environments are all implemented in MuJoCo, and will soon be integrated with MJX + Brax when Batch RGB rendering is supported for fast Visual RL, we will be the first MuJoCo-based and GPU-based simulation benchmark to support the variety of tasks in our task distribution, all with GPU accelerated physics, and we will be faster than Isaac Gym (based on NVIDA PhysX backend).

   • Sim-to-Real Potential and Accuracy of MuJoCo:

     • In a recent work which benchmarked the Sim-to-Real capabilities of policies trained in MuJoCo, Bullet, NVIDIA Flex, and SOFA, the authors made the following conclusion from their extensive experiments:

       “Given the lower distances in both dynamic and quasi-static manipulation tasks shown by MuJoCo, as well as its capability of integrating robotic models, and availability of both CPU and GPU acceleration, we recommend MuJoCo for learning cloth manipulation tasks in a simulation engine.”

     • In future updates of our work with tool manipulation, we plan to incorporate dynamic manipulation tasks, and will be using the current-best simulation benchmark for accurately simulating dynamic bodies and transferring the learned policies to real-world robots.

We hope this summary of updates has addressed as many of the reviewers’ concerns as possible. For specific reviewer questions, we also provide reviewer-specific responses. We once again thank all the reviewers for their valuable feedback.

3. Updated Baselines

   • Related to Update #1, “Updated Results w/ New Dense Reward Functions”, we have updated 3 components of our baselines
   1. Component #1: RL Algorithm Implementations
      • Previously, we used Stable-Baselines3 for our results. However, our updated results now use RL algorithms from CleanRL, a popular RL algorithm library with high-quality single-file implementations with research-friendly features.
   2. Component #2: Added CrossQ algorithm as Baseline
      • We added CrossQ, a highly sample-efficient model-free RL algorithm which consistently provided competitive results across all of our tasks (not including Ping Pong, which we discussed briefly in our paper in Section 4.2 and in our rebuttal response in Section 1).
   3. Component #3: Removed RGB Training Results
      • We provide results for all baselines using State Inputs, and we removed RGB Training Results from our work. We made this decision because Visual RL has 2 limitations in our work:
        1. MuJoCo CPU Training Time / Limitations of MuJoCo XLA (MJX): 100,000 steps of training with state inputs takes 10 minutes of wall-time (roughly), and 300,000 steps of training with state inputs takes 30 minutes of wall-time. Unlike GPU simulated benchmarks, such as Isaac Lab and ManiSkill, which are both based on the NVIDIA PhysX physics simulation backend, we do not have a GPU simulated benchmark. MuJoCo XLA (MJX), a JAX-based implementation of MuJoCo which simulates physics on the GPU, currently does not support batch RGB rendering, and this limitation is an out-of-scope contribution for our work which prevents us from training fast Visual RL policies in MuJoCo simulation. Despite this limitation, we are committed to implementing our tasks with MuJoCo XLA (MJX) using the Brax framework in the future, once Batch RGB rendering is supported with the (currently) experimental “Madrona MJX” batch renderer.
        2. Upper Bound of Performance: Our benchmark focuses on whether we can learn tool manipulation policies with RL, and we use state information from MuJoCo as inputs to our policies since policies trained with state inputs can often represent the upper bound of performance for a given baseline. In the context of our work, the upper bound of performance means “which policy can achieve the highest success rate for a task in the fewest # of total episodes of experience during training”. Referring to our Table above, a policy learned for Scoop Particle with CrossQ+State Inputs in 1500 episodes of training experience is the upper bound of performance for that task.

4. Design Choices for Action Space and Observation Space:

   • Several reviews raised concerns about our simple action spaces, and details of our observation space. We agree that these design considerations are important, and have updated our paper to include a detailed discussion of these items:
     1. Action Space and Observation Space details are provided for each task category in Section 3 of our paper.
     2. In summary, Tool Manipulation tasks have several constraints in real-world settings, and by constraining our Action Spaces, we simplify the RL problem while still learning the correct skills associated with the tool.
     3. We provide qualitative videos at mujoco-manipulus.github.io for all 16 of our tool manipulation tasks to showcase the effectiveness of our learned policies. Each task exhibits the correct behavior, and benefits from learning a simpler optimization problem by using a constrained action space design.