Thank you for your feedback! We have addressed your comments below.

Novelty of Our Paper

We are grateful to your review on our contribution: "the substantial contribution to the community".

As for the novelty of our paper, as our paper proposes the environment and benchmark for robotic manipulation, the novelties mostly lie in the comparisons with other environments and benchmarks.

First, garment manipulation is a novel research direction. While previous studies mostly provide environments for rigid objects, or relatively simple deformable objects (e.g., cables, rings and fabrics), our work first comprehensively provides the simulation and benchmark for the novel garment manipulation direction.

Second, the diversity of objects, interactions and tasks is our novelty. While previous environments mostly only support the simulation of a certain type of object or interaction, our work not only supports simulation for rigid and articulated objects, but also supports different simulation methods for different deformable objects. For example, Position-Based Dynamics (PBD) [1] for shirts, Finite Element Method (FEM) [2] for hats and gloves, and even the simulation of fluid and flow. For these diverse object simulation methods, we further support the interaction coupling among them. The simulation of these diverse objects and their interactions provides a rich environment for various tasks, which has not been achieved in previous benchmarks.

Additionally, while previous benchmarks mostly focus on evaluation in simulation, our work provides a systematic sim-to-real pipeline and the real-world benchmark, paving way to the real-world deployment, comparison and benchmarks.

[1] Bender, et al. "Position-Based Simulation Methods in Computer Graphics." Eurographics (tutorials). 2015.

[2] Ciarlet, et al. The finite element method for elliptic problems. Society for Industrial and Applied Mathematics, 2002.

We hope that we have clarified our novelty well. We are happy to take further questions from you!

Thanks for your valuable comments! We have addressed them below, and hope to hear back from you if you have further questions!

Code Release

These mentioned components with the guidelines for installation and usage have been already made publicly available. Please refer to the Global Response for more details. Welcome to have a try!

Interactions among Multiple Garment Layers

GarmentLab supports simulating the interactions among different layers of garments, and these layers really affect each other. We use position-based dynamics (PBD) [1] to simulate garments with multiple layers as a whole to achieve this feature. As demonstrated in Figure 2 in the Attached Rebuttal PDF, the garment consists of multiple layers, and when the robot is trying to grasp and lift the front layer (from the left sub-figure to the right sub-figure), other layers will have the effects resulted from the robot's operation on the grasped front layer.

Interactions between Garments and Fluid

GarmentLab supports simulating interactions between garments and fluid, and generating the dynamic sequences at the same time. The engine simulates garments and fluid using the same simulation method, position-based dynamics (PBD) [1], and thus supports this feature. As demonstrated in Figure 3 in the Attached Rebuttal PDF, in the first four sub-figures, we adjust the weight of the garment particles to make the garment respectively floats on the water, suspends in the water, and sinks to the bottom. This interaction requires importing both garment particles and fluid particles into the same particle system, and then generating dynamic sequences simultaneously. In the first four images, we adjust the weight of the clothing particles to make the clothes float on the water, suspend in the water, and sink to the bottom. Besides, we can also simulate clothing and fluid in different particle systems separately. As shown in the last sub-figure, the garment and fluid do not have interactions in this case. Additionally, we can adjust simulation parameters, such as particle contact offset and fluid rest offset, to simulate different interaction effects between garments and fluid. You can refer to Section D in Appendix for more details.

Customized Dynamics

GarmentLab supports customized dynamics. We can customize the dynamics in two ways. The first way to use customized meshes of garments or deformable objects. For example, in the left of Figure 4 in the Attached Rebuttal PDF, we import a garment mesh from our simulation assets and a deformable object mesh from our real-world benchmark. These two objects use different simulation methods: the garment uses Position-Based Dynamics (PBD), while the deformable object uses the Finite Element Method (FEM)[2]. We can simulate the interactions between these two objects. This demonstrates that we can combine simulation objects and scanned objects from the real world. The second way is to adjust the parameters of the physical simulation to change the effect of object simulation. You can refer to Figure 3 in the Main Paper to see the effects of physics and Section D in Appendix for the physical parameters that can be adjusted. In summary, our GarmentLab is very flexible. It supports customized meshes of deformable objects and garments, and also allows for adjusting object parameters to modify the physical simulation effects.

[1] Bender, et al. Position-Based Simulation Methods in Computer Graphics. Eurographics (tutorials). 2015.

[2] Ciarlet, et al. The finite element method for elliptic problems. Society for Industrial and Applied Mathematics, 2002.

Thank you again for your constructive feedback! As the discussion period is coming to an end, we want to know whether our responses have addressed your concerns, and if you have further questions. Thank you!

Thanks for your valuable comments! We have addressed them below, and hope to hear back from you if you have further questions!

Efficiency of the Benchmark

To the best of our knowledge, GarmentLab is the first to support all of ray tracing, robot control simulation, and diverse deformable object simulation methods in a whole environment. Therefore, it is a little bit difficult to directly compare the speed and memory of our environment with those of other environments in a fair setting. For example, SoftGym [1] and PlasticineLab [2] do not support real robot control (they only support pickers or manipulators as robots), which requires solving inverse kinematics and controlling the robot. FluidLab [3] does not support real time tracing, which is important for realistic visual effects. Furthermore, the above environments only support a single type of deformable object simulation method.

While our environment supports all of the above features, we also take the speed and efficiency of the environment into account. GarmentLab is GPU-based, placing all simulations of deformable objects and real-time tracing on the GPU, while handling robot-related inverse kinematics on the CPU. This improves the simulator's speed and avoids the bottleneck of CPU-GPU data transfer. Moreover, to improve data collection performance, we support importing multiple robot-object interaction scenarios in the same simulation environment for parallel data collection. Please refer to 2:37 to 2:41 in the Supplementary Video for the demonstrations of simultaneous multi-scenario data collection.

We have designed two settings to compare the speed of GarmentLab (with and without the multi-scenario data collection method respectively denoted as GarmentLab-Multiple and GarmentLab-Single) and SoftGym. First, we evaluate the speed of the robot reaching the target garment and then grasping the garment. Second, we evaluate the speed of the robot directly grasping the garment in the near front without other movement. The following tables respectively show the speed comparisons in these two settings, demonstrating that GarmentLab, with the designed multi-scenario data collection method, is comparable and a little more efficient than SoftGym in terms of simulation for data collection.

Thanks again for this valuable comment! This is really significant for a robotic manipulation environment. We will add the above descriptions, comparisons and tables in the final version of our paper.

[1] Lin, et al. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. CoRL, 2021.

[2] Zhou, et al. Fluidlab: A differentiable environment for benchmarking complex fluid manipulation. ICLR, 2023.

[3] Huang, et al. PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics, ICLR, 2021.

More Input Modalities

Thanks for this valuable comment! The current engine supports tactile detection for rigid-rigid interaction. As demonstrated in Figure 4 (Right) in the "Attached Rebuttal PDF", we replaced the Franka gripper with the Trossen's X300 gripper and installed tactile sensors on its inner side, and can detect the tactile feedback including normal and friction points and forces between the robot manipulator and the rigid object using SDF collision. With this feature, we are able to design interesting tasks like using a rigid stick to nudge and adjust the posture of garments. The clothing provides force feedback to the stick, which is then perceived by the tactile sensors attached to the manipulator. Furthermore, we are actively extending and developing this feature into directly sensing the tactile feedback of garments, which could largely support more interesting and fine-grained manipulation tasks, and we are willing to present such features in our future work, to continuously make contributions to the community.

For other modalities, the engine also supports IMU, Lidar, Radar, Ultrasonic and Generic Range sensors, while these sensors are rarely used in garment manipulation tasks. If some tasks require such modalities, we are super willing to extend such sensory signals to our proposed envrionment.

Thanks for your valuable comments. We have addressed them below, and hope to hear back from you if you have further questions!

Setup of Real-World Evaluation

Thanks for the appreciation of our proposed real-world assets, and pointing out that the real-world evaluation setup is not detailed enough. We have attached detailed descriptions (task, setup, hardware requirement, execution procedure, evaluation methods, etc.) about the real-world implementation protocols in the released document of GarmentLab. Specifically, the Protocol sub-section in the Real-World Benchmark section of the released document.

Thanks again for this valuable comment! We hope that the real-world benchmark can really help the community to evaluate the real-world performance of garment manipulation methods.

Clarifications on Figure 6

For the color scheme of Figure 6, it denotes the per-point correspondence between garments. According to [1], when the learned representations of two points in two garments are similar, the point colors would be similar. Such per-point correspondence is a good way to both (1) evaluate and visualize the learned representation of garments in an intuitive way, and (2) further guide the manipulation of novel garments within few-shot demonstrations in previous garments, by selecting the manipulation points on the novel garment, which are the most similar to those in the demonstrations garments. More details of such representation can be found in [1]. When adding sim-to-real methods (Noise Injection for Domain Randomization, Corresponding Keypoint Representation Alignment, Point Cloud Alignment between Simulation Garments and Real-World Scans), the learned representations, reflected by the color scheme, can be more continuous and smooth across different point on the different garments.

We have revised Figure 6 (as Figure 1 in the "Attached Rebuttal PDF"), showing the procedures of three sim-to-real methods on the left of the green block, and there corresponding effects on the learned representations of garments (making the learned representations more continuous and smooth, with better correspondence across different garments) on the right of the green block.

Thanks again for this valuable comment! We are sparing no effort to make our paper more clear.

Details of Code and Usage

As described in the "Global Response", we have made GarmentLab public, with the documents of installation and usage. Welcome to have a try!

We are very grateful for your tips, and we have taken the details you mentioned into account when designing the benchmark. For UI design, we adopted the Isaac Sim UI interface. As a mature, industrial-grade simulator, Isaac Sim's UI is easy to operate, as verified by practical tests, making it much more user-friendly than most other simulators' basic interfaces. Moreover, it supports moving with the mouse during operation, which is something most simulators [2, 3] cannot do. Additionally, the functions that can be achieved in the UI interface can easily be extended to code APIs, and we have provided detailed instructions on how to use the APIs in the code and documentation, with many example tasks to control the robot and achieve the goal. For debugging tools, Isaac Sim and Omniverse already have a very comprehensive warning and error reporting system. On this basis, we have added numerous assertions at key points in our code to ensure that user input meets the requirements and to ensure that users correctly understand how to use the API. It is worth noting that, as Isaac Sim is still under active development, it sometimes encounters segmentation faults without detailed descriptions. We have included more detailed error or warning statements in our high-level code. This way, users can quickly locate the problem when an error occurs.

Descriptions on PyBullet

Thank you for pointing out that [4] based on PyBullet also supports the simulation of garment manipulation tasks. Compared to Pybullet based methods, our proposed GarmentLab has the following advantages:

1. GarmentLab supports the coupling of garments simulated by different simulation method (PBD and FEB), and the interactions between garments with different materials (e.g., hat, gloves and shirts). On the contrary, previous works with garments simulated in PyBullet could only support one kind of simulation, without the simulation of multi-material interactions.
2. GarmentLab includes GPU acceleration for simulation in the large scale, which supports collecting large scale data and training in an efficient manner.
3. Equipped with high fidelity GPU based PhysX engine, GarmentLab is capable of supporting multi-sensor RTX rendering at an industrial scale.
4. While [4] only demonstrated 4 representative tasks on only a few categories of garments, we present and benchmark 11 garment categories with much more diverse tasks.

We will include the above descriptions in the final version of the paper.

Typos

Thanks a lot for pointing out this. We have thoroughly checked our paper, and corrected them.

[1] Wu, et al. UniGarmentManip: A Unified Framework for Category-Level GarmentManipulation via Dense Visual Correspondence. CVPR 2024.

[2] Lin, et al. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. CoRL, 2021.

[3] Zhou, et al. Fluidlab: A differentiable environment for benchmarking complex fluid manipulation. ICLR, 2023.

[4] Antonova, et al. Dynamic Environments with Deformable Objects, NeurIPS Datasets and Benchmarks Track, 2021.