DexScale: Automating Data Scaling for Sim2Real Generalizable Robot Control

Dear Reviewer, we sincerely appreciate your constructive feedback. We hope that the following response can address your concerns:

1. "...substantial manual adjustments are required to align the hand and object models within the simulation before retargeting them to the end-effector pose. ... Figure 7 further suggests that this approach is labor-intensive and may not be scalable."

DexSim features an automated pipeline for extracting human hand poses from videos and aligning them with objects without any manual intervention. The joint optimization process (see line 222, right column) between the hand and object models is fully automated. Specifically, similar to Liu et al. (2024), we utilize a learning-based approach to iteratively refine and synchronize hand-object interactions over multiple rounds until they are properly aligned. We have clarified this in the revised version of our study.

(Liu et al., 2024) Liu, Yumeng, et al. "EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild." arXiv preprint arXiv:2411.14280 (2024).

2. "...The performance differences between simulation and real-world results for skills+DR and DexSim are minimal ..."

Thank you for pointing this out. In our experimental results, we observed that the performance gap between Skill+DR and DexSim is less pronounced in the object grasping task. One major contributing factor is the limited number of trials conducted in both real-world and simulated environments. To address this, we have increased the number of experiment runs. The updated results are shown below:

To further support our findings, we have included the relevant experiment videos in Section 3.1 of our anonymous link.

3. "A more robust evaluation—potentially using large-scale simulation environments such as the Factor-world or The Colossuem Benchmark, which incorporate perturbation variations—would strengthen this claim."

Our study focuses on real-world experiments, demonstrating improvements in generalization performance on physical robots rather than in simulated environments. Specifically, our experiment on generalizability (Section 5.1) addresses the challenge of bridging the Sim2Real gap. We show that the automatic domain randomization (DR) and domain adaptation (DA) mechanisms in DexSim effectively transfer policies learned in simulation to real-world applications.

In contrast, benchmarks that study generalization across discrepancies between training and testing environments, such as the Factor-World or the Colosseum Benchmark, still evaluate performance in simulated settings. However, as we argue in Section 3, the Sim2Real gap presents a significant challenge. There is no guarantee that a policy performing well in these benchmarks can be successfully deployed in real-world scenarios.

4. "The method itself is problematic due to its many moving parts, where a failure in any single component can undermine the entire process."

The term "moving parts" in the review seems confusing. Generally, in methods implementing the Real-to-Sim-to-Real pipeline, the presence of multiple components is a fundamental characteristic. In the case of DexSim, these components are not only necessary but also integral to the pipeline's functionality. Similarly, methods like RoboGen and GenSim also adopt pipelines composed of multiple interconnected parts. Therefore, the presence of "many parts" is not unique to this method but rather a common feature of such approaches.

5. "... there are alternative methods for improving sim-to-real transfer—such as ASID, SystemID, and GenAug..."

While these studies are indeed inspiring, we found that their methods are not directly comparable to ours. Specifically:

• ASID focuses on refining simulation models to better represent real-world dynamics. In contrast, DexSim interacts with a given dynamics model without access to its underlying function.
• SystemID addresses system identification by recovering nonlinear models of dynamical systems from data. However, it does not consider simulated environments or skill discovery methods relevant to our study.
• GenAug appears to be a data augmentation technique leveraging image-text generative models. Notably, its augmentation approach enforces action invariance. In contrast, DexSim can autonomously refine actions when the target object changes.

6. "Figure 2 includes numerous elements that are not adequately explained in the caption."

The elements in the figures have a one-to-one correspondence with the titles of the subsections and paragraphs in Section 4: Data Engine for Sim-to-Real Generalization. This relationship has been clarified in the revised version of our study.

Dear Reviewer, We sincerely appreciate your constructive feedback and thank you for recognizing the significance of our work. We have carefully considered your suggestions, and we hope that the following response can address your concerns:

1. "...I only see simple tasks like grasping and open a box. These tasks are relatively fault tolerant comparing more dextrous task like opening a cabinet door via the handle. Could you provide evaluation on more dexterous tasks?"

Our experiment comprises three distinct tasks, each involving a unique embodiment. To further support our argument—and in response to the reviewer's request—we additionally demonstrate DexSim's performance on two representative tasks: articulated object manipulation (e.g., opening a drawer) and combined grasping and manipulation (e.g., water pouring). Detailed results and examples can be found in Section 2.1 of our anonymous link.

Additionally, we emphasize that the tasks demonstrated in our experiment are inherently challenging. For instance, the box-opening task requires the robot to sequentially open four flaps, demanding precise control and planning to complete the task successfully. Even more complex is the re-arrangement task, where the robot must reorient both the fork and spoon so they face the front of the plate, while also positioning them accurately around it. Smooth execution of this task necessitates coordinated movement between both arms, ensuring that the utensils are placed correctly and simultaneously.

2. "Why not use existing platforms like MuJoCo or PyBullet?"

Choosing a simulation platform involves balancing several factors: rendering quality, physics accuracy, overall simulation efficiency, and the flexibility and scalability of the simulator.

Unfortunately, we have not found an existing open-source simulator that satisfies all of these requirements. For example, PyBullet and MuJoCo offer strong physics simulation capabilities, but their rendering quality is limited. On the other hand, Blender excels at rendering but is not designed for physics simulation—especially not for robotics applications.

Recently, several simulators have emerged for robotics and embodied AI, such as AI2-THOR, RobotSuite, and SAPIEN/ManiSkill, but each comes with notable limitations:

AI2-THOR uses Unity3D as its simulation backend. While Unity is a well-known game engine, it lacks support for GPU-parallel simulation and tiled rendering, which are crucial for large-scale robotics experiments. RobotSuite separates the physics and rendering engines and integrates them via a Python frontend. This architecture leads to inefficiencies and does not scale well for large simulations. SAPIEN/ManiSkill supports GPU-based simulation and tiled rendering, but it only handles rigid-body dynamics and lacks the flexibility needed to extend or incorporate new features.

IsaacSim/IsaacLab comes closest to meeting all our requirements. However, it is not fully open-source, which limits our ability to modify or extend its low-level capabilities to fit our specific needs.

3. "How would out-of-dataset objects effect the retarget trajectories?"

DexSim includes an automatic domain randomization method to address mismatches between objects in simulated and real-world scenes. Specifically, the reviewer pointed out in the Scene Projection section (Section 4) that the targeted object is included in the asset dataset (OoD), and DexSim matches it with a similar object. Consequently, the skill is learned based on this matched object rather than the actual target object.

However, because our domain randomization process modifies the size, shape, pose, and texture of objects, the learned skills can generalize across different objects. As a result, the model can successfully handle the real object, even if it differs from the one used in the simulation. We have experimentally validated this phenomenon in Section 5.1.

4 "Add Discussion with paper like DexMimicGen: ..."

We have included this paper in our related works and discussed it in detail. The key differences lie in the format of input signals, the methods for skill generation, and the integration of automatic domain randomization and adaptation for Sim2Real generalization.

Dear Reviewer, we sincerely appreciate your constructive feedback. We hope that the following response can address your concerns:

1 "Only a few supplementary videos show different robots, there is no quantitative comparison in the main text or supplementary material."

Response. Table 4 in the appendix summarizes the various embodiments used to demonstrate their performance in real-world scenarios. To better support our claim, we experiment with an additional robot for grasping and manipulating (e.g., grasping a bottle and pouring the liquid into a cup). This experiment is conducted on a dual-armed Cobot Magic robot, i.e., mobile ALOHA (Fu et al., 2024). The success rates are as follows:

For the relevant videos for different embodiments, please check section 1.1 in our anonymous link

Moreover, previous studies on the Sim2Real robotic simulation data engine primarily focus on a single type of embodiment. In contrast, our real-world experiment is conducted on a significantly larger scale.

2. "How does automated DR compare with hand-crafted DR?"

Handcrafted DR refers to selecting the type and scale of DR based on human expertise. We refer to the empirical analysis by Xie et al. (2024), which ranks the importance of various DR features. To evaluate model performance, we apply the top 1, 2, and 3 most important DR features to bridge the Sim2Real gap and report the results as follows.

For the detailed videos, please check section 1.2 in our anonymous link for detailed results.

(Xie et al., 2024) Xie, Annie, et al. "Decomposing the generalization gap in imitation learning for visual robotic manipulation." ICRA 2024.

3. "What is the contribution of individual components (e.g., AI-DR vs. SA-DR)? Similarly, what are the performance gains from object-oriented representations and pose-affordance representations in DA?"

Regarding the comparison between AI-DR and SA-DR, their configurations are automatically determined by the LLM, and as such, there is no guarantee that they will always be included in the DR variants generated by DexSim. This makes it challenging to isolate and evaluate their individual effectiveness in empirical studies. Therefore, we choose to study DR as an integrated component in our analysis.

The same argument applies to object-oriented representations and pose-affordance representations in DA. There is no guarantee they will always be included in DexSim.

4. "Figure 2 mentions image/teleoperation as data sources and RL for policy fine-tuning, but these are not used in the experiments."

We apologize for the misunderstanding. Our Cobot Magic robot is equipped for teleoperation. Since the teleoperation signals are directly compatible with the robot, retargeting is unnecessary. Instead, we can directly transfer these control signals to the simulator to control the simulated robot within the data engine.

Regarding RL fine-tuning, DexSim supports the automatic design of reward and goal functions by leveraging large language models (see lines 270–273, right column). Given these reward functions, applying RL algorithms to DexSim is straightforward. Well-established algorithms, such as PPO and SAC, can be seamlessly integrated into our engine to generate skills.

5. "The reorientation task appears to be a simple pick-and-place..."

The re-arrangement task is significantly more challenging than simple pick-and-place operations for the following reasons:

1. The robot arm must reorient the fork and spoon to ensure they face the front of the plate and position them correctly around it.
2. To achieve smooth execution, both arms must work in coordination, ensuring the fork and spoon are placed in their correct positions simultaneously.

6. "What are the performance contributions of each module?"

In section 5.1, we have conducted an ablation study by removing either the strategic Domain Adaptation (DA) or Domain Randomization (DR) components from our DexSim dataset. We have also added an experiment for comparing different DR methods (either hand-crafted or automated) as mentioned above.

7. "What are the primary failure modes observed in real-world deployments?"

We have included a fail case study in the revised draft. Please see examples in section 1.3 in our anonymous link for detailed results.