HumanoidGen: Data Generation for Bimanual Dexterous Manipulation via LLM Reasoning

We thank the reviewer for the encouraging feedback and valuable comments.

Weaknesses

W1:

The scalability to new objects: To evaluate the scalability of our framework across a diverse set of manipulable assets, we perform supplementary annotation experiments on five object categories: two rigid types of "cup" (with and without handle), "bowl", and two articulated objects, namely "laptop" and "drawer". For transferring annotations across similar objects, we employ the Stable Diffusion-based method introduced in the paper.

For a new object, manually annotating an autonomous action point along with its axis takes only 4 seconds for rigid objects and 6 seconds for articulated objects, thanks to the SAPIEN visualization interface and our annotation-assistive scripts. For automatic annotation transfer, the objects listed in the table are randomly selected from the SAPIEN asset library. The success rates are all above 90%, and the transfer time is under one second, which ensures the scalability of our approach to large-scale asset migration.

Besides, the Stable Diffusion-based method is just one of the approaches we use to simplify the annotation process. Many existing works have explored affordance extraction and structured object axis annotation, such as RoboBrain(CVPR 2025), and CAPNET(CVPR 2025). These methods can be seamlessly integrated into our framework to support zero-shot asset annotation and enable large-scale asset expansion.

Scalability to New Scenes: Our annotation process is object-centric. Once atomic actions are annotated for an object, they can be applied across different scenes involving that object. At the room level, our framework leverages the rich asset information in RoboCasa to directly generalize tasks to 120 scenes, significantly increasing scene diversity. At the table level, our framework integrates constraint-optimization-based target pose solving, dynamic active collision handling, and MCTS-enhanced reasoning. These techniques enable robust adaptation to task scenarios involving object randomization, cluttered environments, and diverse execution strategies.

W2:

Real-World Experiments: We thank the reviewer for pointing out the lack of real-world experiments in our manuscript. In response to the reviewer's suggestion, we have conducted real-robot experiments in both the data generation pipeline and the sim-to-real stages.

We conduct experiments following 5 steps: 1）Real-World Setup and Task Design. We built a real scene corresponding to the simulation environment, which includes a Unitree H1_2 robot equipped with the Instant-Lab dexterous hand, a white table, and the task-related objects. Three tasks are designed for this data collection experiment: "bimanual bottle grasping", "opening a laptop", and "placing a block". 2) Real2Sim Assets. For task-related objects, we prepared both real objects and their corresponding simulation digital assets. Specifically, we obtained the digital assets of standard objects such as Coke cans and Sprite bottles from online asset repositories. For objects whose digital assets can hardly be found, such as the specific plate we used, we generate their digital assets by leveraging Hunyuan3D with fine-grained post-processing applied to ensure their consistency in geometric and appearance. 3) Real2Sim Pose Estimation. Using the existing digital assets, we estimate the objects' current pose in the camera coordinate system with FoundationPose(CVPR 2024). The 6D pose is then transformed into the world coordinate system, allowing the digital asset to be placed at the same pose in our simulation's world coordinate. 4) Trajectory Collection. We use LLM reasoning and the trajectory generation part of our framework to automatically generate execution trajectories in the simulation, and execute them on the real robot to successfully collect real-world data. We collected 25 trajectories for each task mentioned in 1) to train DP, trained for 300 iterations. 5) Sim2Real Policy Transfer. To investigate the effectiveness of simulation data in improving real-world robot policy training, we augment the real-world dataset collected above with additional simulation data. We aim to explore how simulation data contribute to the performance of trained robot policies in real-world experiments. For each aforementioned task, we evaluate the policy performance under the setting of training with 10 real-world trajectories plus N simulated trajectories. The results are as follows:

As shown above, our approach consistently achieves over 85% success rate in real-world data collection, with each trajectory completed in just about 15 seconds. Furthermore, the simulation data produced by our framework significantly enhances real-world execution performance.

W3:

For complex tasks, which typically involve more execution steps and more intricate bimanual coordination, reasoning becomes more challenging for LLMs, leading to a significant drop in success rate compared to simpler tasks. Despite this challenge, our framework still maintains a success rate above 50%, showing its capability in handling bimanual complex and long-horizon tasks. Besides, this planning process can be further enhanced by MCTS to improve both the reasoning success rate and the diversity of execution plans. Furthermore, our collision avoidance mechanisms enable the framework to excel in executing tasks with complex collision scenarios. This is demonstrated by our significantly higher success rate in tasks involving complex collision scenarios compared to RoboTwin, as detailed in our manuscript.

Questions

Q1:

We answer the questions as follows:

1. Our approach is automated with object annotations and the necessary atomic operations provided.
2. We list manual parts of our work as follows: On one hand, the atomic operations in our framework are currently manually predefined to ensure stable performance across diverse dexterous manipulation processes. On the other hand, for annotation transfer, our framework currently requires a separate manual annotation for one object per category to serve as a reference. Besides, we note that once these initial conditions are met, our framework enables fully automatic data collection, eliminating the need for manual effort in constraint generation or demonstration selection.
3. In terms of the manual labor involved in annotation transfer, it currently represents the minimal necessary effort required to scale our framework to a larger object library. We believe this level of manual input is highly efficient compared to fully manual annotation and significantly enhances the scalability of our framework. Moreover, we note that our framework is compatible with methods for automatic object annotation, such as affordance prediction and dexterous grasping pose generation, which could further reduce human intervention in the annotation process.

Q2:

We note some possible approaches for further automating the whole process:

1. Regarding the predefinition of atomic operations, we notice the recent advancements in research on dexterous hand manipulation pose generation. For example, DexGraspAnything(CVPR 2024) and DexVLG generate grasping and manipulation poses for dexterous hands based on 3D and/or 2D perception. Since 2D and 3D information can be provided accurately and reliably in simulation, we believe integrating these methods into our framework could further enhance the data collection process by automatically providing diverse and feasible end-effector pose candidates, reducing the need for manual definition and expanding the range of supported manipulation operations.
2. For object annotation, our framework supports the integration of additional methods to further automate the annotation process. These methods include affordance point and axis prediction approaches, such as CAPNET(CVPR 2025) and RoboBrain(CVPR 2025), which can provide the necessary annotations to enable the successful execution of our framework. In summary, our framework has high scalability, enabling the integration of additional methods to improve the level of automation in the processes mentioned above.

Thank you for your insightful comments. We will explain your concerns point by point.

Weaknesses

W1:

Real-World Experiments: We thank the reviewer for pointing out the lack of real-world experiments in our manuscript. In response to the reviewer's suggestion, we have conducted real-robot experiments in both the data generation pipeline and the sim-to-real stages.

We conduct experiments following 5 steps:

1. Real-World Setup and Task Design. We built a real scene corresponding to the simulation environment, which includes a Unitree H1_2 robot equipped with the Instant-Lab dexterous hand, a white table, and the task-related objects. Three tasks are designed for this data collection experiment: "bimanual bottle grasping", "opening a laptop", and "placing a block".

2. Real2Sim Assets. For task-related objects, we prepared both real objects and their corresponding simulation digital assets. Specifically, we obtained the digital assets of standard objects such as Coke cans and Sprite bottles from online asset repositories. For objects whose digital assets can hardly be found, such as the specific plate we used, we generate their digital assets by leveraging Hunyuan3D with fine-grained post-processing applied to ensure their consistency in geometric and appearance.

3. Real2Sim Pose Estimation. Using the existing digital assets, we estimate the objects' current pose in the camera coordinate system with FoundationPose. The 6D pose is then transformed into the world coordinate system, allowing the digital asset to be placed at the same pose in our simulation's world coordinate.

4. Trajectory Collection. We use LLM reasoning and the trajectory generation part of our framework to automatically generate execution trajectories in the simulation, and execute them on the real robot to successfully collect real-world data. We collected 25 trajectories for each task mentioned in 1) to train DP, trained for 300 iterations.

5. Sim2Real Policy Transfer. To investigate the effectiveness of simulation data in improving real-world robot policy training, we augment the real-world dataset collected above with additional simulation data. We aim to explore how simulation data contribute to the performance of trained robot policies in real-world experiments. For each aforementioned task, we evaluate the policy performance under the setting of training with 10 real-world trajectories plus N simulated trajectories. The results are as follows:

As shown above, our approach consistently achieves over 85% success in real-world data collection, with each trajectory completed in just about 15 seconds. Furthermore, the simulation data produced by our framework significantly enhances real-world execution performance.

W2:

We conducted a quantitative analysis of asset transferability for three types of objects: "cup", "laptop", and "drawer":

As shown above, the time for automated annotation migration is significantly shorter than manual annotation, demonstrating its human-intervention-free advantage. This is especially evident when annotating complex articulated objects, whose operation points and axes often do not align with the object's geometric center or axes, making manual annotation non-intuitive and difficult. In contrast, automated annotation transfer does not face this issue.

However, transfer is not always guaranteed to succeed. On the one hand, there are inherent limitations in the transfer model itself, such as potential point deviations—particularly when the asset is very small or far from the camera. On the other hand, even within the same asset category, the same action may apply to different locations. For example, cups without handles are typically grasped around the center, whereas cups with handles are usually grasped at the handle. Therefore, it is necessary to perform the transfer separately for these two types of cups.

W3:

We conducted a quantitative analysis of time and resource consumption across different stages for 20 tasks. The statistics are as follows:

The above table summarizes data on a per-task basis. However, tasks can vary significantly depending on the number of execution steps involved. On average, a single motion planning step takes approximately 0.53s, and the input to the code generation model typically contains around 3745 tokens per call.

W4:

We agree with the reviewer that LLMs can produce potential errors due to hallucination. We observed this problem in the first experiment of our paper, which led to the failure cases there. We believe this problem can be addressed by our MCTS-enhanced LLM reasoning approach. Specifically, in this approach, we handle such errors by storing relevant failure information at the corresponding node in the search tree where the error occurred. When MCTS revisits that node during future exploration, this information is provided to the LLM as part of the prompt. As shown in our second experiment in the paper, this approach allows the LLM to proactively avoid repeating similar mistakes, which can help it avoid such hallucinations. Besides, our framework executes LLM-generated plans in simulation to collect trajectories once the automatically generated task conditions are satisfied. This helps avoid potential issues related to data quality and experimental safety that may arise from LLM hallucinations.

Questions

Q1:

The GPU and CPU used in our experiments are Nvidia RTX 4090 and Intel Core i9-14900K, respectively. The details of time consumption in our framework are mentioned in the "weakness" section. As for computational bottlenecks, we observed that such bottlenecks exist in GPU and RAM resources when parallelizing multiple simulation environments for training data collection.

Q2:

As mentioned in the "Weaknesses" section, transferable assets must belong to the same category and share identical geometric features; otherwise, the transfer may fail or introduce significant errors. Typical examples include cups with handles versus those without, or drawers with round handles versus horizontal handles, as discussed in the "weakness" section.

Q3:

Referring to the "Weaknesses" section, we do encounter hallucination issues when using LLMs. To manage this complexity, we record the specific cause of any execution error induced by hallucination, such as syntax errors, runtime failures, or task failures, at the corresponding tree node where truncation occurs. This mechanism helps prevent the recurrence of the same hallucination during subsequent planning exploration.

Thank you for your valuable comments. We will explain your concerns point by point.

Weaknesses

W1:

To address the concern about the scalability of our framework to a wider range of manipulable assets, we conduct further automatic annotation transferring experiments on 5 types of objects with the method mentioned in the paper.

Besides, we discuss integrating other works into our framework to effectively handle objects with complex geometry and articulation, as mentioned by the reviewer.

On the one hand, for articulated objects, we note works that focus on predicting affordance points and articulation axes to guide manipulation. Some recent works, such as AffordanceLLM(CVPR 2024) and CAP-NET(CVPR 2025), demonstrate strong generalization capabilities across various object categories, including both numerous single rigid objects and articulated objects. We also note that the input for their prediction and the annotation transformation process, specifically object point cloud sampling and 2D-3D back-projection, can be performed in simulation without noise or error. We believe these methods can provide point and axis annotations on articulated objects, effectively addressing the reviewer's concern regarding articulated objects.

On the other hand, for objects with complex geometry, which pose significant challenges in both annotation and manipulation, we note that this represents a distinct research area where current methods are still immature. Some works, such as DexGraspNet (CoRL 2024) and DexVLG (arXiv:2507.02747), explore generating dexterous hand grasping poses from 2D and/or 3D perception. Although these methods have certain generalization limitations, they show promise in supporting the manipulation of objects with complex geometry. Despite the immature state of research in this area, we note that our framework is compatible with such approaches, offering scalability to handle objects with complex geometry.

W2:

We apologize for any confusion the name "HumanoidGen" may have caused to the reviewer. We would like to clarify that we have explicitly stated that our work focuses on generating data for bimanual dexterous manipulation tasks in the "Data Generation for Bimanual Dexterous Manipulation" part of our title. Both the abstract and introduction sections center around bimanual dexterous manipulation tasks, and we haven't found any content in the main body of the paper that could easily lead readers to misunderstand our work as claiming support for locomotion or whole-body control.

If the concern lies primarily in the naming, we are open to revising it by removing the "HumanoidGen:" part of our title to emphasize the scope of our work as reflected in the main title. Furthermore, we believe our framework also holds potential for future extension to coordinated manipulation involving the lower body.

W3:

We appreciate the reviewer's comments and concerns regarding the limitations of our work. To further demonstrate our framework's capability in handling tasks involving bimanual coordination and dexterous hand manipulation, we conducted experiments on five additional tasks. As shown in the table below, our framework achieves a high success rate across the five new tasks, demonstrating its scalability to new tasks.

We further address the reviewer's concern in detail below:

1. Coordinated Motion: Our framework's adaptability to bimanual coordination tasks is demonstrated by the experiments in our manuscript and the additional "lift pot" experiment. We note that our contribution to bimanual coordination lies in leveraging LLMs to reason about the execution sequence of both arms and to dynamically avoid collisions arising from inter-arm interference at the planning level. Furthermore, we apply task decomposition with MCTS to enable diverse strategies in bimanual manipulation. For instance, in the “dual_bottles_pick_hard” task, the LLM must not only infer the constraints for grasping and placing blocks, but also determine how the two arms should coordinate, including the order of execution, the stacking locations, and the choice of base block or position. These choices, such as stacking on an existing block or selecting an empty region, require the LLM to reason over multiple coordination strategies. Therefore, we believe our framework can effectively handle bimanual coordination tasks using the proposed planning methods.

2. Dexterous Hand Operation: Compared to parallel 2-finger grippers, the dexterous hand's superior structure provides greater flexibility in end-effector poses, but also introduces increased complexity in determining appropriate hand configurations during manipulation tasks. On the one hand, our work aims to strike a balance between flexibility and complexity by defining atomic operations, providing a simplified yet effective approach to enable automatic data collection for dexterous manipulation tasks. We also demonstrated our framework's capability to incorporate additional atomic dexterous hand operations through the additional tasks like 'Rotate Safe Knob' and 'Press Toaster' above. On the other hand, by adding more atomic operations and annotations, integrating additional annotation methods, and incorporating other in-hand manipulation policies as framework components, our framework can extend the current utilization methods of dexterous hands to new tasks and further expand and leverage the dexterity and flexibility of the dexterous hands.

To summarize, our framework supports bimanual coordination through MCTS-enhanced LLM reasoning with collision management at the planning stage. Moreover, we provide a method to balance the flexibility advantage and the complexity inherent in dexterous hand manipulation, and leverage the framework's scalability to enable further utilization of dexterous hands.

Questions

Q1:

We discussed this in the "Weaknesses" section. Overall, we define our framework to be a data collection framework for bimanual dexterous manipulation tasks, with the following key aspects to be noted:

1. We leverage LLMs to generate task scenes and plan scripts, obtaining manipulation trajectories through an optimization-based motion solver.
2. We leverage MCTS to enhance LLM reasoning for stable and diverse plan script generation.
3. We provide diverse atomic manipulation operations in our framework, including grasping, pinching, pressing, pulling, and rotating, as well as basic operations on articulated objects, supporting a variety of bimanual dexterous manipulation tasks.
4. Our framework's effectiveness is primarily demonstrated by our experiments with DP and DP3.
5. The scalability of our framework is demonstrated by the annotation transfer experiment and further enables integration with works on affordance prediction and dexterous grasping pose generation, allowing extension to a broader range of object types.

To summarize, our framework focuses on bimanual dexterous manipulation tasks and demonstrates strong advantages in enabling automatic data collection for applications in this domain.

Q2:

Our framework is highly scalable, supporting extension to complex, contact-rich manipulation tasks. It leverages LLMs for script generation, enhanced by MCTS for higher planning success rate and diversity. Currently, it supports atomic operations such as grasping, pinching, pressing, and rotating, and can be expanded with new annotations, operations, and dexterous policies. In short, our framework efficiently scales to more complex tasks and richer atomic operations.

Q3:

We answer this question from the following perspectives:

1. Annotation scalability and generalization to objects: Our framework supports annotation transfer, enabling extension to diverse objects, as demonstrated in the additional experiments. Its modular design allows integration of new annotation methods (e.g., affordance prediction), atomic operations, and dexterous manipulation policies, facilitating generalization to more complex objects.

2. Scene scalability and generalization to new tasks: As stated in our manuscript, the generated task scripts can be executed directly across all 120 RoboCasa scenes without additional annotation effort. We also elaborate on our framework's adaptability in the response to Question 2, highlighting its capabilities at both the LLM reasoning level and the atomic operation level.

To summarize, our framework demonstrates strong scalability and generalization, enabling effective adaptation to diverse objects, complex manipulations, and a wide range of unseen scenes and tasks through its flexible integration of high-level planning and low-level operation components.

Dear Reviewer,

We wanted to express our gratitude for your insightful feedback during the review process of our paper. We hope we have satisfactorily addressed all your concerns and demonstrated the improved quality of our work. If there are any additional points you'd like us to consider, please let us know. Your insights are invaluable to us, and we're eager to address any remaining issues.

Thank you for your time and effort in reviewing our paper.

Best regards,

The authors

Dear Reviewer mhYR, thanks for your thoughtful review. As the author-reviewer discussion period is near its end, we wonder if our rebuttal addresses your concerns. Please let us know if any further clarifications or discussions are needed!

Thanks for your careful and valuable comments. We will explain your concerns point by point.

Weaknesses

W1:

In response to the reviewer's suggestion, we have conducted real-robot experiments in both the data generation pipeline and the sim-to-real stages.

We conduct experiments following 5 steps: 1）Real-World Setup and Task Design. We built a real scene corresponding to the simulation environment. Two tasks are designed for experiments: "Dual Bottles Pick" and "Close Laptop". 2) Real2Sim Assets. For task-related objects, we prepared both real objects and their corresponding simulation digital assets. Specifically, we obtained the digital assets of standard objects such as Coke cans from online asset repositories. For objects whose digital assets can hardly be found, we generate their digital assets by leveraging Hunyuan3D with fine-grained post-processing applied to ensure their consistency in geometric and appearance. 3) Real2Sim Pose Estimation. Using the existing digital assets, we estimate the objects' current pose in the camera coordinate system with FoundationPose (CVPR 24). The 6D pose is then transformed into the world coordinate system, allowing the digital asset to be placed at the same pose in our simulation's world coordinate. 4) Trajectory Collection. We use LLM reasoning and the trajectory generation part of our framework to automatically generate execution trajectories in the simulation, and execute them on the real robot to successfully collect real-world data. We collected 25 trajectories for each task for 300 iterations. 5) Sim2Real Policy Transfer. To investigate the effectiveness of simulation data in improving real-world robot policy training, we augment the real-world dataset collected above with additional simulation data. We aim to explore how simulation data contribute to the performance of trained robot policies in real-world experiments. For each aforementioned task, we evaluate the policy performance under the setting of training with 5 real-world trajectories plus 100 simulated trajectories. The results are as follows:

As shown above, our approach consistently achieves over 85% success rate in real-world data collection, with each trajectory completed in just about 15 seconds. Furthermore, the simulation data produced by our framework significantly enhances real-world execution performance.

W2:

We note two differences between our approach and the augmentation-style methods:

1. The difference in annotation types. Our method requires annotating operation points and axes on the assets. These annotations correspond to different robot atomic operations, such as grasping and pinching. In contrast, MimicGen and DexMimicGen collect trajectories by teleoperation and annotate different subtask segments in the trajectories for further replay. Compared to their annotation type, which relies on collected task-related trajectories and is hardly generalizable to new tasks, our annotation type is task-agnostic and supports further task extension. By combining manual annotation and annotation transferring, our method allows building a large-scale library with annotated assets, supporting automatic script generation and data collection of diverse tasks.

2. The difference in resource consumption. The table below shows the time cost of our method at different stages:

We evaluated the time cost of our method versus augmentation-style methods under several scenarios: For new tasks with existing assets: Our method can generate trajectories fully automatically, with no manual effort required. The fixed time cost per task before generating trajectory data is 30.95s, which accounts for Scene Generation and Script Generation. The time to generate a single trajectory equals the execution time of one run, averaging 14.4s.

• DemoGen requires teleoperation to collect an initial trajectory and additional manual effort to crop and segment point clouds. It's fixed time cost averages 86s. Generating an additional trajectory only takes 0.01s, but this approach cannot provide data augmentation for vision-based policies and cannot guarantee that the generated data is always feasible.
• DexMimicGen has an even longer preparation time, averaging 253s. Besides data collection, manual segmentation into subtasks is required before trajectory generation. After generation, it still needs to execute trajectories to collect data, with each trajectory averaging 16s.

For new assets and new tasks: Our method requires an additional 4–6s of manual calibration. Nevertheless, it remains the most time-efficient overall. This efficiency stems from the fact that our approach does not rely on a pre-collected trajectory as the basis for generation; instead, it leverages annotated assets.

W3:

We appreciate the reviewer's suggestion on limitations. We will move the limitation section from the appendix to the main body of the paper and further analyze the limitations of our work. We provide a detailed discussion below in response to the limitations mentioned by the reviewer.

• Cluttered scenes: Our framework can handle tasks with cluttered environments and distractor objects. This capability stems from the Active Collision Avoidance and Dynamic Collision Management mechanisms in our framework design. Specifically, the Active Collision Avoidance enables the LLM to plan steps that first move occluding objects aside before manipulating the target object. The Dynamic Collision Management, on the other hand, enables the LLM to determine which collision bodies to consider during path planning. These mechanisms together can effectively manage complex collision scenarios, such as those found in cluttered task environments. In addition, our MCTS reasoning enhancement enables the system to iteratively correct failed attempts and continue searching for a valid trajectory in case the LLM fails to generate a successful plan in one shot. To the best of our knowledge, such capabilities are not present in prior works such as robotwin(CVPR2024), gensim2, and DexMimicGen.

• Deformable objects: Our framework currently doesn't support deformable object manipulation for two reasons: First, the Maniskill simulator lacks deformable object support. Second, our framework targets tasks with clear goals, making operations like flattening a crumpled shirt challenging due to the need for real-time feedback on fabric wrinkles and excessive LLM reasoning steps. However, extending our framework with more diverse atomic skills, such as training specific DP policies for subtasks like flattening, could enable support for these "goal-ambiguous" tasks, highlighting the system's extensibility.

• Contact-rich tasks: Contact-rich tasks require rapid reactions to runtime feedback. For tasks like push-T and shirt-flattening, the need for fine-grained step-by-step corrections arises from their goal-ambiguous nature, where each action can lead to multiple outcomes without a fixed solution. This makes precise trajectory replication impossible in our current framework. However, our system can instead generate alternative strategies, such as directly grasping and placing the T-shaped object. To better handle such tasks in the future, a promising direction is to enrich the framework with additional model-based atomic operations.

W4:

Field of affiliation. We thank the reviewer for the suggestion regarding the appropriate field of affiliation. While our work is indeed related to applications, we believe it also appropriately falls within the scope of deep learning. Our work uses LLMs to generate expert demonstrations for bimanual dexterous manipulation, enhances reasoning with MCTS, and evaluates models like DP and DP3, all closely tied to deep learning methodologies. We appreciate the reviewer’s thoughtful suggestion.

Questions

Q1:

We added real-robot experiments, which are discussed in the "Weaknesses" section. Regarding the difficulties in deploying the framework to real systems mentioned by the reviewer, we believe the main challenge lies in the sim-to-real gap of physics simulation and visual rendering. The former can be addressed by applying system identification to optimize the parameters of the actuators in the simulation. The latter can be addressed by modifying lighting and brightness settings in the simulator to match real-world conditions.

Q2:

Due to the character limit, we refer the reviewer to the discussion in the 'Weakness' section. In addition, our object annotations are reusable across multiple tasks, whereas augmentation-style data generation methods are typically limited to augmenting data only in terms of spatial position.