ManiBox: Enhancing Spatial Grasping Generalization via Scalable Simulation Data Generation

Thank you for your insightful and constructive comment. We greatly appreciate your acknowledgment of the general idea, real-robot deployment, and visual presentation of our work. Below, we address each of your concerns in detail.

W1.1: Novelty of our approach

A: While building on the student-teacher framework, our work introduces the following contributions that distinguish it from prior methods:

• Bounding-box-guided spatial generalization: By incorporating bounding boxes into the student's input space, we address the high-dimensional limitations of visual generalization. This improvement enables adaptability across diverse spatial positions, objects, and backgrounds.
• Discovery of spatial generalization laws: We are the first to identify two spatial generalization laws relating workspace size, data volume, and policy performance. These laws provide actionable insights for estimating data requirements and have been referenced by peers in embodied AI.
• Efficiency and practicality: Our method, ManiBox, achieves strong spatial and visual generalization with minimal computational resources. For example, training requires only 2 hours for the teacher policy, 1 day for data generation, and 2 minutes for the student policy (single GPU, RTX 4090).

W1.2: Relationship between workspace size and data requirements

A: While the overall trend may appear intuitive, our work provides the first empirical quantification and reveals unexpected insights:

• Non-linear scaling: Data requirements grow sublinearly with workspace size, following a power law (Figures 4, 15-16). This efficiency offers practical guidance for scaling.
• Saturation effect: Success rates exhibit Michaelis-Menten kinetics relative to data volume, showing diminishing returns as data increases (Figure 3). These findings deepen the understanding of spatial generalization.

W1.3: Experimental setup and scalability

A: We agree that more challenging scenarios are essential to validate robustness. The "Complex Generalization" experiments in Figure 5 already feature multi-object and cluttered environments, such as non-tidy desktops with computers and other objects, highlighting ManiBox's strong generalization capabilities.

To further support this, we conducted additional experiments on manipulation tasks like "pouring water", "grabbing the handle of a cup", "cluttered table", "messy table" (see Appendix C and Figure 13-14), demonstrating ManiBox's versatility beyond standard grasping tasks.

Compared to prior works like AnyGrasp or the Inria paper, which focus solely on grasping, ManiBox generalizes across diverse task types and settings using its bounding-box-guided policy learning approach. Moreover, ManiBox achieves strong results efficiently, requiring minimal computational resources, making it highly practical for real-world deployment. We have updated the manuscript to include these additional experiments and discussions.

W2.1: Representation with 2D bounding boxes

A: We acknowledge that representing objects with 2D bounding boxes offers only a rough approximation of an object's convex hull, which may present challenges for highly irregular or diverse shapes. However, our method prioritizes simplicity and computational efficiency, making bounding boxes a practical choice for grasping and simple manipulation tasks while enabling exploration of spatial generalization laws.

To address this limitation, ManiBox can flexibly integrate other object detection models. For example, in newly added tasks like "grabbing the handle of a cup" (see Appendix C), Grounding DINO enables detailed detection of specific parts. Similarly, SAM can provide finer segmentation, which we plan to explore in future work to improve generalization to complex object geometries.

W2.2: Reliance on multiple camera observations

A: As shown in Figure 2, the number of cameras does not increase the deployment complexity, as it only impacts the dimensionality of the bounding box representation. Using multiple camera observations enhances policy robustness compared to a single-camera setup. Interestingly, biological intelligence, such as human vision, also benefits from binocular perception, which inspires our approach.

Thank you for your thoughtful and positive feedback. We greatly appreciate your recognition of the strengths of our work, including the effectiveness of our sim-to-real transfer strategy, the robustness of background generalization, and the valuable insights provided by the scaling law relationship.

W1: Object diversity

A: In our experimental results (Figure 5), we demonstrated that our method performs effectively across a diverse range of objects, including various cups, bottles, and different types of fruits. Additionally, we have supplemented two new experiments in Appendix C on the "pouring water" and "Grab the handle of the cup" task, which requires precise manipulation, showcasing the generality of our approach for handling complex objects and tasks.

Furthermore, our method is adaptable to different object detection models. For instance, Grounding DINO can enable detailed detections (e.g., cup handles or door handles), and SAM can provide more refined segmentation. These extensions, planned for future work, will further enhance the scalability of our approach to handle a broader variety of objects and more intricate tasks.

W2: Bounding box usage

A: Bounding boxes are critical for calculating an object's spatial position. Using object detection models like YOLO-World, we determine the object's center from bounding box data (see Lemma 2) and provide this information to the student policy. The student learns to grasp the center of the bounding box, offering a robust and deployment-friendly solution.

Bounding boxes provide strong generalization across diverse objects, positions, and backgrounds, while being computationally efficient. Unlike the ideal privileged object information (e.g., exact object positions) used by the teacher, bounding boxes are practical for real-world deployment and enable rapid student policy training (1 day for data generation, 2 minutes for training on an RTX 4090). As a bridge between ideal object positions and real-world detection, training the student policy with bounding boxes mitigates performance degradation caused by real-world visual inaccuracies.

W3.1: Fixed cubic space constraint

A: Our goal is to explore the relationship between spatial generalization and data quantity by examining patterns across different workspace ranges. Based on the robotic arm's estimated maximum working range (41 cm × 30 cm × 28 cm), we systematically expanded the testing spaces from a fixed point to 5 cm, 10 cm, and 20 cm. This progressive expansion allows us to analyze how spatial generalization evolves with increasing workspace size and changing data distribution. Using fixed cubic spaces ensures a controlled and consistent evaluation framework.

W3.2: Complex camera setup

A: The three-camera setup is based on the default configuration of our robotic system. However, in practice, only two cameras are effectively used for most tasks. For example, the right camera often fails to capture objects during left-arm manipulation. Thus, the third camera primarily acts as a backup and does not introduce additional complexity into the system. The performance and generalization results reported are achieved using two primary cameras for detection and localization.

W4: Fragmented descriptions of settings

A: To improve readability, we have included a pseudo-code representation in Algorithm 1, titled "Training and Deployment Algorithm of ManiBox." This outlines the teacher policy training, simulation data generation, student policy training, and real-world deployment. We hope this addition enhances clarity and accessibility for readers.

W5: Lack of vertical variations

A: In our experiments (Figure 5), we used a height-adjustable table to create objects with varying Z-axis positions. As the student policy relies on bounding boxes to locate objects, it is capable of handling objects in mid-air. To further validate this, we have added "Apple in Mid-air" real-world experimental results in Figure 12 and "cluttered table"(Figure 13-14), demonstrating the system's ability to grasp objects with vertical variations effectively.

W6: Formula 4 confusion

A: We have clarified the meanings of all symbols used in Formula 4 in the revised manuscript.

Thank you for your thoughtful review and for recognizing the strengths of our work, including improvements in object grasping tasks, sim-to-real transfer, and the study on data scaling laws. We appreciate your detailed summary and constructive feedback, which motivates us to further refine our work.

W1: Clarification on Lemma 2

A: While Lemma 2 may draw from established principles in 3D computer vision, such as those in Hartley and Zisserman (2003), our contribution lies in its extension to 2D bounding boxes rather than individual points. Specifically, we compute the 3D object center using multi-view bounding boxes, a novel approach tailored for bounding box-guided robotic manipulation tasks. We have updated the manuscript to provide additional context and references.

W2: Limited comparison to baseline

A: Most object-grasping methods are fundamentally different from ours, as our approach focuses on modeling trajectory data from a simulator and deploying the student policy on a real robot. Only a few algorithms, such as ACT and Diffusion Policy, have been successfully applied to the Mobile Aloha robot, and both rely on real-world data, which in our experimental setting struggles to generalize across the entire workspace.

Additionally, transferring simulator-generated video data to the real world would require extensive domain adaptation and visual data augmentation, making it a separate, large-scale effort beyond the scope of our work. For these reasons, we selected ACT as the sole baseline, as it represents the most relevant approach for comparison in our setup.

Furthermore, the high success rates reported in Table 1 highlight the effectiveness of ManiBox. More importantly, this work focuses on uncovering spatial generalization laws, which we believe is a significant contribution to the field.

Q1: Why does vision-based ACT get a score of 0?

A: As mentioned earlier, ACT requires training on video-action data. While it can work under fixed-point conditions with 100 real-world data samples, this is insufficient for achieving strong spatial generalization. Collecting 20,000 real-world samples, which may be necessary for better generalization, is prohibitively expensive and time-consuming.

When using simulator video data, ACT faces significant visual discrepancies between simulation and reality. It can succeed in fixed-point tasks by memorizing joint position trajectories, enabling sim-to-real transfer. However, for larger spatial volumes, ACT fails due to these discrepancies. In contrast, our method is unique in that, beyond using bounding boxes and random masking, it requires no additional sim-to-real visual generalization techniques.

Given ACT's low success rate in our setup, we ensured the fairness of our experiments by thoroughly validating hyperparameters and input configurations at the outset.

Q2: "Ensure its dynamics are consistent with the real world". How was it ensured?

A: Ensuring consistency involves two key aspects:

• Camera calibration: We calibrated the intrinsic and extrinsic parameters of the cameras through hand-eye calibration and iterative adjustments, using observed bounding box discrepancies as feedback.

• Joint position mapping:

  • We performed real-to-sim mapping by teleoperating the robot and visually verifying that the joint positions in simulation matched those in the real robot.
  • For the gripper, as its joint definitions differ between sim and real, we manually measured 10 gripper open-close distances and fitted an approximate linear function to map the gripper joint positions from sim to real.

These steps have been clarified and detailed in the revised manuscript in Appendix E.4.

Thank you for your continued thoughtful feedback.

In response, in addition to the comparison of ManiBox and ACT in the paper，we have included a more thorough comparison with a heuristics-based approach, specifically the Inverse Kinematics (IK) algorithm from Isaac Lab, alongside other relevant baselines. We conducted tests using 3 different seeds in the simulator, and the comparison results are as follows:

• Inverse Kinematics (IK): This approach directly uses the ground truth object pose as the target end-effector (eef) pose, then computes the joint positions via IK to move the robot arm to the target. However, we observe that IK performs poorly at the edges of the workspace, especially when objects are near the table surface or at low positions. Examples of IK failures when the table is very low can be seen on this anonymous link. The reason for IK failures is possible:
  • The robotic arm may approach singular configurations where the Jacobian matrix becomes singular, causing unstable or undefined IK solutions.
  • Near the workspace edge, numerical inaccuracies may result in imprecise IK solutions. Moreover, approximation methods (e.g., pseudoinverse) may struggle to converge to a valid solution in complex boundary cases.
• Key Steps: In this approach, only key steps of the trajectory are used to model the grasping task.
• Without Random Mask: This variant trains the student policy without random masking for object positioning.

Key Insights:

• IK Limitations: As expected, IK struggles in the workspace boundaries due to singularities, which cause failures when the object is positioned at the edges of the table or at low heights. This limitation is inherent to traditional geometric methods like IK, which rely on precise pose knowledge and are prone to errors in challenging environments.
• Our RL-Based Approach: In contrast, ManiBox (Ours) utilizes RL to learn robust grasping teacher policies by directly planning the joint positions, enabling successful manipulation across a wide range of object positions. Our method does not rely on explicit pose knowledge and is highly effective even at challenging object locations that cause IK to fail.

We hope this extended comparison addresses your concerns and clearly demonstrates the advantages of our approach over traditional heuristics-based methods, particularly in terms of robustness and generalization. Moreover, as previously mentioned, We conducted some complex tasks such as "pouring water" (applying our approach to multi-object manipulation tasks) and "grabbing the handle of a cup"(applying our method to the grasping of detailed parts of irregular objects(Appendix C, Figures 10-14, and Supplementary Material) to demonstrate the scalability of our approach. Thank you for your valuable feedback, and we look forward to your thoughts on these additional experiments.

Thank you for your review and for recognizing the clarity of our presentation, the robustness of our experimental setup, and the potential of our methodology to inspire future research. We also appreciate your acknowledgment of the supplementary video, which supports our work. Your feedback motivates us to refine the manuscript further.

W1: Pseudo algorithm

A: To improve readability, we have included a pseudo-code representation in Algorithm 1, titled "Training and Deployment Algorithm of ManiBox." This outlines the teacher policy training, simulation data generation, student policy training, and real-world deployment. We hope this addition enhances clarity and accessibility for readers.

W2: Weaknesses and failure cases

A: We acknowledge the following limitations of our approach, which are now discussed in Appendix B.2 of the revised manuscript:

• Detection errors: The policy fails when the detection model entirely misidentifies or fails to detect the target object.

• Bounding box limitations: Since the policy uses bounding boxes, it may only grasp the center of irregular objects. Future extensions could leverage tools like Grounding DINO for detailed object detection (e.g., grasping specific parts like handles) or SAM for fine-grained segmentation to handle more complex objects.

• Complex tasks: For tasks like folding clothes, generating suitable simulator data is challenging, as the policy would struggle to identify appropriate grasp points.

W3: Reward design

A: Thank you for your suggestion. We have included a detailed mathematical description of the reward function in Appendix E.1.4 to provide more clarity on its design.

Q1: Does the use of bounding boxes limit object representation for irregular shapes?

A: Thank you for this observation. While bounding boxes provide a simple and efficient representation, they may not capture finer details of irregularly shaped objects. One potential enhancement is to use Grounding DINO for detailed object detection (Appendix C), such as identifying specific parts like the handle of a cup or a door handle, allowing the policy to target these features. Alternatively, segmentation results from SAM could replace bounding boxes for more precise object representation and manipulation.

For objects like bananas, the policy can generalize to elongated shapes if similar examples are included in the simulator data. For instance, in Appendix C, we demonstrate two tasks involving "pouring water" and "Grab the handle of the cup", which includes handling non-spherical objects.

For objects requiring top-down grasps, the teacher policy could be adapted during the data generation phase to include such examples. This approach is similar to the pouring task, where modifying the teacher policy's target enables specific actions. Since this work focuses on standard tasks and exploring spatial generalization laws, these extensions are left for future research.

Q2: What is the history length of input observations for the LSTM student policy?

A: The LSTM’s history length is effectively infinite. We chose not to set a context length to ensure the student policy has access to the full history of observations. This is particularly important in cases where the detection model fails to identify any bounding boxes midway through a task, allowing the student policy to rely on bounding box data from earlier steps to infer the object's properties.

Thank you for your valuable feedback and for acknowledging the improvements in our revised paper. We understand your concerns regarding the use of bounding boxes and the limitations of the grasping policies.

To clarify, the bounding box approach is designed to provide an efficient way to begin learning spatial generalization. It is not meant to limit the scope of manipulation but rather to create a foundation for future work. In our latest version, we have also compared our method against a more heuristic-based approach (Inverse Kinematics algorithm). Our experiments show that while these heuristic methods are prone to failure, especially at workspace edges, our RL-based approach performs robustly across a wider range of scenarios, as shown by the improved success rates in our experiments:

• Inverse Kinematics (IK): This approach directly uses the ground truth object pose as the target end-effector (eef) pose, then computes the joint positions via IK to move the robot arm to the target. However, we observe that IK performs poorly at the edges of the workspace, especially when objects are near the table surface or at low positions. Examples of IK failures when the table is very low can be seen on this anonymous link. The reason for IK failures is possible:
  • The robotic arm may approach singular configurations where the Jacobian matrix becomes singular, causing unstable or undefined IK solutions.
  • Near the workspace edge, numerical inaccuracies may result in imprecise IK solutions. Moreover, approximation methods (e.g., pseudoinverse) may struggle to converge to a valid solution in complex boundary cases.
• Key Steps: In this approach, only key steps of the trajectory are used to model the grasping task.
• Without Random Mask: This variant trains the student policy without random masking for object positioning.

Key Insights:

• IK Limitations: As expected, IK struggles in the workspace boundaries due to singularities, which cause failures when the object is positioned at the edges of the table or at low heights. This limitation is inherent to traditional geometric methods like IK, which rely on precise pose knowledge and are prone to errors in challenging environments.
• Our RL-Based Approach: In contrast, ManiBox (Ours) utilizes RL to learn robust grasping teacher policies by directly planning the joint positions, enabling successful manipulation across a wide range of object positions. Our method does not rely on explicit pose knowledge and is highly effective even at challenging object locations that cause IK to fail.

We agree that further exploration of more complex object representations and advanced manipulation policies (We conducted some complex tasks such as "pouring water" (applying our approach to multi-object manipulation tasks) and "grabbing the handle of a cup"(applying our method to the grasping of detailed parts of irregular objects(Appendix C, Figures 10-14, and Supplementary Material)) would be a valuable direction for future work. However, we believe our current approach provides a solid foundation for tackling the challenges of spatial generalization. Thank you again for your constructive suggestions.