GravMAD: Grounded Spatial Value Maps Guided Action Diffusion for Generalized 3D Manipulation

General Reply: We sincerely thank Reviewer vbZp for recognizing the strengths of our work, including the strong motivation of the proposed method, the integration of multiple components into a unified framework, and the clarity of the figures and ablation studies that provide valuable insights. We address the concerns and questions in detail below.

Novelty and contribution claims are not very clear:

• 1. Discovering sub-goals and predicting them to guide robot actions has been done before [1,2,3,4]. Is the novelty here mainly the 3D part combined with foundation models? Maybe the difference could be emphasised more.
• 2. From the paper, it seems that GravMaps are Cost and Gripper openness maps used in VoxPoser, just renamed. Without properly explaining the difference, it shouldn’t be called a novel contribution.
• 4. Claiming to design two types of tasks (line 124) and then using base tasks from RLBench for one type can come off as being disingenuous about the contribution claims. Rephrasing the 4th contribution to focus more on the evaluation rather than task design would be beneficial.

Response to W1: Thank you for your suggestion. We acknowledge that the concept of task sub-goals has been explored in prior works. The primary innovation of GravMAD, however, lies in utilizing sub-goals as a mechanism to integrate foundation model-based approaches with imitation learning-based methods, rather than in proposing the concept of sub-goal discovery itself. This integration enables more effective task execution by combining the strengths of both approaches.

To address your concern, we have revised the potentially misleading statements in the Introduction section of the paper. We have also cited the four papers mentioned by the reviewer (among which ChainedDiffuser was already cited) and explicitly acknowledged their contributions to the development and application of sub-goals in this context. Once again, we sincerely thank Reviewer vbZp for bringing this to our attention and helping us improve the clarity and accuracy of our work.

Response to W2: Thank you for your feedback. The differences between our GravMaps and the value maps used in Voxposer are primarily as follows:

1. Number of value maps: Voxposer employs multiple value maps, including the cost map, rotation map, gripper map, and velocity map. In contrast, our method focuses on combining only the cost map and gripper map, with their numerical values remaining identical at this stage.
2. Structure and processing: While Voxposer uses dense value maps, we downsample the cost map and gripper map into a point cloud structure that includes position and gripper information $(x, y, z, m_c, m_g)$. This structure, which we term GravMap, provides a more efficient and sparse representation of sub-goals. Additionally, GravMaps are compatible with point cloud encoders, enabling the extraction of rich spatial features and enhancing their utility for downstream tasks.

To clarify these distinctions further, we have included a detailed discussion on the differences between GravMaps and the value maps used in Voxposer in Appendix C.1 of the revised manuscript. Thank you again for your valuable comment, which has helped us improve the clarity of our paper.

Response to W3: Thank you for pointing out this issue. The 12 base tasks were indeed directly selected from RLBench rather than "designed" by us. We have revised the corresponding statements in the paper.

Some parts of the method and design choices need to be better explained:

• How GravMap tokens are used in the network architecture is not described in the main text of the paper.
• Explaining in more detail why sub-goals are extracted differently during training and inference would greatly benefit the clarity of the paper. It becomes obvious while reading the paper, but initially, it can cause confusion.
• It is not clear why rotation maps from VoxPoser are not used. Justifying it would make the overall approach clearer.
• GravMap Synthesis during Inference is not explained clearly. Lines 305-309 are very dense with information, but it’s very difficult to understand how the sub-goals are actually extracted. If it is done just using existing methods, then contribution claims should be adjusted accordingly.

Response to W4: Thank you for your comment. The usage of GravMap tokens in the network is explained in detail in Appendix A.3 and illustrated in Figure 5 of the original paper. To better address the reviewer’s concerns, we have included additional descriptions in Section 3.3 of the revised manuscript (highlighted in blue). While these revisions provide a concise explanation of how GravMap tokens are utilized, the full details, including step-by-step descriptions and illustrations, are available in Appendix A.3 for a more comprehensive understanding. We recommend referring to this section for deeper insights due to space constraints in the main paper.

Response to W5: Thank you for the question. The reasons for using different methods to generate GravMaps during the training and inference phases are as follows: 1. Efficiency and reliability during training; 2. Simplifying the problem by avoiding semantic reasoning. We have added a detailed explanation of this issue in Appendix C.3.1 of the revised paper. Thank you again for your valuable suggestion.

Response to W6: The primary reason we currently do not use the rotation map from Voxposer is the significant distributional shift that may arise between the guidance provided during training and inference. During training, precise rotation guidance can be extracted directly from expert trajectories, ensuring high-quality supervision. However, during inference, off-the-shelf foundation models often struggle to accurately interpret 3D spatial relationships based on visual and linguistic inputs, which limits their ability to provide precise rotation guidance. This discrepancy between the high-quality training guidance and the less reliable inference guidance can lead to substantial performance degradation.

To address this issue, our future research plans include combining rotation information from expert trajectories with object pose data to create few-shot prompts for off-the-shelf foundation models [1]. By leveraging these few-shot prompts, large language models (LLMs) could generate more accurate and effective rotation guidance, reducing the distributional shift from the training data and improving overall inference performance. We have added a detailed explanation of this issue in Appendix C.2 of the revised manuscript. Thank you again for highlighting this point and helping us improve the clarity of our work.

Response to W7: Thank you for your comment. The synthesis process of GravMap during inference proceeds as follows: 1) the robot's observations and task language instructions are processed by the Detector to produce the Context. 2) Based on this Context, the Planner and Composer generate Sub-goals, which are subsequently used to create the GravMap. This inference process is depicted in Figure 2 of the main paper.

Our approach adapts this process from the existing Voxposer method, with significant differences in the implementation. In Voxposer, tools like the open-vocab detector OWL-ViT, Segment Anything, and the video tracker XMEM are used to track object masks, which, combined with RGB-D observations, reconstruct the object/part point cloud as the Context. In contrast, our method relies solely on SoM-driven GPT-4 to interpret instructions and generate the Context. This results in a more lightweight and efficient inference pipeline.

To provide further clarity, Appendix A.2.2 of the paper details the entire GravMap synthesis process during inference, including comprehensive descriptions of the Detector, Planner, and Composer. Moreover, we have shared all inference prompts used with the foundation models on our anonymous project website for full reproducibility: https://gravmad.github.io.

We hope this explanation and the resources provided address your concerns. Thank you again for your valuable feedback.

[1] Yin, Yida, et al. "In-Context Learning Enables Robot Action Prediction in LLMs." arXiv preprint arXiv:2410.12782 (2024).

• What is the difference between GravMaps and the cost maps described in VoxPoser? Is it the downsampling part?
• Why are rotation maps from VoxPoser not used?
• Do you have any intuition on how well your proposed method would scale with more data? Would Imitation Learning catch up at some point (with a lot of training data)?
• Could you expand a bit more about the limitations of your proposed method?
• GravMaps are just an overparameterization of sub-goals at a high level. The fact that the ablation variant (w/o. GravMap) performs so poorly is not very intuitive. Could you provide more details about this ablation variant and maybe you have some intuition about its performance?

Response to Q1: Thank you for your question. We have explained this problem in the above Response to W2, and the detailed explanation has been reflected in Appendix C.1 of the revised manuscript.

Response to Q2: Thank you for your question. We have explained this problem in the above Response to W6, and the detailed explanation has been reflected in Appendix C.2 of the revised manuscript.

Response to Q3: Thank you for your question. To address your concern about how well our proposed method scales with more data, we conducted training comparisons using five different demonstration dataset sizes and visualized the validation curves. The results and detailed analysis have been added to Appendix D.3, with the validation curves presented in Figure 14. From the results, we observe that the model’s performance improves as the number of expert demonstrations and viewpoints increases, but the improvement plateaus with very large datasets. This may be because the model’s parameter size does not scale proportionally with the data volume. We hypothesize that explicitly modeling shared structures, such as sub-goals and sub-stages guided by GravMap-like representations, could be an effective way to continue scaling performance with increasing data.

Regarding imitation learning, we believe that with sufficient high-quality data, traditional imitation learning methods may reduce the gap, but their lack of generalization beyond the training set may still limit their scalability to novel tasks. In contrast, our method leverages foundation models, which offer robust generalization capabilities that complement the strengths of imitation learning.

We hope this provides insight into the scalability of our approach, and we thank you for raising this important point.

Response to Q4: Thank you for your question. A more detailed analysis of our proposed method's limitations and potential future improvements has been provided in Appendix E of the revised manuscript.

Response to Q5: Thank you for your question. In the w/o GravMap ablation, we replaced the GravMap with a single point and its corresponding gripper openness $(x, y, z, \text{openness})$ as input, removing the step of processing sub-goals into GravMaps. The key difference is that GravMaps represent sub-goals as regions (with uniform values near the sub-goal and increasing costs farther away), while the ablation uses only a single coordinate and openness value. The performance drop occurs because relying solely on a point makes the model overly sensitive to slight deviations, focusing on the exact point rather than the broader sub-goal region. GravMaps provide a robust representation, allowing the model to handle variations better. Moreover, GravMaps, as point cloud structures, are encoded into dense features by the point cloud encoder, offering stronger 3D sub-goal representations and improving overall performance.

To better address your concern, we have revised Section 4.4 of the manuscript to include an updated analysis of the w/o GravMap ablation experiment. We hope this explanation clarifies the role and importance of GravMaps.

We appreciate Reviewer vbZp' constructive feedback and will incorporate all suggestions to improve the clarity, completeness, and rigor of the manuscript. Thank you for helping us strengthen our work!

Thank you for your insightful suggestions and valuable feedback regarding the "w/o Cost map" ablation study.

We have started re-training the "w/o Cost map" ablation model based on your suggestion to represent the gripper closing action as -1 instead of 0. Due to the lengthy training process, we estimate that the results will only be available toward the later stages of the rebuttal phase (approximately 30 hours are required to train on the 12 base tasks). If this modification successfully resolves the NaN issue, we will include the updated experimental results in the revised manuscript. Conversely, if the issue persists, we will remove this ablation experiment from the paper, as per your suggestion, to ensure the clarity and scientific rigor of the manuscript.

We sincerely appreciate your constructive feedback, which has not only helped improve the quality of our work but also provided valuable insights for related research. If our response satisfactorily addresses your concerns, we earnestly and kindly request the reviewers to consider raising their scores and supporting the acceptance of this paper.

Thank you for your valuable feedback and for raising your score. We greatly appreciate your recognition of our efforts to enhance the quality of the paper.

We have completed the new experiments and incorporated the results into Appendix D.5 of the revised manuscript. Additionally, we have updated Section 4.4 in the main paper to provide a clearer explanation of the performance of the "w/o Cost map" setting and included a reference to Appendix D.5. The new experimental results, based on your suggestion, show that changing the gripper map's closed state representation from 0 to -1 in the "w/o Cost map" setting allows the encoder to correctly process this data structure. However, removing the cost map still results in a significant performance drop compared to the original model: a 11.97% decrease in testing performance on 12 base tasks and a 21.04% decrease on 8 novel tasks. These findings clearly demonstrate the critical role of the cost map in ensuring the performance of the GravMAD model.

Thank you again for your constructive feedback and support!

General Reply: We sincerely thank Reviewer SKBE for recognizing the strengths of our work, including GravMAD's strong generalization capabilities, the clear structure and logical flow of the manuscript, and the intuitive presentation of technical details and charts. We address the concerns and questions in detail below.

Insufficient argumentation

1. The manuscript mentions in section 3.2 that different sub-goal keypose discovery rules were designed for different types of manipulation tasks, which seems to lack a reasonable explanation and generalizability.
2. The manuscript does not adequately justify the exclusion of GravMap’s guidance in rotation prediction, stating that “rotation actions are difficult to be guided explicitly through value maps.”

Response to W1: Thank you for your comment. Admittedly, our sub-goal keypose discovery rules are specifically designed based on the characteristics of two main types of manipulation tasks, as detailed in lines 262–266 of Section 3.2 of the main paper. These rules are tailored to grasping tasks and contact-based tasks, aligning with the unique properties of robotic manipulation. For grasping tasks, critical interaction moments are typically signaled by changes in the gripper’s contact force or open/close state, reflecting key stages of object manipulation. Conversely, in contact-based tasks (e.g., pushing buttons), where gripper open/close actions are not relevant, key interaction moments are instead indicated solely by changes in the gripper’s contact force.

We acknowledge that this heuristic approach lacks generalizability and does not extend well to tasks involving tool use (as noted in Appendix A.1). However, it remains effective for the two task types discussed in the paper. Moving forward, we aim to address these limitations by incorporating learning-based approaches for sub-goal keypose discovery, which would enhance generalization and expand its applicability across a broader range of tasks.

We sincerely thank you for your constructive feedback, which has guided us in identifying and prioritizing areas for improvement.

Response to W2: Thank you for your comment. The primary reason we currently do not use the rotation map from Voxposer is the significant distributional shift it may introduce between the guidance provided during the training and inference phases. During training, precise rotation guidance is directly derived from expert trajectories, ensuring accuracy and consistency. However, during inference, off-the-shelf foundation models often struggle to accurately interpret 3D spatial information based on visual and linguistic inputs, resulting in less reliable rotation predictions. This discrepancy can lead to performance degradation and inconsistent task execution.

To address this limitation, our future research plans to explore combining rotation information from expert trajectories with object poses to generate few-shot prompts for off-the-shelf foundation models [1]. By leveraging these few-shot prompts, foundation models can produce more precise and effective rotation guidance, thereby reducing distributional shifts and improving inference performance.

We have included a detailed explanation of this issue in Appendix C.2 of the revised manuscript. Thank you again for your valuable suggestion, which has helped us refine and clarify this aspect of our work.

[1] Yin, Yida, et al. "In-Context Learning Enables Robot Action Prediction in LLMs." arXiv preprint arXiv:2410.12782 (2024).

Dear Reviewer SKBE,

We would like to thank you once again for taking the time to review our work and provide feedback on our manuscript. As the extended discussion deadline (December 2) approaches, this is a gentle reminder to let us know if we have satisfactorily addressed the reviewer's concerns, focusing on: clarifying our Sub-goal Keypose Discovery method (Response to W1), adding an explanation for not using the rotation map from Voxposer (Response to W2), presenting GravMAD's results on 10 real-robot manipulation tasks (Response to W3), evaluating the inference time for all models on the 8 novel tasks (Response to W4), providing a clearer explanation of the criteria for "significant changes" in our Sub-goal Keypose Discovery (Response to Q1), and further explaining the model's performance on the Place Cups and Place Wine tasks (Response to Q2). If you find it appropriate, we would be grateful if you could update your scores. We are happy to address any additional remaining concerns. We are grateful for your service to the community.

Regards,

The Authors of Submission 3039

Appreciations to the authors for their efforts. No further comments from my side.

In the ablation study w/o. Cost Map, it is surprising that the method achieves a zero score. Could the authors provide further explanation on why the predicted actions result in non-unit quaternions in this scenario? (This seems to conflict with the statement in Section 3.1, which mentions, "to avoid quaternion discontinuities, we use the 6D rotation representation.") Additionally, if the predicted quaternion is not-unit, we can normalize and convert it into a unit quternion.

Response to W3: Thank you for your feedback. We would like to clarify the relationship between quaternions and 6D rotation representations. These two concepts are not contradictory. As explained in Section 3.1 of the main paper, our model predicts a 6D rotation representation to avoid the discontinuities associated with quaternion prediction. During action execution, the predicted 6D rotation is converted into a quaternion $xyzw$, where $xyz$ represents the vector part defining the rotation axis as a unit vector, and $w$ is the scalar part corresponding to the cosine of the rotation angle. For valid rotations, the resulting quaternion is expected to be a unit quaternion, satisfying the constraint $x^2 + y^2 + z^2 + w^2 = 1$.

In our w/o Cost Map ablation experiment, we observed an issue during training where the GravMap tokens became entirely NaN, leading to gradient explosion. This, in turn, caused the predicted 6D rotations during inference to become completely disordered. As a result, the converted quaternions $xyzw$ were non-unit quaternions that failed to satisfy $x^2 + y^2 + z^2 + w^2 = 1$. This discrepancy rendered the robotic arm incapable of executing valid rotational actions, leading to a success rate of zero.

We sincerely thank you for highlighting this issue, and we apologize for the lack of clarity in our earlier explanation. In the revised manuscript, we have added further clarification on this phenomenon in lines 509–512 (highlighted in blue) to ensure a more thorough understanding.

It is advisable to conduct real-world experiments, as many related works, such as Voxposer and 3D Diffuser Actor, also include real-world demonstrations.

Response to W4: Thank you for the suggestion. Based on your comments, we have conducted real-world experiments, and the results are included in Appendix D.5 of the revised manuscript. Additionally, videos of these experiments are available on our project website: https://gravmad.github.io.

General Reply: We sincerely thank Reviewer fAE6 for recognizing the strengths of our work, including the extensive experiments with diverse baselines and a detailed ablation study, the visualizations that enhance understanding, and the paper's clarity and organization. We address the concerns and questions in detail below.

The novelty of the proposed method is limited, as it resembles a combination of previous works, Voxposer and 3D Diffuser Actor. Specifically, the authors use a similar approach to Voxposer by decomposing tasks and generating 3D spatial heatmaps for each sub-task, which are then integrated into the 3D Diffuser Actor architecture. The primary innovation appears limited to the proposed GravMAD framework.

Response to W1: Thank you for your feedback. While our method draws inspiration from Voxposer and 3D Diffuser Actor, GravMAD introduces three key innovations that go far beyond a simple combination of these approaches:

1. GravMap as a Sub-goal Representation: Unlike Voxposer, which uses voxel value maps for trajectory synthesis, GravMap acts as a crucial bridge between task decomposition and action planning, seamlessly integrating the understanding capabilities of the foundational model with the learning abilities from expert demonstrations into a unified framework.
2. Sub-goal Keypose Discovery: This novel mechanism efficiently identifies critical sub-goals in expert demonstrations, enabling the model to learn actions guided by sub-goals from expert demonstrations. This design allows the model to generalize to new tasks with high precision based on sub-goals during inference.
3. Integrated Framework: GravMAD integrates GravMap with the action diffusion process, creating a unified framework that balances the precision of imitation learning with the adaptability and generalization strengths of foundation models.

Additionally, to address the reviewer’s concern, we have provided a detailed explanation of the relationship and differences between the GravMap used in GravMAD and the value maps in Voxposer in Appendix C.1 of the revised manuscript. We hope this clarification resolves any uncertainties regarding this aspect. Once again, thank you for your valuable feedback, which has helped us improve our work.

The comparison between GravMAD and 3D Diffuser Actor on training tasks indicates that, although the introduction of GravMap improves performance on most tasks, GravMAD's performance heavily relies on both the Detector and GravMap. If the Detector’s predictions lack precision, it can even degrade the method's performance, potentially compromising its robustness in scenarios where camera capture is unclear or ambiguous.

Response to W2: Thank you for your comment. We recognize that the Detector and GravMap are key components of GravMAD, as illustrated in Fig. 2 of the main paper. The accuracy of the Detector is indeed crucial to our method's performance, as acknowledged in the analysis of GravMAD's failure cases in Appendix B.3. These cases demonstrate the limitations of the current Detector in certain scenarios.

To address these challenges, we plan to:

1. Integrate Multi-View Information: Incorporating observations from multiple perspectives to provide a more comprehensive scene understanding and improve detection accuracy.
2. Adopt a Granular Segmentation Model: Leveraging more detailed segmentation to offer foundation models richer labels, improving the context quality generated by the Detector.

To better reflect the reviewer's comments, we have incorporated the above-mentioned future improvement directions into Appendix E of the revised manuscript. Thank you again for pointing out this important area for development！

General Reply: We sincerely thank Reviewer 2Lap for acknowledging the strengths of our work, including the clarity of our presentation, the effective integration of VoxPoser with 3D Diffuser Actor, and the robustness of our results across multiple experiments. Below, we provide detailed responses to the comments and questions.

The proposed Sub-goal Keypose Discover method is kind of naive. The proposed method simply uses the change in the gripper’s open/close state and the change in touch force to segment a whole task, which could suit simple tasks like moving the chicken but might not work for tasks with longer horizons. This segmenting requires complex reasoning over the concrete task description and comprehensive understanding. During inference, foundation models like VoxPoser can be better at dividing subtasks due to the reasoning ability of LLM. However, for this proposed pipeline where different segmenting methods are used in training and testing, there could even be a domain shift for the subtasks' distribution itself due to unaligned segmenting.

Response to W1: Thank you for your thoughtful feedback. While the Sub-goal Keypose Discovery method is relatively simple, it is intentionally designed to be practical and effective for robotic manipulation tasks. We appreciate the opportunity to address your concerns in detail:

• Sub-goal Keypose Discovery: As described in Section 3.2, our method uses changes in the gripper’s state and contact force to segment sub-tasks efficiently, focusing on critical interactions while minimizing complexity. For example, in the push buttons task (Fig. 3 in the main paper), it identifies key sub-goals directly from interaction cues, allowing accurate segmentation without relying on abstract reasoning over task descriptions. This straightforward design aligns well with the requirements of robotic manipulation.
• Long-horizon tasks: Our experiments demonstrate the effectiveness of the method on long-horizon tasks, such as stack blocks. As shown in Tables 1 and 2, GravMAD achieved a 35.33% success rate improvement over the best baseline on the basic stack blocks task and a 16% improvement on the novel stack cups blocks task. These results highlight its robustness and practical applicability.
• Domain shift: We acknowledge that using different sub-goal generation methods during training and inference may introduce distributional shifts. To mitigate this, we apply data augmentation during training by introducing random offsets to sub-goals derived from expert trajectories (Line 279, Algorithm 1). This reduces the risk of distributional mismatch, and further details are provided in Appendix C.2 of the revised manuscript.

We sincerely thank you for your valuable feedback, which has helped us further improve the clarity of the paper. In the future, we plan to enhance the capabilities of Sub-goal Keypose Discovery by incorporating learning-based methods, such as leveraging diffusion models or generative models to generate sub-goals, to tackle more complex robotic manipulation tasks. To better address the reviewer’s comments, the above discussion on future improvements has been added to Appendix E of the revised manuscript.

The comparison and evaluation are limited. The selection strategy of the 12 tasks over 100 language-conditioned tasks from RLBench is not mentioned. The 8 modified tasks are not justified for their novelty, which might cause a doubt on opportunistic choices.

Response to W2: Thank you for your comment. The selection of the 12 base tasks is based on their coverage of the 8 essential action primitives required for robotic manipulation tasks. These tasks are chosen because they provide a solid foundation for learning fundamental skills, enabling generalization to more complex and diverse tasks. To clarify the criteria for selecting the base tasks, we add detailed descriptions in Appendix B.1 of the revised manuscript, highlighted in blue in Figure 7 and lines 1017–1024.

We also appreciate your feedback regarding the definition of the 8 modified novel tasks. To address your concerns, we have added detailed definitions of these novel tasks and further categorized their novelty. These 8 novel tasks introduce three distinct types of challenges:

1. Action Understanding (meat on grill, close drawer): Tasks involving changes in interaction actions with objects.
2. Visual Understanding & Language Reasoning (stack cups blocks, push buttons light, close jar banana, condition block): Tasks requiring novel operational rules or conditions compared to known tasks, including two long-horizon tasks (stack cups blocks and condition block).
3. Robustness to Distractors or Shape Variations (stack cups blocks, push buttons light, close jar banana, close jar distractor, open small drawer, condition block): Tasks requiring interaction based on fixed object attributes such as color, size, distance, or the presence of distractors.

These categories and definitions have been added to Appendix B.2 of the revised manuscript, highlighted in blue, specifically in Figure 10 and lines 1198–1209. This additional explanation aims to better illustrate how GravMAD leverages these challenges to demonstrate improved generalization capabilities.

Can you provide more details on the selection strategy of the 12 tasks over 100 language-conditioned tasks from RLBench? Why do you choose 12 and why these 12? The currently designed novel tasks feature scenes and objects similar to those in the training tasks. Can you design more complex novel tasks?

Response to Q1: Thank you for the suggestions. The selection criteria and design rationale for the 12 base tasks and 8 novel tasks are detailed in our Response to W2 above. Following the reviewer’s suggestion, we have designed 3 additional novel tasks and conducted testing with GravMAD and baseline methods on these tasks. The results of these additional experiments have been included in Appendix D.2 of the revised manuscript.

Since both baselines of VoxPoser and 3D Diffuser Actor have conducted real-world evaluations, a real-world comparison for GravMAD will be more convincing.

Can you evaluate the performance of the proposed method in a real-world experiment? Most setups should be similar to 3D Diffuser Actor and not that hard.

Response to W3 &Q2: Thank you for the suggestion regarding real-world experiments. In response, we have included the results of GravMAD on 10 real-world robotic manipulation tasks in Appendix D.5 of the revised manuscript. Additionally, videos showcasing these experiments have been made available on our anonymous project website: https://gravmad.github.io.

We appreciate Reviewer 2Lap's constructive comments and have incorporated all suggestions to improve the clarity, completeness, and rigor of the manuscript. Thank you for helping us strengthen our work!

Dear Reviewer 2Lap,

We would like to thank you once again for taking the time to review our work and provide feedback on our manuscript. As the extended discussion deadline (December 2) approaches, this is a gentle reminder to let us know if we have satisfactorily addressed the reviewer's concerns, focusing on: clarifying our Sub-goal Keypose Discovery method (Response to W1), refining the selection criteria for base and novel tasks (Response to W2), designing and experimentally testing three additional challenging novel tasks (Response to Q1), and presenting GravMAD's results on 10 real-robot manipulation tasks (Response to W3 and Q2). If you find it appropriate, we would be grateful if you could update your scores. We are happy to address any additional remaining concerns. We are grateful for your service to the community.

Regards,

The Authors of Submission 3039

We thank the reviewers (Reviewer 2Lap, Reviewer fAE6, Reviewer SKBE, Reviewer vbZp) for their valuable feedback.

We are encouraged that the reviewers recognized the strong and clear research motivation behind our GravMAD framework (Reviewer 2Lap, Reviewer SKBE, Reviewer vbZp) and acknowledged that combining imitation learning with foundation models' generalization capabilities to tackle 3D robotic manipulation tasks is both challenging and meaningful (Reviewer vbZp). The reviewers also highlighted our comprehensive experimental design, robust results, and strong generalization capabilities (Reviewer 2Lap, Reviewer SKBE, Reviewer vbZp). Moreover, they agreed that our paper is well-structured, clearly explained, and enhanced by effective visualizations (Reviewer 2Lap, Reviewer fAE6, Reviewer SKBE, Reviewer vbZp). Lastly, the ablation studies were commended for providing valuable insights into the contributions of key components (Reviewer 2Lap, Reviewer vbZp).

We have addressed all the reviewers' comments below and incorporated the corresponding revisions into the revised manuscript. The key updates in the revised version have been summarized in the previous general response, with the modified sections highlighted in BLUE.

One specific point we would like to reiterate is the "w/o Cost map" ablation experiment. We are currently conducting additional training experiments based on Reviewer vbZp's suggestion to address the zero-gradient issue. We expect the results to be available toward the later stages of the rebuttal phase. If this modification successfully resolves the zero-gradient issue, we will include the updated experimental results in the revised manuscript. Conversely, if the issue persists, we will remove this ablation experiment, as suggested by Reviewer vbZp, to ensure the clarity and scientific rigor of the paper.

We sincerely thank the reviewers for their constructive feedback and the valuable time devoted to our work. If our responses adequately address the concerns, we kindly request the reviewers to consider raising the rating and supporting us for acceptance.