HomieBot: an Adaptive System for Embodied Mobile Manipulation in Open Environments

Thank you for your constructive comments. We have carefully considered your suggestions and would like to provide the following clarifications. We have also updated the manuscript, with the modified contents highlighted in orange for your review.

Q1: In Section 3, it is inappropriate to describe the dataset for model training (Section 3.2, Data Augmentation for SFT and DPO) here. This dataset is intended for training the embodied agents, not for constructing the benchmark tasks.

A1: Thank you for your suggestion on article organization, which is very helpful. We have moved the original Data Augmentation section to Section 4.1, the subsection before model training. We have also added more detailed of data augmentation in Appendix and supplementary materials.

Q2: The work aims to evaluate high-level planning and low-level execution abilities. However, there is no individual assessment of these abilities (e.g., the success rate of high-level planning, low-level execution, etc.). The relationship between the proposed evaluation metrics (TP, SER, SRR) and these abilities is unclear.

A2: We are sorry for the incomplete experiments and evaluation in the manuscript. Based on previous results, we refine the error definitions within Section 3.3, separating errors from each module to facilitate a comprehensive capability evaluation of HLP. By further classifying errors that are irrelevant to HLP, we can determine the success rate of each low-level skill. We conduct detailed error analysis in Section 4.5, all error statistics and low-level evaluation can be seen in Fig 4, Table 4 and Appendix H.2, we also provide more case studies in Appendix I. During the evaluation process, TP is primarily used to assess overall task completion, as mentioned in Section 2.2. It mainly reflects the evaluation of the task execution process and supplements the shortcomings of existing success rate evaluation methods. SER is closely related to the model's inherent judgment and logical reasoning abilities, while SRR mainly reflects the model's adjustment and generalization capabilities. We have added more analysis of the relationship between these metrics and the model's capabilities in Section 4.4.

Q3: The problem is not well defined. It would be beneficial to provide a problem formulation regarding high-level planning and low-level execution in the paper.

A3: Thank you for pointing out the issues in our paper writing and organization! We have added Section 3.1 to provide a detailed explanation of the EMMOE problem, as well as the functions of HLP and LLE. More specifically, the process of EMMOE tasks can be described as: the agent needs to make embodied decisions based on current environments and historical execution records in open environments, then navigates and manipulates within a continuous space, the obtained results and feedback will be used for the next decision. We divide this process into two main parts: HLP and LLE. HLP is responsible for embodied decision-making and planning adaptation while LLE handles continuous execution and provide feedback to HLP.

Q4: Some content in the article is unclear. For instance, on page 4, line 207, the statement “if subtask i fails but subtask i + 1 succeeds” lacks clarity regarding the relationship between subtask i and subtask i+1. From my understanding, subtask i and i+1 are the same task but with different model responses (actions). This needs to be clarified.

A4: We are sorry for the confusion caused by our unclear expression. As you inferred, for the $i$th subtask and its input instruction $I_i$, if the execution of model output $O_i$ fails but the next output $O_{i+1}$ succeeds after re-plan, we will choose $I_i$ as the prompt, $O_i$ as the rejected and $O_{i+1}$ as the chosen. We have also refined the algorithm description in Section 4.1 and provided the code implementation in Appendix F.2 and supplementary materials.

Thank you for your constructive comments. We have carefully considered your suggestions and would like to provide the following clarifications. We have also updated the manuscript, with the modified contents highlighted in orange for your review.

Q1: Limited baseline comparisons, which reduces the clarity of contributions relative to existing methods.

A1: Thank you for your beneficial suggestions! We have included evaluations of GPT-4o[1], Gemini-1.5-pro[2], Qwen2-VL-7B[3], MiniCPM-V 2.6[4] on the EMMOE dataset. The evaluation results and comparisons are as follows:

More detailed of evaluation settings and model deployment can be found in Section 4.3 and Appendix H.1.

Q2: Lack of diverse model comparisons, including text-only and zero-shot multi-image baselines (e.g., Qwen2VL, GPT-4o), and no modular vs. end-to-end performance analysis.

A2: Thank you for your valuable suggestions on our experiment. We have now included comparisons between different models and we would like to remind that EMMOE is a challenge including task planning, embodied decision-making, navigation and manipulation simultaneously. In the execution videos provided in our supplementary materials, you can see that our tasks are not only complex but also involve lengthy execution sequences. Current end-to-end models mainly focus on single visual navigation or tabletop manipulation and are not capable yet of completing all these tasks with a single model. Additionally, HomieBot and end-to-end models are not on the same level. HomieBot aims to build a platform that can evaluate different models simultaneously, which means end-to-end models can also possibly be evaluated in LLE module. The information transmitted from HLP to LLE is still relatively advanced natural language, and the subtask is more aligned with the "pick anything" or "go to anyplace" tasks studied in end-to-end models.

Q3: Insufficient analysis of task-specific challenges and bottlenecks within the benchmark; error analysis and reasoning failure modes are explored to some degree, but not in detail.

A3: Thank you for your insightful suggestions which have been extremely helpful in enhancing our work. First, we refined the error definitions within Section 3.3, allowing errors from different modules to be separated for easier statistical analysis. First, we further refined the error definitions in Section 3.3, separating errors across different modules to facilitate more straightforward statistical analysis and evaluation. Next, we collect all errors during the execution process and categorized them, examining the impact of each error on task performance. By further refining and analyzing these subcategories, we also determine the success rate of each low-level skill. Then we summarize several most predominant challenges and bottlenecks within the benchmark:

• Physical grounding problems and model hallucinations.

• Insufficient understanding ability when the execution steps become very long.

• Limited low-level skill performance, especially pick.

Section 4.5 contains a more detailed error analysis, all error statistics and low-level evaluation can be seen in Fig 4, Table 4 and Appendix H.2, we also conduct more case studies to further explore current bottlenecks in Appendix I.

I thank the authors for their response and the revisions made to the manuscript. The addition of new baselines and analysis has greatly strengthened the work, and I have raised my score to a 5. However, the significant changes to both the results and the manuscript make it challenging to fairly assess the current version in comparison to the original submission. For this reason, I still believe that a resubmission would be more appropriate.

Thank you for your response and for acknowledging our changes! We sincerely apologize for dificiencies in the initial submission. We also want so clarify that all methods in the new submission are entirely consistent with the previous version, and all original experimental results are included in the new version. The new additions aim to address the reviewers' concerns and enhance the work, and our contributions stay unchanged. We hope to make any possitive changes to our work based on your constructive suggestions!

Thank you for your constructive comments. We have carefully considered your suggestions and would like to provide the following clarifications. We have also updated the manuscript, with the modified contents highlighted in orange for your review.

Q1: Insufficient Methodology Details. The data processing and augmentation methods are not adequately explained

A1: Thank you for your helpful suggestions and attention to the details of our methods. We have revised the description of the DPO Augmentation in Section 4.1 to make it more understandable and clearer. Additionally, we've added the code implementations for SFT and DPO Augmentation in Appendix F, along with samples of the converted SFT and DPO data. These codes will convert the EMMOE data shown in Appendix B into a dialogue format suitable for training. Furthermore, we've released the anonymized code for data augmentation in our supplementary materials to facilitate understanding.

Q2: No evaluation of existing RL algorithms or LM-based agents on the EMMOE benchmark

A2: Thank you for pointing out the shortcomings in our work! We have added evaluations of GPT-4o[1], Gemini-1.5-pro[2], Qwen2-VL-7B[3], MiniCPM-V 2.6[4] on the EMMOE dataset. The evaluation results are shown as follows:

More detailed of evaluation settings and model deployment can be found in Section 4.3 and Appendix H.1.

Q3: Only one system (HOMIEBOT) is evaluated, lacking comparative analysis

A3: Thank you for your valuable suggestions on our work! First, we would like to clarify that EMMOE is a new challenge and differs from previous benchmarks. Previous benchmarks often simplify either the decision-making or practical execution aspects to evaluate a single capability, while EMMOE simultaneously addresses multiple challenges, including task planning, embodied decision-making, navigation and manipulation. Due to the differences in tasks, existing systems are not fully applicable to EMMOE. To our knowledge, HomieBot is currently the only system capable of performing all these evaluations simultaneously. Secondly, the main contribution of this work lies in proposing a new benchmark and comprehensive evaluation methods. In the newly added Section 4.5, we demonstrate how HomieBot can be used to evaluate different models. HomieBot serves more like a platform where various models or systems can be integrated after adjustments, as shown in Appendix H.1.

Q4: The relatively low success rates (31.8% training, 20% testing) are not thoroughly analyzed. Missing ablation studies or detailed error analysis to understand performance bottlenecks

A4: Thank you for your instrumental suggestions in improving our work. Here's how we addressed them and make performance analysis:

• We refine the error definitions in Section 3.3, enabling errors from different modules to be separated for more straightforward statistical analysis and evaluation.

• We collect all errors occurred during evaluation process and make a detailed classification, evaluating the impact of each error on task performance. By further analyzing these subcategories, we determine the success rate of each low-level skill.

• We conduct more case studies to further explore the current bottlenecks in Appendix I.

Our main findings of current bottlenecks hindering the model’s performance can be summarize as:

• Issues with physical grounding and model hallucinations.

• Insufficient understanding capacity when execution steps become excessively long.

• Limited performance of low-level skills, particularly in action pick.

Section 4.5 provides a more detailed error analysis, with comprehensive error statistics and low-level evaluations presented in Figure 4, Table 4, and Appendix H.2.

References

[1] GPT-4 Technical Report arXiv:2303.08774

[2] Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context arXiv:2403.05530

[3] Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution arXiv:2409.12191

[4] MiniCPM-V: A GPT-4V Level MLLM on Your Phone arXiv:2408.01800

Thank you for your constructive comments. We have carefully considered your suggestions and would like to provide the following clarifications. We have also updated the manuscript, with the modified contents highlighted in orange for your review.

Q1: Overclaim: it does not seem like the first to unify the high-level and low-level. This has been discussed in Robotics Task and Motion planning (TAMP) literature and has been introduced as baselines in previous simulator benchmarks. Connection with previous work and clarity on the unique contribution of this work is needed.

Thank you for your suggestion! We think this misunderstanding stems from our unclear definitions of HLP and LLE in the article. The process of EMMOE tasks can be described as: the agent needs to make embodied decisions based on current environments and historical execution records in open environments, then navigates and manipulates within a continuous space, the obtained results and feedback will be used for the next decision. We divide this process into two main parts: High Level Planning (HLP) and Low Level Execution (LLE). HLP is responsible for embodied decision-making and planning adaptation while LLE handles continuous execution and provide feedback to HLP. From these definitions, it is clear that our HLP differs from the high-level in TAMP. Making decisions in open environments means we won't provide any background information to the planner, it must rely on its own capabilities to make decisions and dynamically interact with the environment. Current embodied decision-making often focuses solely on planning, simplifying the execution to varying degrees. However, a more reasonable evaluation should assess both high-level and low-level processes together, as execution and decision-making are interconnected, with execution influencing decision-making. This is also one of the motivations behind the proposal of EMMOE.

Q2: Lack of failure analysis: The paper does not discuss the limitations and failures modes of the combined high-level and low-level modules.

We are sorry for the incomplete discussions of our methods. We refine the error definitions within Section 3.3, separating errors from each module to explore limitations and bottlenecks in current modules. By further classifying errors that are irrelevant to HLP, we determine the success rate of each low-level skill. We conduct detailed error analysis in Section 4.5, all error statistics and low-level evaluation can be seen in Fig 4, Table 4 and Appendix H.2, we also provide more case studies in Appendix I.

Q3: Limited discussion: It is unclear why the success rate for HOMIEBOT is so low in training and test tasks.

We thank you for your attention to our experimental results. We add more comparisons with other LMMs, combining methods and conclusions mentioned in Q2, we attribute it to the following reasons:

• Insufficient understanding ability when the execution steps become very long.

• Physical grounding problems and model hallucinations.

• Limited low-level skill performance, especially pick.

• The phenomenon of historical forgetting during model inference.

Notably, we measure the average success rate of trajectories, not how many tasks can be completed. This method of measurement is certainly lower than the latter but is more stable and fair. Detailed analysis can be found in Section 4.4 and Section 4.5.

Q4: Clarity: Paper is not well formatted and has lots of typos (line 180 control, 185-186 between)

A4: We are sorry for the unclear writing and we have reorganized the manuscript and corrected the grammatical errors.

Q5: Limited novelty: with discrete action space, the work looks very similar to ALFRED (Shridhar et al.) and FiLM (Min et al.) with a few enhancements. Why not use continuous action space? Habitat Replica allows for continuous action space. HM3D

A5: We think this misunderstanding was raised by our wrong description. Actions for the HLP are discrete and require selection from an available action list. However, once an action is selected, it will be executed in continuous space. The agent must perform each task in continuous space and receives feedback from the actual interaction, then send them to HLP. This is also what previous work based on the ALFRED dataset couldn't achieve. Due to the limitations of the simulator, the agent couldn't perform actions in continuous space or obtain real interaction information. We have revised the inappropriate description of the dataset in the manuscript and we strongly recommend viewing the demonstration video provided in the supplementary materials, where can clearly see that the actions are executed in continuous space.