EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents

Thank you for reviewing our work and providing valuable feedback. We have carefully addressed your concerns below. Please let us know if you have any further questions.

The anounymous link for figures is https://anonymous.4open.science/r/rebuttal-3568/rebuttal_file.pdf. We use "4o" to refer to GPT-4o and "Claude" to refer to Claude-3.5-Sonnet.

Q1: Detection box in EB-Manipulation. (1) How are they provided to MLLMs? (2) Do color or thickness impact performance? (3) Why does performance drop in EB-Navigation?

A1: (1) Detection boxes are directly drawn on images to guide the model’s focus on relevant regions. An example is on the right side of Figure 1 in the anonymous link.

(2) Our additional ablation studies on EB-Manipulation base subset show:

• All detection boxes improve performance over no box.
• Box color has minimal impact; even the color similar to desk (yellow) only causes a slight drop (4.2%).
• Increasing thickness (1px->2 or 3px) slightly reduces performance (<6.2%).

(3) Figure 2 in the anonymous link shows that in EB-Navigation, multiple detection boxes can obscure distant objects, reducing visibility. In contrast, EB-Manipulation keeps objects at a fixed distance, ensuring clear visibility. As a result, detection boxes affect the two environments differently.

In EB-Navigation, we tested using a single box only on the target object that can reduce obstruction, showing consistently improved accuracy. To better reflect real-world scenarios, EB-Navigation omits detection boxes by default, requiring the MLLM agent to detect and recognize objects.

Q2: Differences between high-level and low-level trajectories. Do they vary in difficulty or instructions?

A2: "High-level" and "low-level" refer to different action representations based on their executability in robotic systems (see Section 3, Paragraph 1). The trajectory structure and instructions are consistent across all high-level and low-level tasks; the key difference lies in how actions are represented—either as high-level abstractions or low-level primitives.

In terms of difficulty, our results show that low-level tasks are more challenging for MLLM agents. This is because they require stronger perception and spatial awareness, which remain limitations for current MLLMs.

Q3: How effective is ChatGPT in data generation? How is accuracy verified? Is there a human review process? How do we ensure the accuracy of the generated data?

A3: Our dataset preparation combines GPT-4o and human annotation to ensure high quality. GPT-4o is used not for full data generation but to enhance linguage instruction diversity.

For example, in EB-ALFRED, task descriptions (PDDL) and instructions for the base subset are sampled from ALFRED. For other subsets (e.g., "Common Sense"), we craft 10 examples to guide GPT-4o in generating augmented instructions. To ensure accuracy, we manually review all instructions for correctness and coherence with PDDL descriptions, revising or discarding invalid data. This human-in-the-loop approach ensures dataset reliability.

Q4: How accurate is the multi-step planner? An ablation study comparing it with single-step execution.

A4: The multi-step planner is crucial for improving performance while reducing API/inference costs. To assess its impact, we compared multi-step and single-step execution. Results show significant performance drops with single-step execution on EB-ALFRED base subset. These results confirm the importance of multi-step planning.

Q5: Evaluation of converting multi-step images into text as in-context information.

A5: We tested incorporating multi-step observation descriptions into the context. While this method did not improve GPT-4o’s performance, it led to a 4% gain for Claude-3.5 on EB-ALFRED (Base). We plan to offer this as an optional feature in our code release.

Q6: Limitations of simulation in real-world scenarios.

A6: We acknowledge the limitations of simulations in capturing real-world challenges. Please refer to Q1 of Reviewer yZa7 for a detailed discussion. We will add a discussion on the limitations in the revision.

Q7: Evaluation of qwen2.5-VL models.

A7: We evaluated Qwen2.5-VL models on EmbodiedBench and observed notable improvements over Qwen2-VL. Qwen2.5-VL-72B achieves an overall score of 34.7, surpassing previous open-source SOTA, InternVL2.5-78B (33.9). We will include evaluations of more recently released MLLMs in our updated manuscript.

Thank you for reviewing our work and providing valuable feedback. We have carefully addressed your concerns below. Please let us know if you have any further questions.

Q1: The absence of evaluation in real-world physical environments ... Embodied intelligence is largely meant to operate in the real world, and I would suggest that the authors add some real-world experiments, or some discussion.

A1: We agree with the reviewer on the importance of real-world evaluation. However, there is an inherent trade-off between reproducibility, cost, safety, and real-world applicability. While real-world testing is crucial for practical deployment, simulated benchmarks provide a standardized and easily reproducible environment, reducing the time, financial burden, and safety risks associated with real-world evaluation [1,2]. EmbodiedBench is a step forward in enabling evaluating MLLM agents in diverse simulated embodied tasks. Future research could benefit from more realistic and complex embodied simulations [3] or standardized and cost-effective real-world test suites [4,5]. In our revision, we will add a discussion on this limitation at the end of the main paper.

[1] Evaluating Real-World Robot Manipulation Policies in Simulation. CoRL 2024.

[2] Visualagentbench: Towards large multimodal models as visual foundation agents. ICLR 2025.

[3] Behavior-1k: A human-centered, embodied ai benchmark with 1,000 everyday activities and realistic simulation. ArXiv, 2024

[4] Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. RSS 2023.

[5] Mobile aloha: Learning bimanual mobile manipulation using low-cost whole-body teleoperation. CORL 2024.

Q2: The review includes a very large number of MLLMs, but the VLA model is missing. I suggest adding some experiments.

A2: We appreciate the reviewer’s suggestion. We reviewed VLAs in paragraph 3, Appendix A. While VLA models such as Octo, OpenVLA, and RDT-1B have shown strong performance in manipulation tasks, there is currently no open-source VLA model that can handle both high-level household tasks and low-level navigation & manipulation simultaneously. Consequently, there is no directly comparable VLA model for our evaluation.

However, to explore their capabilities, we conducted experiments on three pretrained VLA models (Octo, OpenVLA, and RDT-1B) using the EB-Manipulation (Base) subset. All models achieved a 0% success rate. This outcome can be attributed to the distribution shift: existing robotic foundation models [6][7] are primarily trained on real-world datasets such as Open X-Embodiment. The large domain gap between these datasets and our simulator environment prevents these models from generalizing effectively without fine-tuning. This reinforces the need for a more general framework, as proposed in our paper, that enables adaptation to diverse tasks without fine-tuning.

While a few VLA models [8][9] have been developed for the same simulator we use, they are not publicly available. Furthermore, other available models [10][11] have input formats that differ significantly from our environment (e.g., proprioception mismatch), making direct evaluation infeasible. The only available VLA model trained in the same simulator as ours is 6D-CLIPort [12], which processes multi-view observations and language instructions to generate 6-DoF actions. Below are the evaluation results on EB-Manipulation:

The model performs well on tasks that rely on visual understanding and spatial reasoning. However, its success rate drops significantly on tasks requiring common sense reasoning or understanding complex instructions, suggesting that action-grounded VLA models may struggle in these areas. In contrast, our MLLM-based agent framework achieves higher performance, with GPT-4o reaching an average score of 28.9. This result highlights the effectiveness of our MLLM agent framework and the potential of general MLLMs as embodied agents.

[6] OpenVLA: An Open-Source Vision-Language-Action Model. CORL 2024.

[7] RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation, ICLR 2025.

[8] MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulation. Arxiv 2025.

[9] HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation, ICLR 2025.

[10] RVT-2: Learning Precise Manipulation from Few Examples. RSS 2024.

[11] SAM2Act: Integrating Visual Foundation Model with A Memory Architecture for Robotic Manipulation. Arxiv 2025.

[12] VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation, NeurIPS 2022.

Thank you for reviewing our work and providing valuable feedback. We have carefully addressed your concerns below. Please let us know if you have any further questions.

Q1: MLLMs performance might relate to the prompt being used, and one single prompt is used for all the models. Exploring how MLLMs work with a different set of prompts might bring more insights and robust conclusion about how well different MLLM work for embodied tasks. Some ablations for the language prompt provided in Sec.5.3 can partially mitigate this issue.

A1: We agree that prompt design plays an important role in MLLM agent performance. In Section 5.3, we analyzed the effects of textual environmental feedback and the number of in-context examples. To further investigate prompt robustness, we conducted additional studies on the EB-ALFRED "Base" subset, evaluating the following variations:

1. "No Guideline" – Removing instructional guidelines intended to assist model generation (see Page 19).
2. "Prompt v2" – Rewriting the original prompts using GPT-4o to rephrase and restructure while preserving similar information.

The results show that removing guidelines has no effect on model performance, while "Prompt v2" causes a slight 4% performance drop. This suggests that our MLLM agent is relatively robust to prompt modifications compared to its sensitivity to environmental feedback (around 10% drop) and in-context examples (more than 20% drop).

Furthermore, we also examined the impact of reasoning in in-context examples (i.e., ReAct prompting [1]). Removing this reasoning step leads to an 8% performance drop for GPT-4o and a 4% drop for Claude-3.5-Sonnet, indicating that reasoning within in-context examples is more impactful than prompt rephrasing or guidelines, though still secondary to environmental feedback and in-context examples.

In summary, our MLLM agent framework is robust to prompt rephrasing and guideline removal. However, environmental feedback and in-context examples, as discussed in Section 5.3, have a much greater impact on performance.

[1] React: Synergizing reasoning and acting in language models. ICLR, 2023.

Thank you for reviewing our work and providing valuable feedback. We have carefully addressed your concerns below. Please let us know if you have any further questions.

Q1: No real-world evaluations. It is hard to say how aligned with the real-world performance

A1: We agree with the reviewer regarding the gap between simulated and real-world tasks. While real-world evaluation is crucial for assessing model performance in practical scenarios, there is a trade-off between reproducibility, cost, safety, and real-world applicability. Simulated benchmarks offer a standardized, easily reproducible environment, reducing the time, financial cost, and safety risks for researchers to replicate results [1,2]. In our revision, we will include a discussion about this limitation in the main paper.

[1] Evaluating Real-World Robot Manipulation Policies in Simulation. CoRL 2024.

[2] Visualagentbench: Towards large multimodal models as visual foundation agents. ICLR 2025.

Q2: How to prevent failure of plans? Any qualitative examples?

A2: In EmbodiedBench, we allow MLLM agents to generate multiple actions at once. If a failure occurs (e.g., invalid actions or unmet goals), they replan using the latest image and interaction history. In Appendix F, we provided four qualitative examples of our MLLM agents (Figures 13–16), demonstrating their ability to replan effectively.

Q3: Some results are repeated and can be (optionally) removed from appendix

A3: In Appendix D, we opt to include both the results from the main paper and additional results to provide a clearer trend across different tasks.

Q4: Using panoramic images for multi-view

A4: We included a panoramic image in our multi-view ablation for EB-Navigation. It provides a top-down perspective, capturing the entire scene. However, we found that it can mislead the agent, negatively impacting overall performance. Examples of the multi-view setup are shown in Figure 1 of https://anonymous.4open.science/r/rebuttal-3568/rebuttal_file.pdf.

Q5: Train policies on this benchmark and show performance ... this is not a necessity since the purpose of the benchmark is to evaluate MLLMs.

A5: We agree that further fine-tuning is a promising direction, but a key challenge is the lack of training data. Currently, the only available embodied planning dataset in our setting is ALFRED, which lacks the structured perception and reasoning. To address this, we are collecting trajectories using our agent framework on the base subset while reserving other subsets for evaluation. This will benefit future fine-tuning MLLM research, and we aim to release the dataset soon.

Q6: Difference between the complex instruction understanding and long-horizon tasks. Any intersection?

A6: The two subsets target different challenges:

• "Complex Instruction" adds longer, relevant/irrelevant context, making user instructions harder to interpret while keeping task complexity similar to the base subset.
• "Long Horizon" increases task difficulty by requiring more steps to complete.

In EB-ALFRED, GPT-4o takes an average of 13.4 steps (base), 14.2 steps (complex instruction), and 23.9 steps (long-horizon), showing their differences. We ensure no subset overlap in our benchmark design.

Q7: How is the data generated for EB-Navigation? Which Python program was used? I did not find this in appendix.

A7: We describe the data generation process in Appendix B.3 but will clarify it further in our revision. The EB-Navigation dataset consists of: 1. scene and object information, 2. initial robot position and pose, 3. target object information, 4. language instruction. We create the dataset using a Python program that ensures the validity of the (1,2,3,4) combinations. The process follows these steps:

• Step 1: Scene and Object Initialization
  We use 90 scenes from AI2-THOR-supported scenes. Each scene initializes a set of objects.
• Step 2: Target Object Selection
  In each scene, we iterate over potential target objects, excluding those inside receptacles or with multiple instances to reduce ambiguity.
• Step 3: Agent Position and Pose Determination
  For each target object, we use AI2-THOR’s GetInteractablePoses to randomly sample a valid agent position, ensuring a distance at least 2.5 meters from the target. The agent's pose is set to either include the target object in view (e.g., base subset) or keep it out of view (long horizon).
• Step 4: Instruction Generation and Augmentation
  Based on predefined templates (e.g., "Move towards the {target object} and stay near it"), we apply GPT-4o to augment linguistic diversity and preserve subset requirements.

After executing the above data generation, we select 60 tasks for each subset to form the EB-Navigation dataset.

Q8: Typos

A8: Thank you for pointing out these typos. We have carefully corrected them in our revised manuscript.