ET-Plan-Bench: Embodied Task-level Planning Benchmark Towards Spatial-Temporal Cognition with Foundation Models

We extend our gratitude to the reviewer for their insightful comments. Below, you will find our comprehensive responses to your primary questions.

1. Comments: Limited novelty in contributions: Task generation using LLMs has been explored extensively in prior work The benchmark tasks are relatively simple and constrained

Response: We wish to further clarify this point with the reviewer. We compared our benchmark with previous work, as presented in Table 1, and demonstrated the advantages of our proposed benchmark. "Including modality (natural language, vision, or both), vocabulary type (open or constrained), data size, and the capacity for unlimited data generation via the provided pipeline. Additionally, it assesses whether benchmarks evaluate the planning abilities of large language models (LLMs), whether data generation is automated or human-annotated, and the consideration of spatial, temporal, or causal constraints." In addition, we also delineated the position of our work in Section 2 Related Work.

2. Comments: Significant limitations in the experimental setup: The simulators (Virtual Home and Habitat) are quite restricted in terms of task complexity and environmental interactions Agents do not have true perception - they receive text descriptions rather than visual input. The spatial and temporal constraints are quite basic

Response: We kindly wish to clarify that there may be some misunderstandings regarding our paper. The entire LLM agent pipeline process images. In the paper, we utilize the perception modules: the perception API in VirtualHome and GroundingDINO in Habitat, to extract object information from the images. This object information is then fed into the LLM to facilitate decision-making. Consequently, we do not employ VLMs at the current stage, as VLM agents present greater challenges that we intend to explore in future work. In addition, the tasks involving the spatial and temporal constraints we have proposed currently pose significant challenges for LLMs. Although the idea is basic, much of the related work does not adequately consider it.

3. Comments: Lack of rigorous analysis and insights: The LLM agent comparison results are presented as they are without novel insights No clear framework for systematic task generation and evaluation Limited discussion of how the findings could improve embodied AI systems

Response: Thanks for your comments. 1) We have incorporated insights of LLM agent comparison results in the revised version: "To verify the validity of our proposed benchmark and to demonstrate that tasks become more challenging after the addition of spatial or temporal constraints, we present comprehensive experimental results to explore the performance of our proposed LLM agent across various tasks in multiple simulators." and "To assess the impact of various LLMs on the evaluation process, we have additionally evaluated the benchmark with more LLM models." 2) We have revised our paper to the best of our ability to improve clarity of framework for systematic task generation and evaluation. 3) We have highlighted the primary contributions of our paper in the Section Conclusion: "Overall, our proposed benchmark can assist researchers in evaluating the performance of their LLM agents in completing complex embodied planning tasks that involve spatial and temporal constraints."

4. Comments: Compared to recent work like "Embodied Agent Interface: Benchmarking LLMs for Embodied Decision Making https://arxiv.org/abs/2410.07166" (note that this is not considered part of related work as the submission is concurrent), this paper: Lacks a principled framework for formalizing different types of embodied tasks Does not provide fine-grained error analysis Misses systematic evaluation of different LLM capabilities

Response: In Figure 3, this framework formalizes various types of embodied tasks through a task classification module and task-specific component retrieval module. For the purpose of better evaluation, we assess each type of task to determine its difficulty. Additionally, we present Case Study and Error Analysis in Appendix A.1 Successful Cases of Embodied Planning Tasks and Appendix A.2 Failed Cases of Embodied Planning Tasks. Furthermore, we evaluated several different LLMs for our proposed benchmark, as shown in Table 3: Evaluation with open source and closed source LLMs including GPT-4, Llama3-70B, Mistral-8x7B and Llama3-8B.

5. Comments: The writing is quite poor. It did not explain the setup clearly. I have to find information in the appendix to continue reading.

Response: We apologize for the extensive appendices included. Due to the page limit, we had to relocate many details to the Appendices. We have revised our paper to enhance clarity.

We thank the reviewer for their insightful comments and your encouraging feedback. Please find our overall concise response to your main questions below.

1. Comments: Minor: two sentences from the abstract are nearly copied to the beginning of the introduction this is unnecessary and should be avoided.

Response: Thank you for your suggestions. We have addressed and avoided this issue in our revision.

2. Comments: Major: At the moment, the presentation Is quite unclear. While the overall goal, using LLMs to both generate tasks and solve embodied AI problems is apparent, the method, the primary assumptions and overall system are hard to understand.

Response: Thank you for your comments. We have revised our paper to the best of our ability to improve clarity.

3. Comments: Major: It's not clear what using the LLM in task generation accomplishes. Is the goal just to provide common sense reasoning over the larger task search space? Otherwise, it seems like these tasks can be generated fairly mechanically as others have done in the past. To be clear my complaint is not that the LLM serves no purpose, my complaint is that it is not well described (again a clarity issue), and that it's hard to tell how much this adds.

Response: Thanks for your suggestion. We have included an example to clarify the use of the LLM in task generation: "After retrieving the relevant information from the environment, we shortlist all possible item combinations that meet the requirements of the commission template. For instance, the task of placing object A into container B necessitates retrieving both object A and container B from the environment. We subsequently employ LLMs to generate tasks adhere to principles of logical common sense and are rephrased in diverse ways to increase the diversity of descriptions. The LLMs receive both the environmental information and the task template as part of the prompt, which also includes rules and examples to ensure the tasks generated are realistic." The basic idea involves first defining a task template and then filling it with object names in the template's blank spaces. The next step is to use a LLM to filter out unreasonable tasks, thereby ensuring the diversity of reasonable task descriptions. Two example prompts of the LLM are shown in Table 8 and Table 9 in the Appendix.

4. Comments: Medium: It says that humans evaluated these tasks based on "reasonableness" which seems highly subjective and difficult to quantify.

Response: We respectfully wish to clarify that there may have been some misunderstandings regarding our paper. In Section 3.2, "we randomly selected 5 simulation environments and chose 50 tasks from each environment for human evaluation, and our findings indicate that approximately 4% to 8% of the tasks were deemed unreasonable." We agree that human evaluation is inherently subjective; however, we do not rely on it to filter out unreasonable tasks. Instead, we employ human evaluation to ensure that the majority of the tasks we generate appear reasonable; in other words, they seem common to humans. Although a small number of tasks were deemed unreasonable or unusual based on our human evaluation, they can still be successfully executed in the VirtualHome simulator. This is because we systematically filter out unreasonable tasks using LLMs and simulator verification during our task generation process. One example of an unreasonable task by human evaluation we mentioned in the paper is "Find the face cream tube and put it into the folder." While some might consider this task reasonable or common in practice, we classify it as unusual due to our relatively strict human evaluation criteria.  Other examples that were deemed unreasonable or unusual based on human evaluation are as follows: "Your task is to retrieve cellphone and deposit it into box, subsequently transferring both into garbagecan.", "Find a bunch of bananas and put them into the basket located by the toilet." and "Find the toothbrush and place it in the coffeemaker's cup."

5. Comments: What is the action space and observation space in this setting?

Response: The action space in this setting includes WALK, OPEN, CLOSE, GRAB, PUT, and PUTIN, while the observation space comprises a 360-degree scan. These concepts were introduced in Section 4.1.

We thank the reviewer for their insightful comments. Please find our overall response to your main questions below.

1. Comments: Section 1 — The manuscript asserts that foundation models struggle with spatiotemporal reasoning; benchmarking Vision-Language-Action (VLA) models seems particularly relevant to this assertion. https://arxiv.org/html/2411.05821v1

Response: Thank you for sharing the paper. However, since it was posted online after the ICLR submission deadline, we were unable to review it. Upon reviewing this paper, I am uncertain where the discussion of spatial reasoning is addressed, although temporal reasoning is mentioned but unclear.

2. Comments: Section 2 — The manuscript states, "We introduce a benchmark for household embodied task planning rather than question answering about the environment. It focuses on skill-level task decomposition rather than lower-level control." (L179-181). This is not sufficiently motivated. As better foundation models become available, skill-level task decomposition requires less analysis and optimisation, compared to spatiotemporal reasoning for low-level control.

Response: We kindly wish to clarify that there may be some misunderstandings regarding our paper. We revised this sentence for clarity: "We introduce a benchmark for household embodied task planning rather than question answering about the environment. This benchmark includes task descriptions and success criteria within a virtual environment. Any methods employing LLMs can be evaluated using our benchmark due to the substantial diversity in task descriptions, ensuring that LLMs will be employed for task decomposition at least once. Currently, the LLM component of the proposed LLM agent baseline focuses on skill-level task decomposition rather than lower-level control." Spatiotemporal reasoning also relies on LLMs/VLMs because our proposed LLM agent baseline also takes spatiotemporal reasoning into account. Low-level control pertains to actions such as moving to an intermediate position or manipulating a specific object. Since VirtualHome does not allow modification of these low-level controls and we can only utilize the low-level control tools provided by VirtualHome, we focus solely on high-level task decomposition while incorporating spatiotemporal reasoning.

3. Comments: Section 3 — Is there a risk that, if tasks and evaluations are performed by LLMs, methods that consist of the same LLMs may obtain inflated performance evaluations? The evaluations would then be in-domain. This phenomenon would be catastrophic, as agents' behaviors would be rewarded in ways that do not necessarily lead to improved behavior in the real-world. For validating the effectiveness of this benchmark for suitability in assessing model performance, the manuscript should establish that performance in this benchmark correlates with performance, e.g., on tasks extracted from multiple pre-existing benchmarks and/or on real-world tasks from large-scale datasets, (e.g., DROID, OXE). Furthermore, to further avoid the test-set leakage issue above, perhaps the proposed method for task generation should include some sort nondeterminism, e.g., through a set of structured perturbations, common in the safety-critical scenario generation literature.

Response: In our study, tasks were generated using GPT-4 and evaluated using several different open-source and closed-source LLMs as summarized in Table 3. The performance of the Llama3-70B model is comparable to that of GPT-4. Consequently, there is no inflated performance evaluation. The primary objective of our work is to introduce an embodied planning benchmark rather than a novel LLM agent. We evaluated different LLMs to establish a LLM agent baseline, thereby verifying the validity of our benchmark. Therefore, our focus was not on evaluating LLMs on other benchmarks.

1. My intention in referencing that work was not to suggest it as a single baseline that should be included; I provided the resource in the spirit of assisting the authors, so that the authors may complete their literature review and gain inspirations for tweaking their experiments to provide relevant comparisons. In the context of VLAs, OpenVLA (https://arxiv.org/pdf/2406.09246v1) is one model that comes to mind for many; that manuscript first became available in June. The reason for requesting VLA baselines was that these are a class of models that pursue better spatiotemporal reasoning for low-level control, but may still have some limitations, which seems central to verifying the manuscript's claim here. This underscores one of my main concerns with the present manuscript in that there seems to be lacking comparisons with other environments and lacking inclusion of relevant baselines.

2. It seems that there was no misunderstanding about the paper here. In my original review, I suggested that the manuscript did not sufficiently motivate why skill decomposition should be prioritized over spatiotemporal reasoning for low-level control. In the authors' response, they now explain that the reason was that the experimental environments they selected do not support the study of spatiotemporal reasoning for low-level control. I still feel as if this is insufficient motivation for even the revised statement. Are the authors suggesting that there are no environments that can be used for studying 'spatiotemporal reasoning for low-level control'? If so, this needs to be asserted and justified. If this is not what the manuscript intends to suggest, it should comment on why it did not select better environments for its analysis.

3. The manuscript proposes a new benchmark. For a new benchmark to be useful for the community, the manuscript must establish a few things: (a) the benchmark can be used to identify the reasons why different model types have different strengths and weaknesses; (b) the benchmark sheds light on a relatively new problem, perhaps one that is under-explored in the literature, so that it cannot be easily solved "tomorrow" and/or encourages significant methodological progress to be made; and (c) the benchmark provides valuable contributions over other similar benchmarks. The proposed benchmark mostly pursues (a), but I feel like (b) and (c) are still lacking. I discuss further concerns with (c) in the next point.

4. The authors offered high-level discussion of their work, relative to RoboGen, in Section 2 Related Work: "RoboGen employs LLMs to generate infinite tasks and scenes for its simulation engine. Our work distinguished by its focus on generating complex tasks in a controlled manner, utilizing various types of constraints to create more realistic and challenging embodied tasks." How can the proposed work claim that its tasks are "more realistic and challenging", without any form of direct comparison and without metrics that govern, e.g., 'realism' and 'environment complexity'? I (still) feel as if the manuscript is lacking crucial experiments here. 

5. See point #1, above.

6. As I mentioned in my original review, "it seems that the usefulness of the benchmark is directly tied to the underlying models’ abilities to perform task generation and evaluation flawlessly." It is true that other researchers can switch out and optimize the evaluating LLM agent, in order to obtain an improved evaluation of model performance under the proposed framework. In order to motivate the community to do this, however, the manuscript should demonstrate that using an LLM to evaluate model performance is reliable. Understanding models' behavior, detecting failure, suggesting grounded remedial actions, scoring performance, etc. are active research directions in the Embodied AI and Robotics communities (see, e.g., https://arxiv.org/pdf/2306.15724v1). These approaches still show significant limitations when attempts are made to use them for assessing agent behavior, hence my original question/concern: How does the proposed framework limit the failure cases of foundation models from affecting the efficacy of model evaluation?

I am also monitoring the discussions with the other reviewers, and I still retain the above concerns from my original review.

I will retain my current score for now.

We thank the reviewer for insightful comments. Please find our overall response to your main questions below.

1. Comments: Although the paper claims that an infinite amount of tasks can be generated. It's unclear whether the diversity is bottlenecked by the scenes the simulator provides and the actions (including the transition models of the actions) the simulator supports.

Response: Thank you for your comments. Although the number of scenes (e.g. 50 scenes for VirtualHome) and actions supported by the simulator are limited, and we acknowledge the difficulty in designing new actions within the simulator, an unlimited number of scenes, objects and their layout can be easily created and loaded to virtual environments. Theoretically, this diversity of scenes and objects should be sufficient. In addition, we can also generate infinite number of tasks by changing the initial position of the agent in the same scene.

2. Comments: It's unclear what new conclusions or insights can be drawn from this benchmark. I hope the authors can provide a more detailed analysis and categorize the failure modes.

Response: Thank you for your suggestion. 1) We have made revisions to the Conclusion section by adding new insights, as follows: "Overall, our proposed benchmark can assist researchers in evaluating the performance of their LLM agents in completing complex embodied planning tasks that involve spatial and temporal constraints." 2) Additionally, we included some key contributions of our paper in the Introduction section: "Embodied planning refers to an agent's ability to formulate plans and execute tasks within a physical environment. Large Language Models (LLMs) and Vision Language Models (VLMs) have demonstrated significant advancements in vision understanding, natural language comprehension, and generation. Although LLMs and VLMs are not inherently designed for embodied planning, there is potential for these models to contribute to this field. LLMs and VLMs possess an extensive repository of knowledge derived from their training data, which enables them to comprehend and generate contextually relevant advice and strategies. Additionally, they have the capability to translate complex tasks into step-by-step instructions through interactions with humans. Furthermore, they can refine plans based on feedback from the environment or human interventions. However, currently, LLMs and VLMs face some challenges in understanding the physical world, including spatial understanding. Effective spatiotemporal reasoning often necessitates the integration of knowledge from multiple domains, such as physics and human behavior. These domains may not be adequately represented in the training data of existing foundational models. To further explore this area, we introduce a new embodied task planning benchmark, ET-Plan-Bench, which features an automatic embodied task generation and evaluation pipeline that is designed to evaluate tasks with spatial and temporal understanding of the environment." 3) Furthermore, a discussion on failure cases has been introduced in the original paper: "The spatial relation constraints poses extra challenges. For example, if the task requires finding a glass near a monitor, the monitor might block the glass from the robot's view, preventing it from correctly verifying the spatial relationship. Detailed explanation can be found in Appendix A.2." 4) Due to page limitations, a detailed explanation of all failure modes can be found in Appendix A.2.

3. Comments: In the proposed modular approach, there are perception modules. It's unclear whether the robustness of these perception modules is being evaluated.

Response: We wish to further clarify this point with the reviewer. We employed the perception API in VirtualHome and GroundingDINO in Habitat. We evaluated the state-of-the-art open-vocabulary object detection model GroundingDINO on VirtualHome images, with further details provided in Appendix A.10.3. Our evaluation revealed that the accuracy of GroundingDINO was suboptimal in VirtualHome due to the inadequate image quality. Consequently, we used the VirtualHome Perception API function 'get_object_list,' which is based on first-person perspective images. While this approach is not flawless, it is acceptable given that we cannot alter the API and it represents the best available option. We also addressed this limitation in Section 4.3: "Some instances of failure are attributable to defects in the perception module, which is provided by the VirtualHome API." Overall, the perception module also influences the overall performance.

4. Comments: The authors should justify the distribution of the generated tasks shown in Figure 1. It's unclear whether the distribution is directly from the generation pipeline or deliberately achieved.

Response: The distribution depicted in Figure 1 is achieved incidentally rather than deliberately. Since the task can theoretically be generated infinitely, we utilized the generation pipeline shown in Figure 2 to produce a sufficient number of tasks for evaluation. After automatically filtering out unreasonable tasks, we fixed the remaining tasks for evaluation and recorded their number in Figure 1.

5. Comments: Since the generated tasks themselves could be wrong. Are the incorrect tasks filtered? I hope the authors can provide details on how the incorrect tasks are accounted for, especially when the methods perform similarly.

Response: Yes, incorrect tasks are filtered through a two-step process. First, we employ LLMs to identify and exclude unreasonable or unusual tasks. The prompt used for this filtering process is detailed in Table 9 of Appendix A.12. Examples of unreasonable tasks detected through this method include: "Please put the cellphone into the bowl and put bowl on the table" and "Locate the toothbrush into condiment shaker, then place them on the floor". The second step involves simulator verification. If the generated task includes at least one shortest path to locate the target object, and this path can be successfully executed in the simulator, the task is considered correct. Otherwise, it is deemed incorrect.

6. Comments: It would be great to see qualitative examples of both positive and negative examples.

Response: Due to the limitations on the main content page, we have provided positive examples in Appendix A.1 and negative examples in Appendix A.2.

7. Comments: Some definitions of the tasks can be elaborated on: 1) it's unclear what difference between regular navigation and navigation due to the distant initial position. 2) how is the optimal path defined? 3) since there are different types of temporal and causal constraints, what is the percentage of these types in the generated tasks?

Response: Following your suggestion, 1) We have added clearer information regarding the definition of navigation with occlusion due to distant initial position in Section 3.1 titled Benchmark Task Definition in the revision: "We designed tasks that target objects located far from the robot's initial position. The robot is then required to locate the target object over a greater distance." In addition, in Section 4.2 Evaluation Results and Main results in Virtual Home, we have discussed the navigation with occlusion due to distant initial position: "the tasks of which the distance between the robot's initial position and the target object position is top 20% largest" 2) For the navigation and manipulation of multiple objects, we provide two versions to complete this task: one utilizing a single arm, and the other utilizing two arms. In the revision, we have included clearer information regarding the definition of the optimal path: "We consider the two-arm version to be the optimal path." 3) Theoretically, an infinite number of tasks can be generated. However, we have provided the statistics for all tasks for evaluation in Figure 1.