ThinkBot: Embodied Instruction Following with Thought Chain Reasoning

Thanks for your insightful review! We address your concerns as follows.

Q1. The improvements are marginal in valid unseen.

The ablated results of the proposed techniques indicate a significant drop from 67.72% to 64.92% (-2.80%) in success rate, which is a great performance gap observed in ALFRED. As a reference, the key idea of the semantic search policy in FILM paper [1] reports a smaller performance gain of +0.24% (from 19.85% to 20.09%). Besides, the ablated results of the instruction completer are significant in hard valid unseen split with an absolute degradation of 22.97%, which is actually more realistic as target objects are often tidied away and stored in receptacles in daily household tasks. Moreover, the problem of understanding sparse human instruction is essential in diverse multi-model embodied tasks including embodied instruction following, high-level planning, object navigation and even robotic manipulation. We have conducted extensive experiments in these domains to verify the generalizability of the proposed method (The results are shown in the responses to Reviewer oWwC-Q6 and Review pZ3K-Q2).

Q2. Difference between ThinkBot and LLM-Planner in high-level planning.

In terms of high-level planning accuracy, the difference between Thinkbot and LLM-Planner is mainly caused by two reasons:

• LLM-Planner directly mimics the retrieved few-shot examples. Besides, Thinkbot generates explicit thought chains to recover the sparse human instruction before predicting the next subgoal, which enhances high-level planning with structured reasoning.

• LLM-Planner only leverages a discrete observed object list, ignoring other information. On the contrary, Based on the explicit thought chain, Thinkbot processes the uncertainty information of the observed objects to refine the thought chain, so that the predicted subgoal can lead to efficient task completion.

For example:

Instruction: Examine a box of tissues using a floor lamp.
Thinkbot: Pickup TissueBox (Attempt) -> Open Cabinet -> Pickup TissueBox -> ToggleOn Lamp (HLPACC=100%)
LLM-Planner: Pickup TissueBox -> ToggleOn Lamp (HLPACC=0%)
Groundtruth: Open Cabinet -> Pickup TissueBox -> ToggleOn Lamp

LLM-Planner fails when the few-shot examples do not contain the case of a TissueBox stored in a cabinet. In contrast, Thinkbot can infer the intermediate 'Open Cabinet' by reasoning about the observed objects (Cabinet, Sofa, Safe) and the localizer's output (Cabinet with high uncertainty).

[1] FILM: Following instructions in language with modular methods, ICLR 2022.

Dear Reviewer,

We would like to ask if your concerns regarding the ablation study and high-level planning addressed, and if there is anything preventing you from increasing your score. Please let us know, and thank you for your time!

Thanks for your valuable feedback! We provide answers to your questions below.

Q1. Why not use step-by-step instructions directly.

The step-by-step instructions still does not align with the subgoal sequence and often misses some information like the target instance position. For example, 'Grab an egg out of the fridge.' ignores 'Close Fridge'. Moreover, our method supports goal statements only without low-level instructions, which is a more convenient and realistic way in household scenarios. To verify this, we conduct comparisons on different methods:

Thinkbot achieves the minimal performance gap when removing the low-level instructions, showing its superior reasoning ability given extremely sparse human instructions.

Q2. Better language models with prompting techniques for better results is expected.

Prompter uses BERT to identify the goal object, while utilizing GPT-Neo as the large language model to guess which receptacle a target object is likely to be in. We replace GPT-Neo with advanced GPT-4o in Prompter, and use advanced prompting techniques:

By incorporating the advanced language model and prompting techniques, Prompter improves from 64.43% to 64.92%, which is neglectable as the cooccurence probablities are basically the same as the vanilla version. Besides, the proposed framework complements the language model, please also refer to the response to Reviewer pZ3K-Q1, where we implement Thinkbot with different LLMs and observe consistent improvements.

Q3. Why not just use information in the semantic map.

The semantic map is often partially-observed, and the instance that lies in closed receptacle is not shown in the built semantic map. Therefore, existing methods [1,2,3] study different localizers to infer the possible position of the target. However, these methods perform planning and localization separately, which struggles with error interactions due to the open-loop nature. To this end, we introduce a graph-based multimodal object localizer based on the partially-observed semantic map and the rationales from the instruction completer, and outputs the Bayesian uncertainty for the instruction completer to conduct closed-loop planning.

Q4. Marginal ablated performance in valid unseen.

The ablation of the object localizer results in a noticeable performance drop in success rate (-0.73%). For comparison, the FILM paper [1] reports a smaller performance gain of +0.24% (from 19.85% to 20.09%) by incorporating it semantic search policy. The object localizer is designed to infer the positions of interacted objects in a partially-observed map, enabling efficient exploration. In long-horizon tasks of ALFRED, the environment gradually becomes close to fully observed as the task progresses. Nevertheless, the performance gaps are more pronounced in PLWGC (-0.87%) and PLWSR (-0.86%). Moreover, the ablated results of the object localizer are significant in hard valid unseen split with a relative degradation of 29.39%, which is actually more realistic as objects are often tidied away and stored in receptacles in daily household tasks.

Q5. Implement the proposed module in other baselines.

We incorporate Thinkbot with FILM and CPEM, and report the success rates in the leaderboard split test unseen in ALFRED:

Note that here we use the non-templated version of CPEM for re-planning. The consistent improvements across FILM, CPEM and Prompter indicate the generalizability of the proposed approach.

Q6. Single benchmark.

We have tested Thinkbot in ALFRED, ActioNet, and a real-world robotic manipulation task (please check pZ3K-Q2), and demonstrates significant improvements across various domains. We further validate it in a subset of HM3D ObjectNav benchmark [4], where we use 80 scenes for training and evaluate 20 scenes in the valid split:

By incorporating Thinkbot with the well-recognized baseline SemExp [5], we boost the performance by a large margin of $4.7$%, which indicates the generalizability of the proposed method.

[1] FILM: Following instructions in language with modular methods, ICLR 2022.

[2] Prompter: Utilizing Large Language Model Prompting for a Data Efficient Embodied Instruction Following, arXiv 2022.

[3] Following Natural Language Instructions for Household Tasks With Landmark Guided Search and Reinforced Pose Adjustment, RA-L 2022.

[4] Habitat-matterport 3d semantics dataset, CVPR 2023.

[5] Object Goal Navigation using Goal-Oriented Semantic Exploration, NeurIPS 2020.

Dear Reviewer,

We would like to ask if your concerns regarding the experimental setting and the generalizability of the proposed method addressed, and if there is anything preventing you from increasing your score. Please let us know, and thank you for your time!

Thanks for your review! Here, we respond to your comments.

Q1. Lack of novelty.

We respectfully argue that our work introduces novelty in addressing the challenges of enhancing LLMs with spatial reasoning capabilities for interactive agents, which is non-trivial. Naive combination of LLM with a localizer struggles to recover attainable subgoals in EIF, because it requires:

• Spatial Reasoning in Partially-observed Scene: Sparse human instructions often omit details of exploration in partially-observed environments. This requires the agent to infer missing information based on spatial clues, which is non-trivial for LLMs due to the lack of explicit guidance in both language and vision.
• Executability for low-level APIs: The embodied instruction following agents often utilize a low-level API (e.g., a pretrained detector) for interaction, which can fail in unstructured scenes (e.g., invalid mask from a weird direction), requiring precise uncertainty feedback to prevent execution errors.

To tackle these, we propose an instruction completer that guides the LLMs to reason intermediate subgoals to reduce uncertainty of the target in partially-observed scene, and introduce a Bayesian object localizer to estimate the uncertainty feedback precisely. To conclude, the proposed reasoning and feedback framework complements the internal capabilities of LLMs (please refer to pZ3K-Q1), and shows significant improvements in a wide range of embodied tasks, including ALFRED, ActioNet, Habitat and real-world robotic manipulation (please check oWwC-Q6 and pZ3K-Q2).

Q2. Try GPT-4o.

We input the instruction with current egocentric image to the most advanced language model GPT-4o:

Due to the high cost of planning each low-level action step with GPT-4o, we evaluate the compared methods in the main valid unseen split for comparisons. To avoid hallucination, we let GPT-4o choose a low-level action from 'MoveForward', 'RotateLeft' and 'RotateRight' for navigation, and select a target object name from the detected object list for interaction. We also provide the distance to each object in the viewing frustum from depth estimation, which can avoid interacting with a remote object. We sample $k=9$ successful demonstrations using $k$-NN from training set for in-context learning, where the pairwise similarity is estimated by a frozen BERT as per [1]. We further simplify the problem by letting MLLMs reason on a dilated online semantic map, addressing the issues of collision and spinning around due to the lack of proprioception and memory. However, both trials of directly using GPT-4o lead to relatively low performances, where the main error mode comes from the internal reasoning of GPT-4o can not achieve efficient exploration. On the contrary, Thinkbot leverages spatial uncertainty feedback for closed-loop planning, which is complementary to the internal reasoning ability of LLMs (MLLMs). As a result, by incorporating more cutting-edge LLMs, our method can consistently improve its performance, demonstrating the versatility of the proposed reasoning and feedback framework.

[1] LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models, ICCV 2023.

Dear Reviewer,

We would like to ask if your concerns regarding more comparisons with advanced MLLMs addressed, and if there is anything preventing you from increasing your score. Please let us know, and thank you for your time!

Thank you for the detailed and constructive review! We answer your comments below.

Q1. Different LLMs.

The proposed method complements the internal abilities of LLMs, which unlocks their spatial reasoning ability by incorporating the uncertainty predicted by the Bayesian object localizer as a feedback signal for closed-loop planning. We implement the instruction completer with a relatively small-size and open-sourced LLM Llama 3.2 3B in valid unseen:

Q2. Real-world applications.

The simulation benchmarks ALFRED and ActioNet used in our paper require a mobile manipulation agent, which indeed pose challenges for deployment. But Thinkbot is a general reasoning framework that can be easily extended to other environments with atomic skills. To verify its practicality, we test it in a constrained yet realistic language-conditioned robotic manipulation task pick up the blue cube. The atomic skills used are move(position) and gripper(openness). For simplicity, the object localizer is implemented using a pretrained Grounded-SAM [1], with its confidence score serving as the uncertainty measure. In each test episode, the object locations are randomized, and distractors (e.g., blocks of other colors) are added to evaluate the generalizability:

Thinkbot enhances programmatic generation by recovering dense and actionable instructions like slightly adjusting the gripper pose for better grasping based on the visual feedback, leading to a sizable improvement. We have updated the real-world execution videos in the supplementary material.

Q3. Inference time.

We benchmark the time consumption in a single NVIDIA RTX 4090 GPU. In line with [3], we query the LLM after every 25 steps or subgoal completion to ensure consistency. Hence, the delay of LLM reasoning only affects steps that require important subgoal decisions (slow thinking), while most steps just involve path planning to achieve the subgoal (fast thinking), resulting in an acceptable average time per step:

We also provide the time analysis of each component in the whole system:

In summary, the whole system executes at 1.37Hz on average. The INT4 version of Llama 3.2 3B only requires 1.75 GPU memory, which can be easily deployed in consumer-size GPUs.

Q4. Error recovery and self-improvement.

Figure 6 in the manuscript shows when the agent mistakenly opens the wrong cabinet instance, our instruction completer outputs 'close cabinet, go to another cabinet, and open cabinet' to recover from the error interactions. To validate the error recovery of Thinkbot, we add noise to the low-level actions: with 20% probability, the agent randomly chooses 'RotateRight' or 'RotateLeft'. 'MoveAhead' and interaction actions are not randomized, as their errors count and don't change the environment:

The slight performance drop shows the robustness of the proposed agent.

In terms of self-improvement, the current method we evaluate on ALFRED and ActioNet is static and re-intialized at the beginning of each episode. However, we can easily combine the proposed method with Voyager [4] by storing successful subgoal trajectories in a ever-growing knowledge base for few-shot learning. Besides, the densified instruction can also be used to finetune the instruction completer, which formulates a self-improvement loop like STaR [5].

Q5. Generalization of the Reasoning Process.

Our prompts specify the general reasoning and feedback format, and there is no information leakage or manual adjustment. To address the generalization of the reasoning process to new scenarios or tasks, we can just replace the atomic skills in the prompt, then the agent can complete the tasks autonomously. The superior performances across diverse test suites including ALFRED, ActioNet, Habitat (view oWwC-Q6) and real-world robotic manipulation verify the generalizability of the proposed method.

[1] Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks, arXiv 2024.

[2] ProgPrompt: Generating Situated Robot Task Plans using Large Language Models, ICRA 2023.

[3] FILM: Following instructions in language with modular methods, ICLR 2022.

[4] Voyager: An Open-Ended Embodied Agent with Large Language Models, TMLR 2023.

[5] STaR: Bootstrapping Reasoning With Reasoning, NeurIPS 2022.

Dear Reviewer,

We would like to ask if your concerns regarding more generalization studies addressed, and if there is anything preventing you from increasing your score. Please let us know, and thank you for your time!

Thanks for your interest in our work. We sincerely appreciate your pioneering work in incorporating affordance-aware semantic map into the field of embodied instruction following, and include it in the revision (L101 in page 2). We are welcome to further discussion!