Asking Before Acting: Gather Information in Embodied Decision-Making with Language Models

Thanks for your time and efforts in reviewing our paper! We highly appreciate your thoughtful and constructive suggestions. Your thoughtful and constructive suggestions have been invaluable to us, and we have carefully considered each comment. Our responses to your queries are outlined below:

---

Q1. why not use RLHF?

In summary, we use human models during evaluation, to enable real time human interaction and save human efforts, while RLHF is a training method to learn human values. To be specific:

• In ABA, we focus on the human (external information source) in the loop setting, where the agent can interact with humans in real time during deployment.

  While typical RLHF methods like [1] collect human feedback dataset and train the model. During rollout, RLHF cannot help model the interactions with humans.

• In ABA, for the same question, the human may provide different answers in different scenarios. For instance, asking "where is the cup?" may yield different responses in different rooms

  While in RLHF, the learning objective is the concept of values / preferences, which is fixed, and cannot fit in our setting.

While we have not incorporated RLHF in the current work, we acknowledge the potential for future research to explore combining RLHF to enhance the asking quality of ABA.

[1] Ouyang, Long, et al. "Training language models to follow instructions with human feedback." Advances in Neural Information Processing Systems 35 (2022): 27730-27744.

---

Q2,4. navigation and missing related works

Thanks for pointing out these related works! We have added them to the related work section and highlighted the modifications in blue.

---

Q3. writing issues

Thanks! We have fixed the typos in our manuscript and these modifications are highlighted in blue for your convenience.

---

We sincerely appreciate your thorough review and constructive suggestions. If you have any further comments or inquiries, please do not hesitate to let us know. And we genuinely hope for your reconsideration of the score.

Dear Reviewer, we would like to thank you again for your efforts and time in providing thoughtful feedback and comments. We’ve revised the paper according to your suggestions and replied to all the questions and concerns. Since the discussion period is ending soon, we would greatly appreciate it if you could let us know whether you have any additional comments.

Thanks a lot!

Thanks for your time and efforts in reviewing our paper! We highly appreciate your thoughtful and constructive suggestions. We have carefully considered each comment. Our responses to your queries are outlined below:

---

Q1, Q2 and Q4. insufficient experiments, new baselines and benchmarks

New baselines:

We've carefully considered your recommendations, and here's our feedback:

1. Reflexion [1] and Adaplanner [2]:

   These methods focus on learning from previous mistakes through multiple rollouts, while ABA aims to perform tasks within a single shot. Therefore we are orthogonal and focus on different settings. We acknowledge the potential for future research to explore combinations with [1,2] to enhance performance.

2. CLIPort [3]

   We believe [3] is not suitable here since:

   • Different settings. Both ABA and ReAct are few-shot methods, while [3] needs to collect an expert dataset to do imitation learning.

   • In section 4.2, our tasks involve the agent asking questions or reasoning to gather information, a scenario not directly adaptable to imitation learning-based methods like [3].

3. BUTLER in ALFWorld variants

   We have already compared BUTLER in the original ALFWorld. We show ABA which use $K=2$ trajectories can significantly outperform BUTLER with $10^5$ training trajectories. For similar but more challenging environments, we believe it is not necessary to compare with it again.

New benchmarks:

Regarding new environments, our experiments cover three types, including text-based tasks, robot arm manipulation, and real-world embodied environments, totaling 29 different tasks. As acknowledged by other reviewers, our experiments are “thorough” (from P7Hr), “sufficient” (from 23Y9), and “comprehensive” (from umUa) experiments. If there are specific aspects or types of experiments you believe are lacking or could be supplemented, we would greatly appreciate further clarification and any suggested benchmarks.

---

Q3. There is no ablation showing that the external source is providing guidance and not just the answer.

In Section 3.3 and Appendix A, we showed how the human model is designed. It prompts the LLM with environmental information and is prompted to answer the questions. The human model does not have access to the oracle action, and it will not provide the answer.

---

Q5. What is the expert policy used to train ABA-FT?

As outlined in Appendix E, we designed a rule-based policy according to the PDDL planning trajectories provided along with the environment to collect the data.

---

Q6. Is there any visual model used for the robotic control experiments?

As outlined in appendix J, we use a hand designed captioning module, which can access the simulator information and label the object type and its ground truth coordinates. As discussed in Huang et al. (2023), this module can be implemented through the depth image and a combination of object detection model, an object segmentation model, and a video tracker. But we temporarily leave this module hand-designed since it is not the key point of our work.

---

We appreciate your thorough evaluation and remain open to further discussion or clarification on any points. And we genuinely hope for your reconsideration of the score.

Dear Reviewer, we would like to thank you again for your efforts and time in providing thoughtful feedback and comments. We’ve revised the paper according to your suggestions and replied to all the questions and concerns. Since the discussion period is ending soon, we would greatly appreciate it if you could let us know whether you have any additional comments.

Thanks a lot!

Thanks for your feedback.

---

Q1. Whether [1,2] should be used as baselines?

We would like to begin by summarizing the overall discussion.

• In the reviewer’s original review, mCAD asks for more baselines like [1-2].

• In our original response, we explained these methods are not suitable for baselines since they have different settings. [1,2] allow rollout in the same environment multiple times, whenever they fail, they can restart the environment and learn from previous mistakes. In contrast, we focus on solving the task within 1 trial. Therefore, it is not proper to compare with [1,2] directly.

• In the reviewer’s following feedback, mCAD agrees [1,2] and our method are different formally, but mCAD suggests they are similar in practice.

• In our following response, we explained that it does not make sense to claim these few-shot methods incorporating environment restarting equals one-shot methods. We cite [1] to illustrate that with only 1 shot, [1] degenerate to ReAct, which is a baseline cited in our paper. So, [1,2] are not similar to our method in practice.

• In the previous feedback, mCAD acknowledged that our method is zero-shot, while [1,2] are not, but mCAD proposed new similarities between our methods and [1,2]: (1) both methods need multiple interactions to give an answer, (2) both methods use prompting.

This time, our response is as follows:

First, we believe it is way enough to prove [1,2] cannot be directly used as baselines according to previous rebuttal.

As for the new points raised by the reviewer, we believe mCAD might have misread our paper: reviewer mCAD seems to confuse the following three concepts (a) multiple actions are taken in 1 trajectory; (b) multiple offline trajectories used for In-Context-Learning as in ABA; (c) [1,2] include online trajectories as prompting.

• In the reviewer’s feedback, mCAD claims we need to do several asking actions in 1 trajectory (type (a)), but mCAD cited the ICL part (type(b)) to support this claim.

• mCAD suggested that we use ICL prompt (type (b)), while [1,2] also use prompting (which is type(c), but mCAD mistaking it for type(b)).

Besides this misreading, it is quite weird to ask someone to compare two methods simply because both of these methods use prompting regardless of other settings.

---

Q2. Whether the human model can output ground truth action?

We thank the reviewer for acknowledging that the previous criticism does not exist in Section 4.2. Now let’s look at Section 4.1, ALFWorld, to solve the reviewer’s concern. To finish the task, the agent needs to find the object, pick it up, wash it / heat it / cool it, or even open a desk lamp to check it, before placing it to the target location. As outlined in Appendix A, the human model only knows the placement of all the objects. In this paper, we propose that, instead of finding the object by onerous trial and error, it is more efficient to query in natural language. Therefore, in the first step “finding the object”, the information may seem like the action, but this is exactly what the agent learned to ask. Furthermore, we would like to remind the reviewer that, for all the following steps, the human model does not know what task is assigned to the agent, it does not know what to do with the target object, it does not know the target location, so it can not produce the ground truth action.

---

Q3. Should we address hallucination in our paper?

We thank the reviewer for reading the related paragraph. However, as you may agree, the human model does not belong to the ABA method. It is just a proxy of real humans during evaluation, so as to evaluate ABA with minimal human involvement, which is not the key component of this paper. What's more, this result can be easily improved by using larger and stronger LLM. The Appendix C shows Vicuna 7B human model results, which only shows up in a small fraction of experiments in Section 4.1, while other results in Section 4.1 and Section 4.2 use GPT-35, GPT-4 as human model.

The question about hallucination is raised out of nowhere. We do not even mention the word “hallucination” in any related analysis or in our main paper. Although LLMs are well known for suffering from hallucination, how to solve this problem, however, is out of the scope of this paper.

Thanks for your time and efforts in reviewing our paper! We highly appreciate your thoughtful and constructive suggestions. Your thoughtful and constructive suggestions have been invaluable to us, and we have carefully considered each comment. Our responses to your queries are outlined below:

---

Q1. about the setting and contribution: the "asking before action" paradigm

After reading the review, we would like to clarify our setting in case of any potential misunderstandings:

Currently, LLM based agents are designed to solve tasks on their own without external interactions, which is identified to be inefficient in scenarios with limited information. To address this concern, we concentrate on a special type of setting with human / external information sources in the loop (refer to Def 3.1), and propose the “asking before action” paradigm. This distinguishes our work from purely autonomous agents.

We believe human / external information in the loop is important and promising [1-6]. We are not primarily concerned with completing complex tasks through trial-and-error but with facilitating decision-making through interaction with humans.

We take a significant step forward in the Human-in-the-Loop (HITL) setting. Previous works including [1-6] ask for oracle action, template questions, etc. ABA proposes gathering necessary information in natural language. This contributes to a novel setting, a set of newly designed benchmarks, and helps extend the boundary of current research.

Looking ahead, we foresee combining ABA with recent advances in purely autonomous works, such as [7] for navigation and [8] for pick-and-place tasks, to build more intelligent embodied agents.

To summary, our contributions are:

1. identify a novel setting (~Def 3.1) that can possibly contribute to building a more embodied AI in the real world.

2. design new methods which can mimic human behavior, and "representing a shift towards more intuitive, adaptive AI systems".

3. a set of benchmarks with a set of human model designs that can facilitate later research in this direction.

---

Q2. about the scenarios: Why Not Provide Complete Information at the Beginning/Feed to Other Baselines?

We appreciate your question about complete information. We believe providing complete information is impractical for real-world tasks due to scale differences between simulators and reality.

While simulators may handle scenarios with dozens of objects, the real world poses challenges in enumerating all relevant information. To illustrate, consider seeking assistance in preparing lunch—an exhaustive list of utensils and ingredients would require significant human effort and might overlook unforeseen circumstances. Therefore, for real-world scenarios, providing complete information is not viable, and we have not supplied it to other baselines.

Q3. Why use Contextual MDP rather than POMDP:

Although Contextual MDP is a type of POMDP, we opted for the former because:

• Contextual MDP provides a latent context $c$ which helps to model the task information, while POMDP does not.

• $c$ helps better formulation: we can ask pertinent questions to infer the context $c$, the human model inside the environment $H$ will provide different answers to the same questions according to the different $c$. while POMDP can not.

• Contextual MDPs are well defined and widely accepted, especially in RL generalization theory, multi-task, and meta RL domains. Please refer to [9] for a thorough survey.

---

Q4. Error analysis on ABA:

We provided error analysis of a typical failure case in Appendix F, and we appended several new failure cases at the end of Appendix F (modifications are highlighted in blue). The analysis can be summarized as follows:

Two key failure types are identified:

• undesirable answers from human models during evaluation

  Note we use human models to mimic humans during evaluation to minimize human involvement, but this may sometimes provide undesirable answers.

• mismatches between agent commonsense and the environment

  For instance, the task is to collect eggs from the garbage can, but the agent believes the eggs are no longer suitable for eating and proposes to buy new ones.

We believe incorporating real humans during deployment and refining prompting can mitigate these failures, leaving room for future research. For detailed information, please refer to Appendix F.

---

Once again, we sincerely appreciate your feedback, and we hope our responses address any concerns you may have. Your insights are crucial to the improvement of our work, and we welcome any additional feedback, and we genuinely hope for your reconsideration of the score.

---

[1] Khanh Nguyen and Hal Daume ́ III. Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning. arXiv preprint arXiv:1909.01871, 2019.

[2] Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12527–12537, 2019.

[3] Kunal Pratap Singh, Luca Weihs, Alvaro Herrasti, Jonghyun Choi, Aniruddha Kembhavi, and Roozbeh Mottaghi. Ask4help: Learning to leverage an expert for embodied tasks. Advances in Neural Information Processing Systems, 35:16221–16232, 2022b.

[4] Felipe Leno Da Silva, Pablo Hernandez-Leal, Bilal Kartal, and Matthew E Taylor. Uncertainty- aware action advising for deep reinforcement learning agents. In Proceedings of the AAAI con- ference on artificial intelligence, volume 34, pp. 5792–5799, 2020.

[5] Iou-Jen Liu, Xingdi Yuan, Marc-Alexandre Coˆte ́, Pierre-Yves Oudeyer, and Alexander Schwing. Asking for knowledge (afk): Training rl agents to query external knowledge using language. In International Conference on Machine Learning, pp. 14073–14093. PMLR, 2022.

[6] Wenlong Huang, Fei Xia, Ted Xiao, Harris Chan, Jacky Liang, Pete Florence, Andy Zeng, Jonathan Tompson, Igor Mordatch, Yevgen Chebotar, et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022b.

[7] Shah, Dhruv, Błażej Osiński, and Sergey Levine. "Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action." Conference on Robot Learning. PMLR, 2023.

[8] Huang, Wenlong, et al. "Voxposer: Composable 3d value maps for robotic manipulation with language models." arXiv preprint arXiv:2307.05973 (2023).

[9] Kirk, Robert, et al. "A survey of generalisation in deep reinforcement learning." arXiv e-prints (2021): arXiv-2111.

Dear Reviewer, we would like to thank you again for your efforts and time in providing thoughtful feedback and comments. We’ve revised the paper according to your suggestions and replied to all the questions and concerns. Since the discussion period is ending soon, we would greatly appreciate it if you could let us know whether you have any additional comments.

Thanks a lot!

Thanks for your time and efforts in reviewing our paper! We highly appreciate your thoughtful and constructive suggestions. Your thoughtful and constructive suggestions have been invaluable to us, and we have carefully considered each comment. Our responses to your queries are outlined below:

---

Q1. balance the performance and the efficiency

We appreciate your observation regarding the balance between performance and efficiency. In ABA, our key design focuses on prompting the agent to ask questions only when necessary, achieved through the injection of a proper prior. In the ABA-FT variant, we further enhance this efficiency by treating known information as a type of metadata, aiding the decision-making process on whether to ask questions.

To demonstrate the efficiency of these designs, we conducted an ablation study by counting the total number of questions asked in V7B ALFWorld successful trials, and the results are as follows (which is also added to Appendix L):

The results indicate that the average number of questions closely aligns with the minimum necessary questions. This study substantiates our claim that ABA effectively minimizes the number of questions asked. We will leave minimizing the number of questions to our future work.

---

Q2. Fix the typos

Thanks! We have promptly rectified these errors in our manuscript, and the modifications have been highlighted in blue for your convenience.

---

Q3. Missing related works

We are grateful for your recommendation of additional relevant works. We have incorporated these insightful contributions into our related works section, alongside other vision-language navigation works like TD-STP, SEvol which we believe to be related to the embodied decision making setting. The modifications have been highlighted in blue.

---

We sincerely appreciate your thorough review and constructive suggestions. If you have any further comments or inquiries, please do not hesitate to let us know. And we genuinely hope for your reconsideration of the score.

Dear Reviewer, we would like to thank you again for your efforts and time in providing thoughtful feedback and comments. We’ve revised the paper according to your suggestions and replied to all the questions and concerns. Since the discussion period is ending soon, we would greatly appreciate it if you could let us know whether you have any additional comments.

Thanks a lot!