LLM+A: Grounding Large Language Models in Physical World with Affordance Prompting

Dear Reviewer xvo1,

We greatly appreciate your insightful comments and the time you took to review our manuscript. Your feedback is essential to improving our work, and here we address your concerns:

1. [Limited Generalization]: The experiment of this study depends on the simplified tasks, such as pushing cubes and put cubes. These objects are rather simple with regular shapes. This is crucial to discretize the actions. However, the experiments may not cover all possible scenarios with more complex envs and irregular daily objects, leading to potential limitations in the model's applicability outside the tested conditions.

Thank you for your valuable suggestions. We agree with you that the experiment should include more complex tasks to verify the generalization ability of our method. Accordingly, we have incorporated additional testing scenarios. First, we diversified the shape of objects, introducing cubes, pentagons, star-shaped, and moon-shaped objects. We conducted an evaluation of our method over 100 episodes in the Block2Position task, resulting in a success rate of 38%. Notably, this rate is nearly consistent with the success rate reported in our paper. An example is shown in Figure 5. Second, we applied our method in complex scenarios, involving potential interferences with object positions. More specifically, we randomly change the object position at any given time step within an episode. An example of environmental observation and robot trajectories is provided in Figure 6 for your reference. The results demonstrate the robustness of our method, notably its ability to dynamically recalibrate motion sequences.

2. [Affordance Predictions Accuracy] The accuracy of affordance predictions from LLMs could be a potential weakness. If the affordance values are inaccurately predicted, it might lead to sub-optimal or erroneous robotic actions. Variability of landscapes, backgrounds, and language descriptions predicting affordances across different objects or environments could impact the overall performance.

We agree with your concern about the affordance prediction accuracy. Variability of landscapes, backgrounds, and language descriptions across different objects or environments could impact the over-performance. In our experiments, the task instruction has no pre-defined template, grammar, or vocabulary for each task. For instance, “move the red block to the top right of the board” can also be described as “push the red block to the upper right corner”. The experiment results show that LLM is robust to the variability of language descriptions.

Besides, the robustness to the variability of landscapes and backgrounds is mainly dependent on the capability of detection models, recent visual-language models can achieve pleasant detection/segmentation results with respect to text inputs, and the accuracy of the perception module is not the focus of our work.

Furthermore, even if the affordance prediction can make mistakes sometimes, as our method takes in the detection results as the planner’s input which composes the closed-loop control, it can greatly alleviate the planning prediction error in one round by consistently adjusting its planning trajectory.

Dear Reviewer qNCi,

We greatly appreciate your insightful comments and the time you took to review our manuscript. Your feedback is essential to improving our work, and here we address your concerns:

1. The paper title claims physical world, however, in the evaluation, it only considers simulation tasks on table top (2D). Physical world robotics interaction is much more complicated than simulation and will absolutely break the assumption of this paper. In my point of view, the technique proposed in this paper only applies to a very limited setup. Basically, given some 2D points (target positions) how to use robot arm (source positions) to reach it and generate some trajectories. In contrary, techniques such as code as policies is general and can extend to physical world.

Thank you for your comments. In [1], the simulated environment “Language-Table” consists of a simulated 6DoF robot xArm6 implemented in PyBullet equipped with a small cylindrical end effector. Third-person perspective RGB-only images from a simulated camera are used as visual input. Also, similar real-world experiments are tested using UFACTORY xArm6 robot arms. The authors have demonstrated that the policy performance in Language-Table was highly correlated with policy performance in the real world. In [2], the Pick&Place task is built based on the Ravens benchmark [3] set in PyBullet. Although the rendering software stack may not fully reflect the noise characteristics often found in real data, the simulated environment can roughly match the real-world setup. Despite the 2D environment, we can transfer our method into a 3D setup using the following pipeline: we first invoke open-vocab detector OWL-ViT [4] to obtain a bounding box, and feed it into Segment Anything [5] to obtain an object mask. Then the object point cloud can be reconstructed with the mask and the RGB-D observation.

[1] Lynch, Corey, et al. "Interactive language: Talking to robots in real time." IEEE Robotics and Automation Letters (2023).

[2] Shridhar, Mohit, Lucas Manuelli, and Dieter Fox. "Cliport: What and where pathways for robotic manipulation." Conference on Robot Learning. PMLR, 2022.

[3] Zeng, Andy, et al. "Transporter networks: Rearranging the visual world for robotic manipulation." Conference on Robot Learning. PMLR, 2021.

[4] Minderer, M., et al. "Simple open-vocabulary object detection with vision transformers”. arXiv 2022. arXiv preprint arXiv:2205.06230.

[5] Kirillov, Alexander, et al. "Segment anything." arXiv preprint arXiv:2304.02643 (2023).

2. The technique proposed by this paper may highly depend on the choice of LLM. Ablations w/ different LLMs is required to show generality.

Thank you for your valuable suggestion. Our method currently still depends on the powerful GPT4. GPT4 is prone to generate outputs as we instructed that can be executed in the code, which is necessary for the manipulator policy. However, open-source models like Llama2 or Vicuna often throw errors both syntactically and semantically even if we show exemplars and instruct the output format in the prompt. Maybe instruction finetuning can alleviate this issue, but this is not the focus of our paper.

3. In SayCan paper, there is an "open source environment" section. Looks like the tasks are similar. It will be interesting to see the comparison w/ SayCan

Thank you for your comments. The work SayCan aims to provide real-world grounding by means of pre-trained skills. In comparison, our work utilizes LLMs as both high-level task planners and low-level motion controllers without dependency on pre-trained skills. Therefore, it may be unfair to include SayCan as the baseline.

Is the policy able to re-try if the first trial fails?

Yes. We query the LLM to re-plan sub-tasks and motion sequences every five time steps to correct any possible fails. To approve the effectiveness of re-plan, we applied our method in complex scenarios, involving potential interferences with object positions. More specifically, we randomly change the object position at any given time step within an episode. An example of environmental observation and robot trajectories is provided in Figure 6 for your reference. The results demonstrate the robustness of our method, notably its ability to dynamically recalibrate motion sequences.

Dear Reviewer 4STn,

We greatly appreciate your insightful comments and the time you took to review our manuscript. Your feedback is essential to improving our work, and here we address your concerns:

Limited technical contribution: this work comes off as an empirical evaluation of certain style of prompt engineering for robotic manipulation. Other than the fact that some prompt worked for a small set of robotic manipulation tasks, I am not sure what I learnt from this paper.

In this work, we demonstrate that a pre-trained LLM (GPT4), equipped with only an off-the-shelf detection model, can guide a robot manipulator by outputting high-level sub-task plans and low-level sequences of end-effector waypoints. This is quite a challenging task. First, the proposed method does not depend on pre-trained skills, motion primitives, or trajectory optimizers. Second, the proposed prompting method is task-agnostic so can easily generalize to heterogeneous tasks. Third, since LLMs are not trained for grounded physical interaction, the proposed affordance prompting effectively improves the executability of both the sub-task plans and the control sequences. We appreciate that you denote that the number of robotic manipulation tasks is limited in the previous manuscript. In response, we plan to conduct more realistic robotic manipulation tasks in revision, such as “open the drawer”, “push the button”, etc.

Furthermore, the evaluation tasks are too simple. Unclear how affordance prompting would scale to more complex or more real world tasks, for instance manipulation in cluttered settings or situations with partial observability.

Thank you for your valuable suggestions and we agree with you that the experiment should include more complex tasks. Accordingly, we have incorporated additional testing scenarios. First, we diversified the shape of objects, introducing cubes, pentagons, star-shaped, and moon-shaped objects. We conducted an evaluation of our method over 100 episodes in the Block2Position task, resulting in a success rate of 38%. Notably, this rate is nearly consistent with the success rate reported in our paper. Besides, an example is shown in Figure 5. Second, we applied our method in complex scenarios, involving potential interferences with object positions. More specifically, we randomly change the object position at any given time step within an episode. An example of environmental observation and robot trajectories is provided in Figure 6 for your reference. The results demonstrate the robustness of our method, notably its ability to dynamically recalibrate motion sequences. Additionally, although partial observability may occur in real-world tasks, our method can address this issue through re-planning in the fixed interval.

Unclear how well LLM+A will do without a really strong llm such as GPT4. The use of GPT4 also makes it a computationally slow framework for online deployment. To that end, it would be good for the authors to report execution time/time to solve for their evaluation tasks. I encourage authors to also do real-world evaluation to really put the runtime in perspective. Lastly, I'd also like to see LLM+A performance with other opensource models like Llama2 or Vicuna.

Thank you for your valuable suggestion.As the currently most powerful LLM, GPT4 encompasses plentiful commonsense and excellent abilities of reasoning and instruction following. Additionally, GPT is prone to generate outputs as we instructed that can be executed in the code. However, open-source models like Llama2 or Vicuna often throw errors both syntactically and semantically even if we show examples and instruct the output format in the prompt. Maybe instruction finetuning can alleviate this issue, but this is not the focus of our paper.

To increase the time efficiency, we query GPT4 to in a fixed interval generate sub-task plans and motion sequences instead of re-planning new trajectory waypoints every time step. Specifically, the planned sequence will be updated only after the robot finishes five waypoints in the previous query round. The average query number of each episode in evaluation tasks can be referred to in the following table. We do not include the query time since it depends on different sources of OpenAI APIs.

Dear Reviewer WA1X,

We greatly appreciate your insightful comments and the time you took to review our manuscript. Your feedback is essential to improving our work, and here we address your concerns:

The description of the Motion Controller could be improved as it is different in the experiments.

In the motion controller, given the decomposed sub-tasks and the affordance values from the sub-task planner, the executable motion sequences for the robots are generated. This is consistent among different tasks in our experiments. We will clarify the related descriptions to avoid possible misunderstandings.

The work is relegating too much emphasis on affordance prompting, but this prompting is overengineered. For instance, Pick&place example does not show enough the advantage of affordance prompting.

We aim to explore the adaptation of the proposed affordance prompting technique. Therefore, we do not further engineer the prompting style for each experimental task, which is verified to be effective in Pick&Place task. To further show the robustness of our method, we have incorporated additional testing scenarios. First, we diversified the shape of objects, introducing cubes, pentagons, star-shaped, and moon-shaped objects. We conducted an evaluation of our method over 100 episodes in the Block2Position task, resulting in a success rate of 38%. Notably, this rate is nearly consistent with the success rate reported in our paper. Besides, an example is shown in Figure 5. Second, we applied our method in complex scenarios, involving potential interferences with object positions. More specifically, we randomly change the object position at any given time step within an episode. An example of environmental observation and robot trajectories is provided in Figure 6 for your reference. The results demonstrate the ability of our method to dynamically recalibrate motion sequences. Additionally, we plan to conduct more realistic robotic manipulation tasks in the final version, such as “open the drawer”, “push the button”, etc.

Results accuracy is very low comparing to other LLMs methods to solve planning.

Most of the previous planning-based LLM methods (such as “Language Table”) depend on either pre-trained low-level skills or massive multi-modal data to encompass diverse robotic tasks. In contrast, we utilize LLM as both the high-level sub-task planner and the low-level motion controller without the need for additional training process/data which is advantageous in many real-world applications (e.g., the scenario where the agent needs to fast adapt to heterogeneous tasks). We acknowledge that accuracy is a current shortcoming of our method, but we believe the performance of such methods will improve along with the improvement of prompt design and the base LLM.

The example used for explaining why current methods do not work: “It may move directly right and then push the block” is not enough demonstrated with SOTA algorithms and maybe too biased by comparing with ReAct. For instance, Driess et al. PALM-E and interactive language can solve these type of table top problems.

We would like to explain that the proposed method can be effective in many robotic manipulation tasks except for push blocks. The affordance prompting aims to infer goal-conditioned affordance values, which indicate the executable priorities of different parts of interacted objects. For example, when instructed to “pick the pot on the table”, the robot can be guided to hold the handle of the pot with affordance prompting. We plan to report more experimental results to demonstrate this advantage.

Previous works on affordances and tools: this could be improved.

Thank you for your valuable suggestions. We have included the recommended references in the current manuscript.

It is not totally clear for me, who is setting the object parts and how the affordance values are being generated. As this is totally different in the pushing and the pick&place experiments.

We manually choose the object parts for different kinds of tasks. For the pushing task, we distinguish the four edges of the bounding box as object parts. For the Pick&Place task, we detect the center of the bounding box as object parts. Then, we prompt the LLMs to directly generate affordance values ranging from 0 to 1 to represent the usefulness of each object part in the sub-task planner, and an example is shown in Listing 1 in the Appendix.