Creative Robot Tool Use with Large Language Models

Q3: You should give the same set of attributes to all objects in the tasks:

• Thanks for the suggestion. We have cleaned up all the task descriptions. For robot arm experiments, the task description contains (1) graspable points for non-rectangular objects and the position of rectangular objects and (2) bounding box sizes of each object. For quadrupedal robot experiments, the task description contains each object's center position and size. Since tasks in quadrupedal robot experiments are loco-manipulation tasks, we also include the weight of each object in the scene.
  • Imbalance description in Milk-Reaching; Inaccurate cube weight of Cube-Lifting. Thanks for the insightful question. We have cleaned up the object description and changed the cube weight to the actual value.
• After rerunning the experiments, we noticed that RoboTool has a slight performance drop (0.87 -> 0.83), but it is insignificant.

Q4 & Q5: How to make sure the description describes the scene without heavy human design? Does the description hint at the final plan?

• We use human-designed templates due to the limitation of existing Vision Language Models. As mentioned in the response to Q1.3, the SOTA VLM models are insufficient in 3D reasoning, and it is hard to get a reasonable estimate of object positions and sizes directly without the help of existing APIs. Moreover, our tasks require information about the constraints and weights, which are hard to get from VLMs. Hence, to explore the complex creative tool-use tasks with existing foundation models, we have no choice but to provide a template description.
• Based on the reviewer's suggestion in Q4, we have cleaned up the description and made sure that different objects in the scene are provided with similar amounts of information, such as positions, sizes, and weights. Please note that the provided information is sufficient but may be unnecessary for finishing the tasks. In other words, we have provided distractors, such as other objects in tool-selection tasks, which require LLMs to filter the information and reason based on the most important information.
• Even with the same amount of information, the state-of-the-art baselines still have low success rates, as shown in Table 2. RoboTool performs better than baselines by a large margin, highlighting the task complexity of creative tool-use problems.

Table 2: Success rates of RoboTool and additional baselines.

Thank you for the detailed and insightful reviews! We appreciate that you found leveraging LLMs for creative tool use is a compelling approach, our method is effective, and the benchmark is well-designed. Here are our responses to your concerns:

Q1: Limited task diversity.

We want to highlight that designing six creative tool-use tasks and their variants to show the discriminative tool-use behaviors contributes to the robotics community, as discussed in [1]. Unlike many robotic manipulation tasks inspired by household activities, creative tool use is more like solving intricate physical puzzles with many design thoughts injected into the process. RoboTool is the first step towards designing a standard robot tool use benchmark. We hope our research could inspire more work on this critical yet underexplored topic.

Ref:

[1] Qin, Meiying, Jake Brawer, and Brian Scassellati. 2022. “Robot Tool Use: A Survey.” Frontiers in Robotics and AI 9: 1009488.

Q1.1: Are the object layouts changed?

Thanks for the question. The positions are indeed randomized. For robot arm experiments, we randomize the initial configuration of objects. We randomize the initial position of movable objects and the robots for quadrupedal robot experiments. We show the randomization ranges in Table 1 and add it to Appendix G in the updated draft.

Table 1: Randomization of objects in each task.

Q1.2: In milk-reaching experiment, the number of objects in the descriptions is limited. It may not be hard for the LLM to pick a related object.

• We increase the number of objects that may serve as tools to five, including a hammer, a pineapple toy, a lock, a tomato toy, and a cube. We also provide the same amount of information for each object, including (1) graspable points for non-rectangular objects and the position of rectangular objects and (2) bounding box sizes of each object. Considering the non-rectangular shape of the hammer and pineapple toy, we added a description related to the orientation. However, the orientation description does not affect the tool selection in experiments. After rerunning experiments, we noticed that RoboTool can still correctly select the hammer as the tool among ten runs.
• Besides selecting the correct tool, how and when to use the tool are also important. For example, the Can-Grasping task has three objects, five skills (2 have a discrete action parameter, 1 has continuous action parameters), and 15 planning steps. The underlying complexity of planning is significant, and any errors in the intermediate steps could lead to a catastrophic failure. Despite the difficulties, RoboTool achieved a high success rate across different domains, indicating it is not just randomly picking an object to interact with.

Thank you for the constructive feedback and suggestions! As the discussion period is ending, we would appreciate you kindly checking our response. Please do not hesitate to contact us if there are other clarifications we can offer. Thanks!

Dear reviewer xccj,

With the rebuttal period ending tomorrow, we would like to ask whether our response addressed your questions and alleviated your concerns. If so, could you please kindly consider raising the score?

Best,

Authors

Thank you for the valuable feedback. We appreciate that you found leveraging LLMs for creative tool use is a highly desirable research direction and well-explored in this paper. Here is our response regarding concerns about novelty, baselines, and benchmarks.

Novelty of the paper.

We respectfully disagree with the reviewer about the novelty of RoboTool. Here is a summary of our contributions:

• Novel Problem: Creative tool use is an important yet under-explored problem in robotics. It will greatly expand the capability of robots to solve tasks that are impossible originally.
• Novel Benchmark: We designed a novel challenging benchmark to test specifically the creative tool use behavior, which contains two robot embodiments and three categories of creative tool use. Such a benchmark is highly needed and non-trivial to design [1].
• Novel Method: Although the planner-coder LLM agent has been explored in existing literature, blindly using existing API-based methods such as Code as Policy fails to address the challenges posed by creative tool use in terms of cognition, reasoning, and planning. As shown below, the suggested baselines failed to complete any of the tasks in our benchmark with a high success rate.

Ref:

[1] Qin, Meiying, Jake Brawer, and Brian Scassellati. 2022. “Robot Tool Use: A Survey.” Frontiers in Robotics and AI 9: 1009488.

ICLR is focused on representation learning, while RoboTool is a training-free method.

• Despite the name, ICLR is about more than just learning representation, as documented by the non-comprehensive topic list on the official website. This year in ICLR, there are at least 90 submissions related to training-free LLM Agents. When looking at the accepted papers of this year NeurIPS, one can find at least 20 impactful training-free API-based LLM agents, such as HuggingGPT [2], Reflexion [3], DEPS [4], AVIS [5], CAAFE [6], and SheetCopilot [7]. With increasingly capable LLMs, it is critical to explore what they can or cannot do, which is underexplored especially in the physical interaction and 3D reasoning domains. RoboTool is one of the first steps towards reasoning implicit environment- and embodiment-constraints and long-horizon hybrid discrete-continuous planning. Therefore, RoboTool is well-suited to the top AI publication venues.

Ref:

[2] Shen, Yongliang, et al. "Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface." NeurIPS 2023.

[3] Shinn, Noah, Beck Labash, and Ashwin Gopinath. "Reflexion: an autonomous agent with dynamic memory and self-reflection." NeurIPS 2023.

[4] Wang, Zihao, et al. "Describe, explain, plan and select: Interactive planning with large language models enables open-world multi-task agents." NeurIPS 2023.

[5] Hu, Ziniu, et al. "AVIS: Autonomous Visual Information Seeking with Large Language Models." NeurIPS 2023.

[6] Hollmann, Noah, Samuel Müller, and Frank Hutter. "GPT for Semi-Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering." NeurIPS 2023.

[7] Li, Hongxin, et al. "SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models." arXiv preprint arXiv:2305.19308 (2023).

Thank you for the feedback! We appreciate you kindly checking our response as the discussion period ends soon. Specifically, we have added new baselines and showed that our proposed RoboTool outperforms the suggested baselines by a large margin. If you have further questions, please do not hesitate to let us know, and we are happy to answer them. Thanks!

Dear reviewer b6vg,

With the rebuttal period ending tomorrow, we would like to ask whether our response addressed your questions and alleviated your concerns. If so, could you please kindly consider raising the score?

Best,

Authors

Thank you for the encouraging review! We are glad that you appreciate the thoroughness and depth of our work. Here are our responses:

Focus of "long-horizon hybrid discrete-continuous planning".

Thanks for the question. We explain the hybrid task-motion planning problem here in more detail. An important part of TAMP problems is the interdependence of the motion-level and task-level aspects of the problem. Ignoring such interdependence between them will make it unable to solve the problem. We borrow this example from the seminal TAMP survey [1]:

• Consider a task where the robot needs to place a particular object (name $A$) at a location. It might select a high-level plan "skeleton" such as "moveF$(q_0, \tau_1, q_1, p_0)$, pick[$A$]$(q_1, p_0, g)$, moveH[$A$]$(g, q_1, \tau_2, q_2)$, place[$A$]$(q_2, p_1, g)$", where
  • moveF action involves robot movement when the gripper has no object with it.
  • moveH[$A$] action involves robot movement when the gripper is holding object $A$.
• The action primitives are discrete options. However, the free parameters are continuous values, such as robot configurations $(q_0, q_1, q_2)$, a grasp pose $(g)$, placement poses $(p_0, p_1)$, paths $(\tau_1, \tau_2)$. The skeleton imposes constraints on the choices of those values, or conversely, there could be no satisfying set of values, and the skeleton needs to be changed. For example, if there is object $B$ occupying the target location, the robot needs to first remove $B$ and then place $A$ at the target location.

This example showcased the importance of hybrid discrete-continuous planning. We will modify the problem formulation accordingly in the final version to highlight this.

Ref:

[1] Garrett, Caelan Reed, et al. "Integrated task and motion planning." Annual review of control, robotics, and autonomous systems 4 (2021): 265-293.

Analysis of the combinatorial nature of the parametrized skills.

• For the robotic arm, we have a skill library that includes[get_position, get_size, open_gripper, close_gripper, move_to_position. Among them, get_position and get_size take in discrete object_name as input, while move_to_position takes in 3D position of the target gripper as input. The longest plan (Can-Grasping) requires about 15 steps. A more detailed description of the parameterized skills can be found in Appendix B in the original draft.

State of the Art

• Thanks to the reviewer for pointing out these two papers. We added them into the additional related work in the appendix and will incorperate them to the final version of related work.

Experiment results

• Compare with a PDDL algorithm: Although we formulated creative tool use as a planning problem, solving such tasks requires more than just geometric planning. PDDL and other formal-logic planners, for example [2], are able to solve the Can-Grasping task, which requires only geometric planning. However, they cannot solve some other tasks in this work, for example, Cube-Lifting, which requires identifying a hidden lever structure in the scene and finding a way to activate this mechanism. The traditional planners are not able to reason based on common knowledge or manufacture nonexistent tools, which are critical to solving other creative tool-use tasks.
• Randomization: For robot arm experiments, we randomize the initial configuration of objects. For quadrupedal robot experiments, we randomize the initial position of movable objects and the robots. We show the randomization range in Table 1 and add it to Appendix G in the updated draft.

Table 1: Randomization of objects in each task.

Ref:

[2] Toussaint, Marc, Kelsey Allen, Kevin Smith, and Joshua Tenenbaum. 2018. “Differentiable Physics and Stable Modes for Tool-Use and Manipulation Planning.” In Robotics: Science and Systems XIV. Robotics: Science and Systems Foundation. https://doi.org/10.15607/rss.2018.xiv.044.

Thank you for your constructive feedback. We are pleased to hear that you found our experiments impressive and that the modules in RoboTool contribute to performance improvements. Please find our detailed responses below.

The prompts contain hints on what not to do when solving the problem.

• We removed the prompt related to "offset=1.0" and "careful when calculating negative values." After rerunning the experiments, we found some decrease in RoboTool's success rate (0.87 -> 0.83); however, it was insignificant. We updated Table 1 in the draft accordingly.
• The prompts related to the negative values and distances aim to remedy LLMs' deficiency, which hardly hints at how to plan for a specific task. Please note that we use the same prompts for all experiments, not specifically engineered for each task. Designing prompts containing general rules to follow is common in many embodied LLMs literature.
  • Regarding the prompt mentioning that the distance between objects' centers and boundaries are different, it is related to the 3D reasoning capability of gpt-4: It sometimes mistakenly thinks that the distance between objects is the distance between their centers without considering the size of the objects. We want to reduce such mistakes by reinforcing the distance concept in 3D world. However, RoboTool decides whether to use the distance between centers or the distance between boundaries for downstream planning.

Is the gripping offset in Fig. 2 code for the hammer engineered based on the shape of the hammer?

• The numbers are not engineered based on the shape of the hammer. The calculator prompts contain an example of calculating a generic object's grasping position. In other generated codes, RoboTool follows the examples in the prompt and automatically gets the grasping offset based on the object's bounding box size. We provide more generated codes in Appendix I.
• We need such an example in the prompt since RoboTool uses general motion primitives, such as move_to_position(pos) and close_gripper() to provide extra planning flexibility, instead of more granular primitives, such as pick(object) and place(object) in many existing methods [1, 2, 3, 4]. RoboTool needs to calculate an accurate gripper target position, which is difficult to rely solely on the prior knowledge of LLMs alone.

Ref:

[1] Singh, Ishika, et al. "Progprompt: Generating situated robot task plans using large language models." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.

[2] Ahn, Michael, et al. "Do as i can, not as i say: Grounding language in robotic affordances." arXiv preprint arXiv:2204.01691, 2022.

[3] Huang, Wenlong, et al. "Grounded decoding: Guiding text generation with grounded models for robot control." arXiv preprint arXiv:2303.00855, 2023c.

[4] Liang, Jacky, et al. "Code as policies: Language model programs for embodied control." 2023 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2023.

The engineering effort put in motion primitives.

• Thanks for the question. Our push_to_position() skill is based on Unitree's built-in walking mode gait with a small gait height and a motion planner to generate waypoints to move objects around (as shown in Appendix B.1 in the original draft). There is surging interest in obtaining better loco-manipulation skills [6-8], which is manipulating objects with a quadrupedal robot. We provide one simple implementation of such loco-manipulation skill, which is already sufficient to perform complex long-horizon tasks. Although obtaining such motion primitive is not the focus of this work, we are eager to integrate more diverse skills in RoboTool.
• Instead, in this work, we focus on high-level task planning and assume access to a library of existing motion primitives, similar to SayCan [2], GroundedDecoding [3], PPDL with LLMs [5] etc. Please note that our method is general and is compatible with skills derived from different methods, such as classical planning, RL, or imitation learning.

Ref:

[5] Silver, Tom, et al. "Generalized planning in pddl domains with pretrained large language models." arXiv preprint arXiv:2305.11014, 2023.

[6] Nachum, Ofir, et al. "Multi-agent manipulation via locomotion using hierarchical sim2real." arXiv preprint arXiv:1908.05224 (2019).

[7] A. Rigo, Y. Chen, S. K. Gupta, and Q. Nguyen, “Contact optimization for non-prehensile loco-manipulation via hierarchical model predictive control,” arXiv preprint arXiv:2210.03442, 2022.

[8] M. Sombolestan and Q. Nguyen, “Hierarchical adaptive locomanipulation control for quadruped robots,” arXiv preprint arXiv:2209.13145, 2022.

Thank you so much for your feedback and suggestions to improve our work! We hope that our replies and additional results have addressed your concerns. As the discussion period ends in a few days, we would like to ask if you have any further questions, and we are happy to provide more clarification. Thank you!

I thank the authors for their thorough rebuttal and new experiments.

Having gone over this again now, I think the paper has merit but I still have a few concerns w.r.t. robotics framing and the experiments:

1. Your first contribution seems to have merit but is a bit arbitrarily narrow. LLMs encode common-sense knowledge about the world, so that you can use them to improve robot interaction with the world is not that surprising in itself. The way you do this using multiple LLM queries is the novel part and seems to have merit based on the ablations you have shown. However this has some weaknesses:

   a) The paper focuses on tool use (probably because of the high-profile role it has played in theories of animal intelligence), but since the embodiment of tool use is abstracted away (relies on pre-engineered perception and motion primitive modules) it seems these methods could be used for any task planning problem. This is a good thing in a sense but it makes the focus on tool use seem like an arbitrary restriction chosen more for PR than practical considerations.

   b) In particular, the paper claims that it solves a motion planning problem but the code examples I've seen seem to mainly only move_to(get_position_of_object(...)) with possible +object_size/2 added or something similar. It is very difficult to get a sense for how much of a motion vs. task planning problem it is actually solving. This is extra problematic because the "calculator" component in your novel architecture is supposed to come up with the (real-valued) parameters but it mainly seems to just need to plug in getters for object positions. The lack of non-trivial motion planning in the experiments also undermines your proposed architecture a bit. These planner properties could probably also have been tested individually instead of everything engineered together into a complex experiment.

2. Your second contribution on creating a benchmark for tool use is also a bit weak because some of your benchmarks (especially the quadruped one) seem a) a bit contrived and/or b) a lot of work to reproduce. Again, all the hard embodiment problems of tool use are engineered away, and doing this engineering to get a quadruped robot to reliably push objects on sofas seems like potentially a lot of effort. Is there a sim version of this btw?

That said, since the architecture does show improved results in your experiments (even if it is arguably mostly "task planning for tool use", and only evaluates LLM generalization on two different versions of ChatGPT). I will tentatively raise the score one step while I digest the other reviews / discussion.