Grounding Robot Policies with Visuomotor Language Guidance

• How does this approach compares to methods like AHA and RACER, which are instruction-tuned VLMs for failure detection and reasoning. In particular, AHA demonstrates that failure reasoning can improve base policies through a similar guidance mechanism.

A: We appreciate the feedback; however, the baselines suggested by the reviewer are still in the pre-print stage and were just recently made available on Arxiv (AHA on 1 Oct 2024, the same day as the ICLR paper deadline) and RACER one week before that). This makes the comparison requests unreasonable for the timeline of this conference. Nevertheless, we will consider these methods in future advances of our work.

• The paper claims that their approach is generalizable across various base policy classes, tasks, and environments. However, in the simulation experiments, evaluations were only conducted on a test set within the same domain distribution as the training data. No evidence was provided to demonstrate generalizability to different tasks or environmental perturbations

A: By showing that our framework can be used as a zero-shot policy, we demonstrate that it can be used to generate guidance in out-of-distribution cases. Even if the base policies were not exposed to a certain environment or configuration, the system would still be able to provide guidance grounded on trackable elements in the scene. It is worth mentioning that the generalizability of our approach is restricted by the capabilities of the VLM used and the precision of the off-the-shelf perception models queried by the agents (GroudingDINO, SAM, AOT). If the environment, objects, or task are out-of-distribution for these models, our framework might struggle to provide meaningful guidance.

• The authors also suggest that their approach can be used as a zero-shot policy to solve tasks without demonstrations. However, they did not provide any comparisons with other zero-shot methods, such as Manipulate-Anything, VoxPoser, or Code-as-Policies, which would strengthen their claim

A: Even though our method can be applied as a zero-shot policy, our system was designed to also encompass policy grounding and iterative improvement. As such, we adopted design choices that make our framework easily integrate with a diverse set of base policies. This flexibility makes the direct comparison with dedicated zero-shot methods challenging as they often rely on pre-trained low-level policies, dedicated perception systems, fixed-set actions, or multi-camera setups. Nevertheless, we acknowledge that a deeper comparison with other zero-shot methods could contextualize the zero-shoot capabilities of our framework, and will consider this for future work that explores this facet of our framework.

• In Figure 2, the example shows the need for granular object search due to the cluttered environment and large scene. However, this level of detail seems unnecessary for the real-world experiments presented in the paper. It would be beneficial to include more real-world experiments in complex, cluttered environments, where granular object search would be more impactful and better showcase the framework's strengths.

A: In Figure 3, we demonstrate how our method benefits from multi-granular detection. In this qualitative example, the target object (white knight) could not be directly detected by the off-the-shelf perception modules. However, by performing the multi-granular search and first locating a “board” and then looking for the “white knight” within the “board” region, our method successfully located the target object and used it to ground the robot's behavior. We updated the paper with a more intuitive and representative figure to demonstrate this use case.

• The paper could benefit from a clearer explanation of the system through the figures. Providing more detailed descriptions, especially in figures like Figure 2, would help readers better understand the design principles and flow of the system.

A: We appreciate the feedback, we have attached a new version of the paper with updated figures for a clearer presentation of the method.

• Additionally, the definitions of "1% and 10% guidance" are unclear and need further clarification. Explaining how these percentages are determined and applied in the system would provide valuable context for understanding their role in guiding the base policy.

A: The guidance percentage is defined by the alpha parameter in the equation (3), lines 285 and 286. Its value, between 0 and 1 controls the element-wise weighted average between the base-policy distribution and the distribution generated by our framework. A value of “10% of guidance” translates to an alpha of 0.1, meaning that the logits of each possible action given by the base policy are scaled by 90% and combined 10% of the logits provided by our guidance function.

• A detailed explanation of the full inference process

A: We appreciate the feedback. We included an updated version of the paper with a clearer representation of the process of guidance generation. We represent graphically how each agent interacts with each other to generate the guidance function, a process referred to as Generate_Guidance$(o_t, s_{t+1})$ in the algorithm I. We provide a summary of this process, already described in Section 3.3 and further clarified in the revision of the paper: Given an image of the environment and a robot state, the group of self-orchestrated agents aims to generate a guidance function (Python code) to modify the action distribution of a base policy. Each agent is a conversational Visual Language Model (VLM) prompted to exercise a role. These agents use tools to enhance their responses through function calling. The agents interact in a “group chat” fashion, sequentially via natural language. The order of speakers is defined by the agents. Figure 2 and lines 198-225 depict the flow of information exchanged between these agents during the guidance generation procedure. The specific content of the messages exchanged and the order of speakers is flexible, however, the agent's prompt outlines an overall procedure to be followed. The diagram in a new figure added in the Appendix depicts this procedure. The “Advisor agent” coordinates the process and is responsible for the code generation; the “grounding agent” can interact with perception models and is responsible for detecting, segmenting, and tracking elements in the scene to ground the code on existing elements in the scene; The “robotic agent” ensures that the code satisfies the platform restrictions and the expected input/output formats; The “Monitoring agent” is used to analyze the execution of the robot in the environment and raise possible reasons of failure.

• How are Generate_Guidance and Infer future state implemented?

A: The algorithm proposed leverages the Agentic Framework to generate the “guidance code”, which then alters the action probabilities of the base policy. The Agents are instantiations of VLMs empowered with tool-using capabilities. During the guidance code generation, these agents exchange several messages with each other (in natural language) in a self-orchestrated manner. With that, defining on their own the content and recipient agent of every message exchanged until the guidance code is considered to be adequate for the robot, task, and current state of the environment. This process is explained and exemplified in sections 3.2 and 3.3. The process of inferring future states is given by the Dynamics model of the robot, inferring future robot states from current state and action. This process is highly dependent on the robotic embodiment used and the action space of the base policy. For our experiments, the action and state space of the policy match, meaning that the policy outputs the poses that the robot should be moved to, and its gripper state. This simplifies our formulation to a case where the possible future states of the robot are directly obtained by the possible actions that the policy can output.

... (continue in response 2/2 )

I would like to thank the authors for their clarifications. After reviewing the response, I still have a few concerns as outlined below:

1. Coupling with a specific robot policy: The proposed approach appears to be tightly coupled with a specific robot policy (3D Chained Diffuser), which utilizes a high-level macro action to guide a low-level trajectory generation policy. The VLM guidance is implemented by modifying the distribution of intermediate macro actions based on VLM outputs (the proposed method itself is also computationally expensive). However, it remains unclear how this approach could be adapted to more general robot policies that use standard action spaces and do not have a hierarchical structure (e.g., diffusion policy, ACT, etc.). There is limited insight into the algorithmic aspects of integrating VLM guidance into such more general settings.

2. Comparison to related work: On the system side, the improvement seems primarily driven by VLM-based object understanding (e.g., detection). Related works, such as OK-Robot [1] and VoxPoser [2], also employ VLMs to ground language to objects and use specific robot policies (e.g., AnyGrasp) to manipulate those objects. It is unclear how the proposed method addresses the limitations of these existing approaches or advances beyond them in a meaningful way (e.g. the proposed method can guide a specialized robot manipulation policy (with few assumptions on its architecture) to perform better in challenging dexterous/precise manipulation tasks.).

Given the authors' efforts to clarify certain aspects, I am increasing my score to 5. However, I believe the concerns outlined above merit further discussion and exploration.

[1] Liu et al. OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics. In RSS 2024. [2] Huang et al. VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Models. In CoRL 2023.

We sincerely thank you for the positive rating and feedback. In response to your comments, we provided explanations to address them as well as revisions to our paper to aid in clarity. We hope to highlight the strengths of our framework, increasing your confidence in its contribution to the field. Thank you once again for your constructive review and valuable insights.

1. The experimental results show performance of three algorithms: Act3D, 3D Diffuser Actor, and a RandomPolicy. The reasons behind this choice is not clear and it would be helpful to understand what's behind this, and whether other types of algorithms would also be suitable, and how their performance is expected to be.

A: We choose baseline policies that showcase diversity in representation and achieve strong performance in the language-conditioned manipulation field. The distinct output formats, and policy architectures, showcase that our framework could be adaptable to a wider range of policies that follow similar specifications. Section 3.4 discusses how our framework could be suitable for a wider range of policies. The random policy showcases that our framework can guide even untrained policies in diverse environments.

2. The real robot evaluations is interesting, although the task defined is not interactive at a physical level, being only a reaching task. The results are interesting and valuable, however, real-robot challenges arise often when interaction with environments exists.

A: We showcase tasks that require a complex understanding of the scene, e.g., pressing buttons in a specific order and reaching for a white knight in a cluttered environment. Both of these tasks, although relatively simple in their low-level motion, are challenging due to the fine-grained perception and long-term reasoning required. We also clarify that the button-pressing task includes physical interaction with the button.

3. Can you clarify the choice of algorithms chosen? Are results available from other algorithms too? Are there specific reasons behind this choice?

A: The chosen base policies (Act3D and 3D diffuser actor) are strong baselines in the field of language-conditioned manipulation. These approaches are fundamentally different in their output formats (classification vs. regression) and architecture; they represent informative model classes that act as great use cases for illustrating the versatility of our policy guidance framework. By improving the performance of both algorithms, we demonstrated that our framework can address and improve a diverse set of policies. The performance of these methods is reported in Table 1.

4. What are the limitations of the learning from scratch? Can the system effectively incorporate preferences/demonstrations?

A: The main limitations of the learning-from-scratch process are based on the need to improve upon motions that require precise low-level control of the robot or reasoning about fine-grained interactions with the scene (e.g., areas of contact). Crucially, these limitations are mainly explained by limits on the current capabilities of the grounding models (GroundingDino, SAM, and AOT) and challenges in the ability for Visual Language Models (VLMs) to perform time-dependent scene understanding. Still, even with the current capabilities of these grounding models and VLMs, we are able to show increments of performance improvements on different tasks.

5. Can you elaborate on why the self-improvement experiments plateau?

A: Due to the intrinsic limitations of the perception models used and the scene understanding of the monitoring agent, our ability to detect and correct failures becomes restricted. Because of that, our self-improvement experiments reach a plateau in performance. A future direction we are willing to explore is how to combine the self-improvement approach proposed with other exploration methods such as Curiosity-driven RL. Combining both could lead to improvements in the low-level control of the robot while allowing sample efficiency by targeting the exploration with our system’s guidance.