/10

Poster4 位审稿人

最低3最高4标准差0.4

ICML 2025

Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models

Lucy Xiaoyang Shi,brian ichter,Michael Robert Equi,Liyiming Ke,Karl Pertsch,Quan Vuong,James Tanner,Anna Walling,Haohuan Wang,Niccolo Fusai,Adrian Li-Bell,Danny Driess,Lachy Groom,Sergey Levine,Chelsea Finn

OpenReview PDF

提交: 2025-01-24更新: 2025-07-24

TL;DR

Hi Robot enables robots to follow open-ended, complex instructions, adapt to feedback, and interact with humans.

摘要

关键词

Machine LearningRoboticsLanguageVision-Language Models

评审与讨论

审稿意见

评分: 32025-03-13

This paper proposes a hierarchical framework for tackling the complex instruction following challenge in vision-language-action-based robotic control. The paper highlights challenges in existing methods that struggle with following intricate instructions. The proposed method, Hi Robot, addresses these issues by decomposing tasks into a high-level VLM policy, which interprets complex prompts and user feedback to generate low-level commands, and a low-level vision-language-action (VLA) policy. Evaluations on various tasks demonstrate that Hi Robot outperforms baselines.

update after rebuttal

I acknowledge the effort put into this work and appreciate that the authors have partially addressed my concerns. However, I still have reservations regarding the novelty of the work, how the two-layer framework is aligned, and its generalizability. Therefore, I am maintaining my score.

给作者的问题

Have you tested Hi Robot on unseen domains?
During task execution, if an instruction interrupts the task, can the system restore its previous state to resume the original objective?
What is the main difference between Hi Robot and RT-H?

论据与证据

The main claims are well-supported within the evaluated domains.

方法与评估标准

The methods and evaluations are well-justified for the problem.

理论论述

No proofs.

实验设计与分析

The experiments robustly validate Hi Robot’s performance within the tested domains.

补充材料

No supplementary material.

与现有文献的关系

Vision-Language-Action Models, Robot Control

遗漏的重要参考文献

其他优缺点

Strengths:

The related work is well investigated.
The idea is very intuitive.
Tested across diverse real-world robotic platforms.

Weaknesses:

Limited contribution: hierarchical VLMs with synthetic data generation.
Lack of tests in unseen domains.
No validation of the quality of the generated data.
Only the average values of the metrics are counted, ignoring uncertainty and statistical significance.

其他意见或建议

Include a discussion of failure modes.
Add more technical details to improve reproducibility.
Including statistical tests would strengthen claims.
include cross-domain tasks.

作者回复

2025-04-01

Thank you for your thoughtful feedback. We address each point below and will incorporate these improvements in the revision.

Limited contribution: hierarchical VLMs with synthetic data generation.

While individual components build on prior work, Hi Robot's novel synthesis enables critical real-world capabilities:

Open-ended instruction following (e.g., "Can you make me a vegetarian sandwich? I don’t like pickles though.")
Real-time feedback integration (e.g., "That’s all I want")
Unseen task generalization via synthetic data (demonstrated in §5.3 and video supplementary at https://hi-robot-vla.github.io/)

No validation of synthetic data quality

We evaluate data quality end-to-end through policy performance, as offline metrics for embodied data remain an open challenge. Future work could explore:

Automated fidelity checks for physical plausibility
Language-grounding consistency metrics
Interaction diversity

Statistical significance reporting

We conducted 20 trials per task per method (more than typical real-world robotic experiments). Error bars will be added to all plots in the camera-ready version.

Lack of tests in unseen domains

We evaluated generalization through:

Instruction perturbations:
- "I want something sweet" (requiring object categorization and physical grounding)
- "I’m allergic to pickles" (requiring semantic knowledge and physical grounding))
Partial task execution: The model is trained only on full-table cleaning, but we request the robot to “clean up only the trash, but leave the dishes”

Future Work: Cross-environment transfer (e.g., kitchen at home → kitchen in restaurant).

Discussion of failure modes

Common Failure Cases:

High-level:
- Temporarily ignore instruction: E.g., grabbing cheese when the robot is close to it despite user’s lactose intolerance (due to training bias toward proximal objects)
Low-level:
- OOD recovery: Dropped objects (recovery behavior is absent from training data)

Mitigations (Future Work):

Stronger instruction-following model
Adversarial data generation for edge cases
Diverse data collection including failure recovery

Add more technical details

We will expand:

Appendix Table: Full hyperparameters (learning rates, architecture specs)
Data Generation: Prompt templates and filtering examples
Failure Logs: Representative error cases

Can Hi Robot resume interrupted tasks?

Current Implementation:

Can revert to previous objectives with explicit user permission
Future: Auto-resume via success detection (e.g. via value function learning)

Difference from RT-H?

Feature	RT-H	Hi Robot
High-Level Action Space	Primitive movements (e.g., "Move arm forward")	Semantic commands (e.g., "Place a slice of bread on the chopping board")
Synthetic Data	✗	✓ (enables open-vocab feedback)
Instruction Scope	Seen tasks	Open-ended

Key Advantage: Hi Robot's rich language-action space supports real-world ambiguity and feedback (e.g., handling "This isn't trash" corrections).

Thank you for your suggestions—they have strengthened our paper. We will address all points in the revision.

审稿人评论

2025-04-02

I'd like to thank the authors for their responses. However, could you further elaborate on the training details? Additionally, how can VLM and VLA be grounded?

作者评论

2025-04-07

Thank you for your follow-up question. Below, we provide additional technical details:

Input Modalities
- Both the high-level policy and low-level policy are conditioned on two or three images, depending on the specific task. For each task, we use:
  - One third-person camera view.
  - One wrist-mounted camera per robot arm (one or two arms).
- Each image has a resolution of 224×224 pixels. These images are separately processed by the model’s vision encoder, and their resulting latent tokens are then concatenated.
Language Conditioning
- We also condition the policies on natural language instructions, tokenized using the language tokenizer from the underlying LLM. The language tokens are concatenated with the vision tokens inside the model to enable multimodal reasoning.
Model Initialization
- While our method can be trained from scratch or finetuned from any VLM backbone, in practice we use PaliGemma [1] as the base model. This is an open-source, 3-billion-parameter VLM that offers a good balance between performance and computational efficiency.
- We unfreeze the full model for finetuning.
Optimizer and Hyperparameters
- We use the AdamW optimizer [2] with $\beta_1=0.9$ , $\beta_2=0.95$ , and no weight decay.
- Gradient norm is clipped to a maximum magnitude of 1.
- We maintain an Exponential Moving Average (EMA) of network weights with a decay factor of 0.999.
- The learning rate starts with a short warm-up (1,000 steps) and then remains constant at $1 \times 10^{-5}$ .
- Batch size is 512.
Training Duration and Resources
- Training the high-level policy is highly efficient, taking about 2 hours on 8×H100 GPUs.
- The low-level policy follows a similar training pipeline, though training times can vary depending on the dataset size and complexity of the target tasks for action prediction.

We hope these details clarify our training pipeline and hyperparameters. Please let us know if there is any other information we can provide.

References

[1] Beyer, Lucas, et al. “PaliGemma: A versatile 3B VLM for transfer.” 2024.

[2] Loshchilov, Ilya, and Frank Hutter. “Decoupled weight decay regularization.” 2017.

审稿意见

评分: 32025-03-14

This paper, inspired by "System 1" and "System 2" cognitive processes, proposes a hierarchical VLM-based system to interpret high-level instructions and convert them into commands for a low-level VLA model. To train the model, the authors employ both human-labeled and synthetically generated interaction data. Some real-world experiments are conducted to demonstrate the model's ability.

给作者的问题

Please refer to the motivation and experiment concerns mentioned above.

论据与证据

Yes.

方法与评估标准

Yes.

理论论述

NA.

实验设计与分析

The experiments are only conducted on real-world robots. The demos are convincing, but the comparison with other methods has significant shortcomings in experimental settings. For example, for the comparison with GPT-4o high-level instruction decomposing experiments, what is the user prompt? What if using in-context learning, chain-of-thought, o1 or deepseek r1 model for better reasoning?

补充材料

NA. The video page was initially empty, and the video was provided after the review began.

与现有文献的关系

This paper provides a dual system for robotics. This work is one of the early efforts in understanding, translating, and decomposing high-level human instructions, and it combines VLA models to design an integrated model from human instructions to actions.

遗漏的重要参考文献

Yes. The concept of “System 1” and “System 2” is not the first proposed in this paper, some other paper should be discussed, such as [A-C].

[A] Li et al. HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation. ICLR 2025.

[B] Bu et al. Towards Synergistic, Generalized, and Efficient Dual-System for Robotic Manipulation. arXiv:2410.08001.

[C] Zhou et al. Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detection. arXiv:2412.04455.

其他优缺点

Strengths:

The paper is well-written and easy to follow.
The hierarchical understanding and reasoning for high-level human instructions are necessary for robotic VLA models.

Weaknesses:

The motivation and method are somewhat disconnected. The proposed method merely involves understanding high-level instructions, which are cascaded structures rather than a second system.
This work only experiments with the pi0 VLA model as the action model. Different VLAs may have specific preferences for different low-level language command styles. How to address this issue to make the proposed method adaptable to different action models?

其他意见或建议

NA.

作者回复

2025-04-01

Thank you for your constructive feedback. We address your comments in detail below and will update our paper accordingly.

For the comparison with GPT-4o high-level instruction decomposing experiments, what is the user prompt?

The user prompts for evaluation (e.g., "Hi robot, can you make me a sandwich with cheese, roast beef, and lettuce?") are the same across baselines and are included in the paper. If you are asking about the system prompt for GPT-4o, we provide it below (example for the Table Cleaning task) and will include it in the camera-ready version:

You are an AI assistant guiding a single-arm robot to bus tables. The robot can optionally place trash in the trash bin and utensils and dishes in the plastic box. Every 3 seconds, you can issue one instruction from a provided list. You will receive images from two cameras: one for a global view and one on the robot's wrist for detailed views. Interpret the user's instruction into one from the provided list for the robot to execute. Adhere strictly to the user's instruction. If ambiguous, reason out the best action for the robot. Only provide the exact instruction from the list without explanation. You will select your instruction from the following list: put food container in trash bin pick up chopstick drop wrapper in trash pick up plastic plate pick up the cup pick up white bowl place bowl to box pick up spoon place trash to trash bin drop box in trash place take out box to trash move to the left pick up container drop plate in bin pick up the trash pick up plastic bowl go higher place spoon to box pick up the paper container drop fork in bin pick up the bowl pick up the plastic container go lower pick up box move to the right drop plastic lid into recycling bin pick up wrapper put bowl in box pick up the container put the plate in the bin pick up cup put cup into box throw it in the trash pick up food container pick up blue cup drop the bowl into the bin move towards me pick up napkin rotate counterclockwise put the cup in the bin throw trash away rotate clockwise drop plastic bowl into box open gripper pick up plastic cup pick up the plate close gripper move away from me go back to home position <truncated due to character limit>

What if using in-context learning, chain-of-thought, o1 or deepseek r1 model for better reasoning?

In-context learning: We explored this with GPT-4o but found it did not generalize beyond in-context examples and significantly slowed inference due to long visual context. The VLM struggled to reason across many images for closed-loop tasks (e.g., sandwich making). Future work could improve learning from long-context inputs or summarize visual observations into text.
Chain-of-thought: A promising direction for future work.
o1: While strong in mathematical reasoning, its slow inference (>10s per step) makes it impractical for real-time robotic control.
Deepseek R1: Does not support visual reasoning, limiting its applicability to our task.

some other papers should be discussed, such as [A-C].

We will cite these concurrent works in the camera-ready version. Key differences:

HAMSTER [A]: Focuses on high-level VLMs generating 2D end-effector trajectories. Our work outputs language commands, enabling finer dextrous behaviors (e.g., separating sticky cheese slices during sandwich making) and on-the-fly verbal corrections. The approaches are complementary; trajectory prediction could be integrated as part of chain-of-thought reasoning in future work.
RoboDual [B]: Separates generalist (latent representations) and specialist (actions), emphasizing training efficiency. We focus on open-ended instruction following.
Code-as-Monitor [C]: Uses VLM-generated code for failure detection. Our work uses VLMs as high-level policies to guide low-level VLAs and interact with humans.

The proposed method merely involves understanding high-level instructions, which are cascaded structures rather than a second system.

Our framework explicitly separates high-level reasoning (VLM) from low-level execution (VLA). The VLM acts as a "second system" by decomposing abstract instructions into actionable commands, unlike monolithic VLAs that conflate reasoning and execution. We will clarify this distinction in the paper.

Different VLAs may prefer different command styles. How to make the method adaptable to other action models?

Hi Robot is architecture-agnostic and can integrate any language-conditioned policy. Future work could:

Fine-tune the high-level policy using successful rollouts (e.g., via SFT) to adapt to a low-level policy’s "affordance" (i.e., its language-following capabilities).
Use policy performance as feedback (e.g., via RLHF) to align the high-level policy’s outputs with the low-level policy’s strengths.

Thank you again for your thoughtful suggestions—they have strengthened our paper. We will incorporate these changes in the revision.

审稿意见

评分: 32025-03-16

This paper presents Hi Robot, a hierarchical vision-language-action (VLA) model for open-ended instruction following. The system integrates a high-level vision-language model (VLM) that interprets complex prompts and user feedback with a low-level VLA policy that executes atomic actions. A synthetic data generation pipeline augments training by creating user interactions that improve generalization to diverse tasks. Hi Robot is evaluated on three real-world robotic applications, demonstrating superior performance over GPT-4o-based high-level policies and flat VLAs. The results highlight significantly improved instruction accuracy, task progress, and real-time adaptability to human corrections.

给作者的问题

None