HAMSTER: Hierarchical Action Models for Open-World Robot Manipulation

Thank you for the detailed review! We are happy that you agree our approach significantly outperforms the baselines in the real world, and that we leverage a VLM in a computationally efficient manner. We respond to individual points below:

Q: Why do you only use the front_cam view in RLBench? Can HAMSTER generalize to multi-view?

A: We only use front_cam as it is this view that contains best visibility of most tasks. Other standard camera angles like left_shoulder or right_shoulder usually have a large portion of the image blocked by the robot arm.

Despite this, HAMSTER is quite robust to different views. We have the camera pointing toward the robot base because this view point has best visibility and is easy to set up, but the HAMSTER is robust to common viewpoints. See the below video for a demonstration: video

Q: Why did the authors switch from RVT-2 to 3D-DA for this setting [simulated experiments]?

A: We used 3D-DA as it performed similarly to RVT-2 on RLBench, and at the time of writing, we encountered implementation issues with RVT-2 at the Colosseum. For consistency, we have now added results for HAMSTER with 3D-DA and the vanilla 3D-DA on the real robot in the paper (see the updated Fig 3). Hence, we have now evaluated HAMSTER with 3D-DA both in sim and real. A brief results are shown below.

Q: The authors do not discuss common failure modes of the proposed approach.

A: Thank you for the suggestion. We have uploaded failure videos to the supplementary website and updated the failure analysis in Appendix E. System failures are categorized as follows:

• VLM Path Generation Failures – Whether the VLM produces a reasonable path. Video
• Path Following Failures – Whether the low-level policy follows the predicted path. Video
• Action Execution Failures – Whether the low-level policy fails even after following the correct path. Video

We observed distinct tendencies across methods:

• For RVT, when the VLM prediction is correct:

  • 72% of failures are due to RVT not following the trajectory.
  • 28% are due to action execution failures.

• For 3DDA:

  • Only 10% of failures are due to not following the trajectory.
  • 90% are due to action execution failures.

This discrepancy likely arises because RVT includes a re-projection step, complicating trajectory adherence, whereas 3DDA uses a vision tower that processes the original 2D image, simplifying trajectory interpretation.

See this pie chart for a breakdown of failure modes. We have also included this figure in the appendix. Thanks again for the suggestion!

Q: Why human ranking was used as an evaluation metric for the VLM's trajectories? No information about human study details? Why 2D trajectories are not evaluated based on MSE?

A: Thank you for pointing this out! We have now added more information about the human evaluation to Appendix D.2, restated here:

We use human evaluations to evaluate VLM-produced trajectories instead of MSE against some ground truth trajectory because there are many possible trajectories that can solve any single task. MSE is a metric that assumes only 1 possible ground truth trajectory.

We recruit 4 human evaluators who have not seen the path outputs of HAMSTER before to grade the VLMs based on the instruction: “Provide a rank for each method (1 for best and 4 for worst). In your opinion, which robot trajectory is most likely to succeed. Here is an illustration of the sample to be graded by the human evaluators link. Traj goes from blue to red, blue circle means close gripper, red circle means open gripper.'” The evaluators are allowed to give multiple trajectories the same score if they believe those trajectories are tied.

Q: Sim experiments results: Only 8% increase in performance, high variance of results?

A: Thank you for highlighting the issue of high variance. Following your suggestion, we ran the simulated experiments with additional seeds (now 5 in total) to reduce the variance. The updated results show that HAMSTER+3D-DA outperforms 3D-DA in every Colosseum variation category, as shown in the revised Table 2 (reproduced below).

This represents a 31% relative increase in success rate, from 35% to 46%. We believe this improvement is significant, considering the VLM is trained on RLBench but not on Colosseum. As shown in Figure 13 in the appendix, Colosseum scenes are visually distinct from those in RLBench. These results demonstrate that the VLM significantly enhances 3D-DA's ability to generalize to new environments.

Q: How is time encoded in the 2D paths? 2 trajs w/ different lengths? Can this approach encode the speed that the gripper should be moving?

A: We assign a normalized value between [0, 1] to each point in the 2D trajectory, 0 for the starting point and 1 for the last point. We then scale the red channel with respect to the value for each point. Since the last point is 1 for all trajectories, two trajectories with different length will all end with the same color.

We do not encode the speed at which the gripper should move, however it is a great suggestion for future work!

Q: Why does the VLM generate 2D trajectories in image space instead of also predicting gripper motion or rotation?

A: We agree that predicting gripper rotation could provide more information to the low-level action model. However, we chose to predict 2D trajectories and gripper actions for the following reasons:

1. Complexity of Gripper Orientation: The orientation of the gripper is influenced by factors like the spatial relationship between the robot, camera, and table, making it more complex to predict. For example, the gripper pose in camera coordinates varies significantly depending on whether the camera is on the same side or the opposite side of the robot. Predicting 2D paths, which are less affected by such factors, allows the VLM to generalize better across various camera angles in zero-shot settings.

2. Cross-Robot Generalization: To train the VLM with off-domain data from different robots, we require a representation that is transferable across robots. While gripper orientations may vary between robots, the locations of end effectors (2D paths) are more consistent and easier to transfer.

By focusing on 2D trajectories, we ensure better generalization and compatibility across diverse setups and robot types.

Q: Why do you think 3D-DA performed worse when the language prompts were removed?

A: 3D-DA uses a visual attention mechanism that attends CLIP image feature tokens to corresponding CLIP language embeddings. We believe this attention mechanism is important to 3D-DA’s performance; we have updated Appendix B.2 with this discussion.

Let us know if this addresses all of your concerns. Except for the ongoing experiments, we would be happy to address any remaining concerns. Thank you!

Q: Why do you only use the front_cam view in RLBench? Can HAMSTER generalize to multi-view?

Following up on the original response for thie question, now have new experiments demonstrating generalization to a different viewpoint! The experimental environment is shown in this image, where the right camera is the original one, and the left camera is the novel camera. Since the viewpoint robustness is already demonstrated in video, we focus on evaluating the action-level model here. More specifically, we tested on 6 seen objects and 3 seen containers with 10 different combinations.

The results are summarized in the table below. Both OpenVLA and HAMSTER+RVT2 follow the same setup as in Figure 2 of the paper. HAMSTER+RVT2 (Concat) incorporates the 2D trajectory as additional channels, instead of overlaying it on the image, as suggested by Reviewer MHuh. In the metric, score is using breakdown point as in Figure 2 and defined in Table 3, while fully success only count on trails which are full successful.

From the results, it is evident that both versions of HAMSTER+RVT2 perform robustly even when the camera setup is changed, despite the VLM only being trained on front_cam. In contrast, OpenVLA exhibits a significant performance drop when tested with a different camera from the one it was fine-tuned on.

We thank the reviewer for this insightful suggestion and will incorporate these findings into the revised manuscript.

Thank you for the detailed response and for providing additional experimental results. The video and quantitative results on novel view generalization are great! I am more convinced by the simulation results with the extra seeds and error bars included.

I think the human evaluation of 2D trajectories experiment weakens the paper. I would recommend moving it to the appendix and using that space for the generalization results. To make the results more compelling, you could add more evaluators or provide details on how the evaluators were instructed/trained to grade results (I am not expecting this in rebuttal period). If these evaluators were unfamiliar with robotic arms, it's not clear how well they could evaluate the quality 2D projection of a robot trajectory.

Thanks for the follow-up! We have made changes to the paper based on your suggestions. Detailed response below:

Q: Move human evals to appendix and put in generalization results?

Thank you for this suggestion. We have followed this suggestion and now highlight all of the new results in the main paper’s experiments section in the revised rebuttal PDF.

Q: More details, adding more evaluators, and evaluators familiar with robot arms?

We have added an additional evaluator for a total of 5 and updated the corresponding table in Appendix Table 5. The evaluators are robot learning researchers who have never seen the predictions of the evaluated VLMs before. Hence, they should be familiar with the quality of robot trajectories. We also list the full instructions given in Appendix D.3, along with all of the above details!

Thank you again for following up. Your suggestions have helped us improve the paper as we have now demonstrated novel view generalization, improved simulation results, highlighted these experiments in the main paper, improved the human evaluations, and updated writing to clarify details! Please let us know if you have any further concerns.

Thank you for the detailed review! We are happy you find the problem important, presentation clear and the off-domain VLM fine-tuning idea inspiring. We respond to individual points below:

Q: Additional technical and experimental investigation?

A: Thank you for your suggestion! We have added discussions on various aspects, including different intermediate representations, methods for providing path information to the low-level action model, the impact of different training demonstrations, and the effect of the number of points on the trajectory. We will update the paper to include the results once the corresponding experiments are completed.

Q: Focused evaluation of the hierarchy of HAMSTER: Why not just use start and end points instead of the full path?

A: This is an interesting suggestion! We agree that start and end points can be easily extracted from the 2D-paths, however, they often do not contain sufficient information for specifying the task for the low-level policy. See the two following examples: Consider the task of wiping the desk and putting the towel back, see image. Just the start and end points cannot sufficiently inform what the robot need to do with the towel. Similarly, consider the task “Press the yellow button, then blue button and then red button” image. The start and end point alone cannot communicate with the sequence of actions. Per your suggestion, we are also running additional experiments with training HAMSTER+3DDA with just the gripper open/close points without drawing any paths. If they finish after the rebuttal, we will include these experiments in the next version of the paper. Thank you!

Q: Comparison to ReKep [1] b/c it doesn’t need a lower level action policy + produces keypoints directly?

We note that comparing our method to [1] is out of scope, as per ICLR policy. The paper was released to arXiv on September 3, 2024, and hasn’t been published in a peer-reviewed venue. Only works published before July 1, 2024, should be considered for comparison.

Despite this, we compare our method with Rekep: Rekep requires much more state information than imitation learning-based action models. As a model-based method, Rekep needs a TAMP model, a TAMP optimizer, a collision oracle, and a motion planner. It also relies on accurate point tracking for closed-loop optimization. HAMSTER does not require any of these.

For key-point prediction, Rekep runs K-means clustering on DINOv2 features to get keypoint proposals and selects relevant ones with VLM. This approach depends on the quality of DINOv2 features and hyperparameters for K-means clustering, so it likely struggles in cluttered environments like video. We’re currently deploying ReKep in our real-world experiments for quantitative comparison; we’ll share updates as the experiments progress and include a full comparison in the camera ready.

[1] Huang, Wenlong, et al. "Rekep: Spatio-temporal reasoning of relational keypoint constraints for robotic manipulation." arXiv preprint arXiv:2409.01652 (2024).

Q: Why draw paths on images? Compare against other methods of giving path information to the low-level policy?

A: We draw VLM-produced paths onto the image for the lower-level policy to condition on because it is a generalizable method of conditioning the lower level policy on paths that works for any image input policy.

Per your suggestion, we are now running additional experiments comparing against a version of HAMSTER+RVT2 which concatenates just the path image with the RGB input image (6 channel input) to specify the path. Note that this already is less generalizable across lower-level policies, as we cannot easily perform this with 3D-DA due to it using pre-trained CLIP image encoder backbones that take 3-channel RGB images as input.

Q: No experiments showing “drastically reduced number of needed demonstrations”?

A: Thanks for pointing this out. We agree that in the related work section (Sec. 2), we mentioned our methods would “drastically reduced number of demonstrations” without explicit experiments to support it. Since this is not the primary focus of the paper, we have modified this sentence to better reflect the focus of the paper, that “Our work takes a hierarchical approach, enabling us to use specialist lower-level policies that take in additional inputs the VLMs cannot support, such as 3D pointclouds, to enable better imitation learning.”

Further, we agree with your suggested experiment. Hence, we are now running additional experiments in which we train HAMSTER+3D-DA on only 50% of the data to compare it with vanilla 3D-DA. We will add those results in the next version of the paper as soon as they are finished.

We have now added new experiments addressing pending concerns above! We list them below and have updated the paper with these results in Appendix D.3.

Q: Why draw paths on images? Compare against other methods of giving path information to the low-level policy? We now have new experiments in the real world giving path information to RVT2 by concatenating the path image as 3 additional channels in addition to the RGB image channels. Here we conducted 10 pick-and-place trials with 6 seen objects and 3 seen containers, and also with novel camera viewpoints. The results are as shown below, where column 4 (HAMSTER+RVT2) is the one used in Figure 2 which draws a path directly on the image, while column 5 (HAMSTER+RVT2 (Concat)) is the new experiment. We can see that although drawing directly on the image is a more generalizable way for different low-level action models, HAMSTER with concatenated images works even better with RVT2. Thank you for the suggestion!

Q: Additional technical and experimental investigation?

The above experiment table now also demonstrates an additional experiment investigating the hierarchy of the model as suggested. We demonstrate that HAMSTER is much more camera-view invariant than OpenVLA. This specifically highlights the benefits of hierarchy; HAMSTER’s VLM is trained on additional web pixel point data containing images of different scenes w/ diverse views, and RVT2 is a specialized imitation learning policy which performs camera view reprojection and is more robust to novel views.

Q: No experiments showing “drastically reduced number of needed demonstrations”? Finally, we have also added new experiments demonstrating that HAMSTER can perform better than the underlying imitation learning model even with far less demonstrations. Our results, obtained over 5 seeds across 5 short-horizon tasks in Colosseum (slide_block_to_target, place_wine_at_rack_location, insert_onto_square_peg, stack_cups, setup_chess) shown in the table below, demonstrate that with just 50% of the data, HAMSTER+3D-DA performs 2x better than 3D-DA with 100% of the data.

Please let us know if you have any remaining concerns. Thank you!

Thanks for the followup! We have now addressed your two points on multi-modality and a Rekep comparison:

Q: Specific path’s effects on multi-modal action distributions?

Thanks for the clarification about your original question. We have run new experiments demonstrating that yes, indeed HAMSTER’s low-level policy is able to learn a multi-modal distribution! We evaluated HAMSTER+RVT2 with the same path drawn on the image (see the setup here) but with different objects to place and receptacles of varied sizes and heights. This RVT2 policy is able to vary the low-level actions to change the height at which it picks up and places the objects despite being conditioned on the same path: see video 1 grape in bowl, video 2 milk in bowl, video 3 grape in mug

HAMSTER is able to do this because (1) the 2D path still underspecifies the low-level environment actions in terms of depth and rotation, and (2) we explicitly train the low-level imitation learning policy with noisy ground truth paths so it does not overfit to following the exact action sequence outlined by the path.

If our paper is accepted, we will add these experiments to the camera ready. Thank you for the great suggestion that allowed us to demonstrate that HAMSTER can produce multi-modal action distributions!

Q: Comparison to keypoint based methods like ReKep b/c it doesn’t need a lower level action policy + produces keypoints directly?

In our previous discussion, we addressed the question of why a low-level policy eliminates the need for the state information that ReKep requires. In short, the difference lies in the contrast between model-based TAMP systems and model-free learning-based methods.

Now, we also address why fine-tuning the model is essential, even when ReKep demonstrates high accuracy in its benchmarks. We evaluated ReKep using their official repository, utilizing the same input image and the same bounded range of point cloud data. In one test scenario, we tasked the network with predicting a trajectory to press four buttons within the same image.

The results are visualized here,where the leftmost column is the input to each VLM in HAMSTER and ReKep, the right 4 columns are their prediction under different tasks.

In the results image, you can see that ReKep’s pipeline misidentifies where the button is in 3 out of 4 of the tasks, while HAMSTER correctly identifies 4/4 times. It’s important to ensure the keypoints for the buttons are correctly placed on top of the buttons to be able to solve button-related tasks here. Without correct keypoints, and without a low-level policy that can learn to correct for these keypoint identification errors, ReKep would fail at executing these tasks.

What’s more, even for the successful case, ReKep proposes a constraint which presses the gripper 5 cm below the keypoint. However, the button is only 3 cm thick, therefore this operation could harm the robot.

These results highlight the strengths of HAMSTER vs ReKep in that it does not require state information (through use of a low-level policy) and our use of a fine-tuned VLM for path prediction results in accurate object-centric paths.

We detail the ReKep output results below: press the yellow button, press the blue button, press the button with color of fire, press button with color of leaf.

Please let us know if you have any further concerns preventing a higher score. Thank you!

Thanks for the quick response. Following we address both your concerns:

Q: Impacts of training low-level action models with different intermediate representations, i.e., the key-points vs. the 2D-path?

We have previously demonstrated through examples that just keypoints extracted from start/end points or gripper open/close points are insufficient in solving manipulation tasks, particularly long-horizon ones, such as wipe the table and put it back and press green, blue red button in order.

Further, inspired by the reviewer’s question, we investigate a stronger keypoint representation: drawing VLM-predicted path keypoints without drawing the lines in between.We find that both qualitatively and quantitatively that the 2D paths are more effective than just keypoints.

We agree with the reviewer that for some tasks keypoints may be sufficient but it need be the case as shown below. For example, consider the task of `stack cups’, where it is hard to understand distinguish the path with just the keypoints as shown here: path image vs. this keypoints image

This observation is also verified empirically as shown below. Over 5 seeds in Colosseum, we find that just drawing keypoints performs worse than drawing full trajectories; however both strategies are better than just the low-level policy alone. This outcome may be due to the fact that providing complete 2D paths, rather than isolated keypoints, makes it easier for the low-level action model to interpret and follow the guidance effectively.

Q: Multimodal experiment: still following path? Does path-guidance impose additional constraints?

We clarify that we actually explicitly aim to constrain the policy with path guidance. As we have demonstrated experimentally, this helps the policy generalize to new tasks with less data, (Figure 3). This path constraint is reasonable as our fine-tuned VLM is able to produce high quality paths (see Table 5 and multi-view consistency video).

Therefore, the multi-modal experiments in our previous response successfully demonstrate that the policy is still following the path in 2D space, but able to produce multi-modal low-level actions by adapting to the heights of various objects.

However, we agree with the previous suggestion that “maybe a very simple policy architecture with MLP as the action head is sufficient.” This actually highlights one of the strengths of HAMSTER: that it is not tied to a specific low-level imitation learning architecture. We apologize for the previous misunderstanding regarding these experiments: we will later include experiments in the camera-ready with a simple Gaussian MLP policy head.

Please let us know if this addresses your concern about path guidance constraints.

We are happy to discuss further and attempt any suggested experiments in the remaining discussion period.

Thank you for the insightful question! The high-level VLM is expected to handle new camera viewpoints well, as demonstrated in this video. For the low-level controller, we would expect a 3D policy to generalize better than a 2D one since it's lifted to a 3D space in a common frame of reference rather than being strictly viewpoint dependent. We are currently running experiments to validate this, we will report back soon.

Thank you for the positive feedback! We are happy you find the paper sound, our evaluation convincing, and the baseline comparisons fair and detailed. We respond to individual concerns below:

Q: Unify notations in the problem formulation.

Thanks for the feedback, we have now fixed these issues!

Q: What is the use of the intermediate representations li of the hierarchical VLM?

A:$l_i$ is a typo that instead should refer to the path produced by the VLM, $p_i$. We have fixed this typo in Section 4.1.

Q: In the fine-tuning phase, are the outputs of VLM normalized across datasets? For instance, the pixel locations are normalized to [0,1] or not?

A: Yes, we normalized the pixel location to [0, 1] with respect to the height and width of the image. We have clarified this in Section 4.1.

Q: What is the length of the output of the Ramer-Douglas-Peucker algorithm?

A: We do not limit the length of the RDP algorithm. The output length is typically between 2-5 points for each short horizon task. We have added this detail to appendix Section B.

Q: Which part of the system has the 3D knowledge of the world? Does VLA project 3D paths in the real world to images or does the policy deal with it?

A: The low-level action model handles the 3D understanding and predicts accurate actions. After the VLM predicts 2D paths, we draw these 2D paths on the image. The low-level action model takes this updated rgb image along with the point cloud as input to predict actions in 3D.

Q: Minor typos

Thank you! These are fixed.

Let us know if this addresses all of your concerns. We would be happy to address any concerns that still remain. Thank you!

Thanks for the followup! We respond to each point below:

Q: Human-engineered prompts for the VLMs?

We clarify that our prompt is quite short—in fact, our RT-Trajectory baseline prompt in Fig 10 is significantly longer than ours in Fig 8—and also not hand-engineered for our experiments: we use the same prompt template that was used for fine-tuning the VLM on path prediction for all HAMSTER evaluations in simulation and real.

In fact, even after training, the VLM is quite robust to prompt variations! We have shown results in this google doc where we use a screenshot from OpenVLA of the coke on Taylor Swift task and use varied questions and prompt templates to query the VLM. We bold the words which are changed over the original prompt template. Benefiting from the strong ability from the original VLM, it can understand different ways describing the same task and generalize to different prompt templates.

Q: Human effort: Conditioning on paths? Running the Ramer-Douglas-Peucker algorithm?

We believe this approach to be quite general that does not require any scenario-specific engineering. We draw paths directly on the images in the same way no matter the low-level imitation policy or task, both in simulation and in real world experiments. We also run the RDP algorithm with the same parameter ($\epsilon=0.05$) for all data as a simple preprocessing step before training the VLM. We did not tune this parameter, we simply selected it at the beginning as it is the default parameter and the paths simplified under this param look reasonable.

If accepted, we will add the above important discussion to the appendix for the camera ready. Please let us know if you have any further concerns precluding a higher score. Thank you!

We thank you for the thoughtful feedback. We are happy you find the paper interesting and like the hierarchical approach. We respond to individual comments below:

Q: Only works on pick and place tasks?

A: We include more than just pick and place tasks (see appendix Table 3 for the full task list). We have additionally grouped tasks by their type below:

We can see HAMSTER outperforms OpenVLA across all task groups and is therefore not limited by the VLM’s 2D perception capabilities in solving more complex tasks. For clarity we have now added this as Table 4 in the appendix and reference it in Section 5.1.

Q: No pushing/rotation tasks due to VLM’s 2D perception?

A: The aforementioned “knock down” task in the table above involves pushing an object down. We also demonstrate the ability of HAMSTER’s VLM to generate rotations and pushing in the following video stow object in shelf, close drawer. Due to time constraints, we don’t have full task evaluations for rotation tasks but we are running these experiments for the future version of the paper.

Q: You’re not solving 3D perception with VLMs? 2D only is a limitation?

A: The 3D information can be used for two purposes: reasoning and understanding the task, and accurately predicting the action. Our VLM, although limited by monocular RGB input, can still address many 3D reasoning challenges, such as inferring the top, middle, or bottom drawer as shown in this image. For action prediction, we leverage an action model that incorporates 3D input, enabling us to handle more complex tasks effectively.

Q: Novelty — comparison against prior work on VLM keypoint methods?”

A: The two most similar VLM waypoint generation works would be Code-as-policy in RT-Trajectory [1] and LLaRVA [2]. RT-trajectory introduces waypoint generation as a method of specifying tasks to policies; in contrast, our novelty comes from a pipeline for using cheap, cross-modal data to fine-tune VLMs to produce accurate paths for 3D imitation learning policies. We directly compare with RT-Trajectory in Table 2, replicated below:

Where we see that HAMSTER’s VLM is best-ranked by humans for accurate path predictions, demonstrating the need for fine-tuning on cross-modal data. LLaRVA, on the other hand, uses waypoint prediction as an auxiliary objective to fine-tune a monolothic VLA; our experiments in Figure 3 demonstrate that our hierarchical VLA approach is superior to monolothic ones (OpenVLA). More detailed information can be found in Section D in Appendix.

[1] Gu et al., RT-Trajectory: Robotic Task Generalization via Hindsight Trajectory Sketches, 2024. [2] Niu et al., LLARVA: Vision-Action Instruction Tuning Enhances Robot Learning, 2024.

Q: Future plans incorporating 3D/multi-view data into the VLM?

A: We appreciate the reviewer’s suggestion! In future work, we plan on incorporating multi-view input to the VLM and fine-tuning the VLM on 3D data for effective high-level even in multi-view robot setups.

Let us know if this addresses all of your concerns. Except for the ongoing experiments, we would be happy to address any remaining concerns. Thank you!