VIRT: Vision Instructed Transformer for Robotic Manipulation

Despite the paper naming the method with new names e.g., Robotic Imagery Pre-training or robotic gaze strategy. I don't see this paper introducing significantly new concepts or method. Imagery Pre-training could be considered just as a standard image-conditioned policy. Robotic gaze strategy can be considered as conditional policy with fixed stages, where the condition is the stage and object detection result.

The paper says "RIP eliminates the need for task-specific priors or manual annotations" however, for robot data, the most expensive part to obtain is the robot action, annotation on task is relative easy. If the pretraining still require robot action (teleoperated data), I'm not sure how much more scalable it is compare to existing approaches.

Dividing tasks into stages, where each stage only has one object of interest, is a strong assumption for unstructured manipulation tasks -- How do we define the stage for any manipulation task consistently? What if the demonstrator does the tasks in different orders or stages? What if there are multiple objects of interest?

The paper writing is a bit confusing -- after reading the abstract and introduction, I don't have a good idea what is the " imagery paradigm" the vision instruction for tasks. Or the Transformer-based policy network architecture. I feel the paper could reduce the motivation and connections to cognitive science and instead be more direct about the technical contribution to improve clarity.

It is surprising to see the task performance for prior works (both ACT and Diffusion Policy) are so low, especially for simulation tasks, since it does not seem particularly challenging. Also, the paper does not provide an evaluation of the existing benchmarks, so it is not clear whether the implementation of baselines is correct.

1. The contributions of this paper could be articulated with greater clarity and supported by a more comprehensive evaluation. The paper presents two key contributions: Robotic Imagery Pre-training and Robotic Gaze. However, while each contribution has potential, the overall impact could be enhanced with further refinement and empirical comparison:
   1. Robotic Imagery Pre-training: The paper suggests that prior work in pretraining typically relies on text annotations, which is presented as a limitation. However, integrating language as part of the design is often a deliberate choice, considering its utility for task specification in manipulation tasks, where language enables easy and flexible task specification. Similar to the proposed approach, using a goal image as a conditioning factor is well-established, especially in navigation tasks, as seen in works like ViNT[1] and NoMaD[2]. Given this context, I would suggest clarifying the novelty of using goal images for pretraining in this work. Furthermore, the evaluation could benefit from including comparisons with alternative pretraining methods to strengthen the case. The original DROID paper has demonstrated the benefits of pertaining. It would also be valuable to include a rationale for the choice of goal image for pretraining task specification. In addition, it is unclear how such pretraining setting would benefit downstream tasks.
   2. Robotic Gaze: The robotic gaze component is intriguing and has promising potential. However, the current presentation lacks clarity, particularly regarding how the object of interest is determined—whether this is learned by the policy or manually defined. Even if it is manually defined, distinguishing this approach from related methods like visual prompting, which generates affordances based on natural language (e.g., MOKA[3]), could make a stronger case for its contribution. Additionally, a comparison with visual prompting methods might reveal unique advantages or insights that enhance the impact of this work.
2. A more thorough evaluation could further highlight the contributions. Adding comparisons to established pretraining methods and including common benchmark experiments alongside the customized tasks in the paper would provide a well-rounded assessment.

To strengthen the paper, I would recommend focusing on a single primary contribution and conducting a more in-depth evaluation and analysis. For example, the paper could either concentrate on demonstrating how goal-image-conditioned pretraining facilitates better task understanding and generalization or explore how robotic gaze enables the policy to acquire complex skills.

[1] ViNT: A Foundation Model for Visual Navigation

[2] NoMaD: Goal Masked Diffusion Policies for Navigation and Exploration

[3] MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting

Minor:

1. Please consider using $\citep$ instead of $\cite$.
2. The introductory quotes, while intriguing, could benefit from a clearer connection to the core contributions and themes of the paper. I do not see how it is connected with the main contribution or argument.

1- While the study provides insights, the analysis could be more comprehensive. For example, the role of the uncertainty score is somewhat underexplored. Further exploration on how the score manages discrepancies between predicted and actual actions, especially in uncertain segments, would add value. Additionally, it would be insightful to see the impact of only using RIP without RG fine-tuning.

2- Figures 1 and 2 are somewhat inconsistent with the textual description. For instance, Figure 1 only shows image inputs, though VIRT also uses proprioceptive data. Additionally, the “Query Chunk” element in Figure 2 lacks clarity, and it would help if the paper provided more detail on this element's purpose and its role in both RIP and RG modules.

3- Despite the advantages of a vision-based approach, the method relies heavily on object detectors, which might limit its adaptability to novel or unseen objects. Handling scenes with multiple, visually similar objects could be challenging and would benefit from further clarification.

4- The experimental section could benefit from clearer task definitions. The number of task segments, initial observations, target vision instructions, and object crop regions should be explicitly shown for each task. Additionally, the segmentation of tasks into stages should be thoroughly explained in the methodology section for clarity.

Given that many prior works [1][2][3][4] utilize similar model architectures/goal representations for learning visuo-motor policies, the technical contribution of the work is diminished.

The comparison with baselines insufficient. More relevant approaches (e.g., [4][5]) should be compared and discussed.

For real world experiments, the success rates for ACT and diffusion policy are extremely low, could authors give more explanations and analysis on this?

[1] Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goals, RSS’24;

[2] Goal Conditioned Imitation Learning using Score-based Diffusion Policies, RSS’23;

[3] MimicPlay: Long-Horizon Imitation Learning by Watching Human Play, CoRL’23;

[4] ALOHA Unleashed: A Simple Recipe for Robot Dexterity, CoRL’24.

[5] 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations, RSS’24