Building Generalist Robot Policy from Pre-trained Visual Representations

• One of the main findings in this submission is not new, as Shang et al. [1] have shown the importance of using spatial features for Transformer-based visual encoders and the performance scaling regarding model sizes. This submission skips other commonly used pre-trained vision foundation models (in robotics) such as ViT [2], MVP [3](or MAE [4]), SigLIP[5], and recent work like RADIO[6] and Theia [1].

• The paper studies the visual representations in building general policies, but the evaluations are conducted on a very limited scale.

  • First, the evaluations are done in only one simulation suite -- Metaworld. The number of tasks is not explicitly mentioned in the main paper or the Appendix. From Table 12 in the Appendix, it looks like there are only 10 tasks. This scope is a bit far away from being called a generalist policy.

  • The evaluations should consider other benchmarks with different visual domains. Available simulated benchmarks such as LIBERO [7], CALVIN [8], RoboCasa [9], DMC [10], and Habitat might be helpful (like what Cortexbench organized). I strongly recommend the author also use real-world data like OXE [11] or DROID [12], just like what OCTO [13] or OpenVLA [14] did, where the data show more complex visual distributions.

  • Settings are not clear, such as the number of random runs and amount of demonstration used.

• More importantly, this submission lacks analysis on why different visual representations exhibit different performance under generalist policy learning. This question is very interesting to me, but unfortunately, I can not find any explanations in the submission. I wish the authors could discuss any in-depth connections between the policy performance and how visual representation/encoder is obtained, such as datasets, objectives, properties, and architectures. It could also be more quantitative measurements you find in your study.

• What's the purpose of inputting multiple views? How do the findings in this submission transfer to other settings with more or fewer views? Within the scope of this submission, I would assume DINOv2 gets the best performance because of its better cross-view alignment performance using its feature. Is that true? The study is missing.

• The value of different augmentation techniques is unclear. Though different techniques have different effects on different models, DINOv2 seems to be the best most of the time. It would be good to further investigate why and how to further improve the best-performing model.

• The proposed few-shot adaptation looks interesting, but the connection to pre-trained visual representations is unclear. The self-distillation applies to the latent feature in the policy network (towards the head part from my understanding). How does this specific technique address the discrepancy between seen and new visual representations? Why do different pre-trained visual representations exhibit different performances? I also recommend testing LoRA [15] in the few-shot adaptation evaluation. More interestingly, one recent preprint ReVLA [16] claims that resetting the visual encoder weights to the original ones after fine-tuning could benefit generalization. The authors may also consider this direction. I also recommend authors compare to other adaptation methods or continual learning methods to thoroughly investigate the adaptation method.

Overall, I believe the technical findings and evaluations in the current submission are limited, the novelty is somewhat but needs further careful evaluation. The clearness of ideas is good, but not for technical details like environmental settings. Based on these reasons, I recommend a rejection at this moment.

References:

[1] Shang et al., "Theia: Distilling Diverse Vision Foundation Models for Robot Learning", CoRL 2024

[2] Dosovitskiy et al., "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale", ICLR 2021

[3] Xiao et al., "Masked Visual Pre-training for Motor Control", 2022

[4] He et al., "Masked Autoencoders Are Scalable Vision Learners", CVPR 2022

[5] Zhai et al., "Sigmoid Loss for Language Image Pre-Training", ICCV 2023

[6] Ranzinge et al., "AM-RADIO: Agglomerative Vision Foundation Model - Reduce All Domains Into One", CVPR 2024

[7] Liu et al., "LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning", NeurIPS 2023

[8] Mees et al., "CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks", IEEE RA-L, 2022

[9] Nasiriany et al., "RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robots", RSS 2024

[10] Tassa et al., "DeepMind Control Suite", 2018

[11] Open X-Embodiment: Robotic Learning Datasets and RT-X Models, ICRA 2024

[12] DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset, 2024

[13] Ghosh et al., "Octo: An Open-Source Generalist Robot Policy", RSS 2024

[14] Kim et al., "OpenVLA: An Open-Source Vision-Language-Action Model", CoRL 2024

[15] Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models", 2021

[16] Dey et al., "ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Models", 2024

1. Lack of real robot experiments. Due to the large visual gap between simulation and the real world, and considering that the focus of this paper is to study the pre-trained visual representations which is pre-trained on real-world data, only simulated experiments can not support the arguments of authors. Besides, the diversity of simulated tasks is also very limited. It would be good if authors show more real robot results that are consistent with simulation results, and more challenging simulation tasks beyond MetaWorld might be good, such as RoboMimic/ManiSkill/RLBench.
2. Lack of novelty. The most interesting takeaway from this paper is the usefulness of local features over global features, which however is mostly obvious and well-known to the community. Besides, even the technical contributions of this paper seem to be very incremental.
3. Overclaim of the title. Though the title is about "building generalist robot policy", I would suggest that authors carefully choose a humble title to better reflect their actual contributions of the paper. An example of the generalist robot policy is [1].

[1] https://www.physicalintelligence.company/blog/pi0

Many works on robotics vision representation learning have not been mentioned, such as [1-6]. [1] Real-world robot learning with masked visual pre-training. [2] Masked visual pre-training for motor control. [3] Language-driven representation learning for robotics. [4] An unbiased look at datasets for visuo-motor pre-training. [5] Spatiotemporal Predictive Pre-training for Robotic Motor Control. [6] Learning Manipulation by Predicting Interaction.

The authors make many findings; however, all experiments are conducted solely in the metaworld simulation environment, lacking real-world experiments.

Although local features are more effective than global features, such a comparison is not fair, as the former tends to generate more computational load. I believe that the use of global features in previous robotics vision representation learning works was to create a fair and simple evaluation baseline. In reality, many end-to-end learned generalist policies utilized local visual features, such as RT2-X, OpenVLA and Octo. Therefore, I don't consider this to be a new discovery.

In section 5.1, why when there are 5 demonstrations available for each task, the fine-tuning dataset Dft contains 500 samples (5×10 = 50).

From Figure 8, it can be seen that the proposed self-distillation adaptation method does not yield significant advantages. Furthermore, comparing it with other adaptation methods, in addition to end-to-end fine-tuning, would be better and more convincing, such as designing adapters.

The evaluation benckmark: i think only a single simulator with three views is not enough for getting these conclusions, the authors should introduce another simulator and conduct some real world experiments.

The lack of vision language pretraining: the vision encoder utilized by vision language models (like qwen) should be involved into evaluation, and the authors should testify the performance on unseen language tasks.

The performance gain of self-distillation: comparred with finetuning, the performance gain of self-distillation not enough to demonstrate its efficiency.

1. Experiments Limited to Simulation without Real-World Validation The experiments in this study are conducted exclusively in simulation, with no validation on a physical robot. While the work explores visual representations for robotic models, the lack of real-world testing severely limits the relevance of its findings. Given the substantial sim-to-real gap, conclusions drawn solely from simulated environments are unreliable, as these environments are often overly simplified and do not accurately represent real-world conditions.

2. Limitations of Metaworld as a Benchmark Metaworld is a relatively simple simulation benchmark, even within the realm of simulation-based studies. A significant limitation is its low image resolution, which lacks sufficient detail for robust evaluation. Although the paper does not report image resolution, it is commonly known that Metaworld images are only 112 x 112 pixels. This resolution is inadequate for making meaningful assessments of different visual encoders’ effectiveness.

3. Unreliable Experimental Results Several observations in this paper contradict prior research. For instance, the authors claim that R3M outperforms other pre-trained models (lines 288–293). However, multiple studies, including [2], ACT, and Diffusion Policy, have found that backbones pre-trained with CLIP or ImageNet are more effective for manipulation tasks.

4. Omission of Numerous Related Works The paper overlooks a substantial body of relevant literature, such as [1,2,3,4], which focuses on pre-trained visual representations. This oversight suggests a lack of familiarity with key works in this domain.

5. Incomplete Implementation Details Important details about the implementation are missing. The paper does not specify the number of demonstrations used for training, the number of tasks evaluated, or the performance of methods across varying task difficulties (easy/medium/hard/very hard). Appendix A provides training hyperparameters, but no information on the experimental settings is offered, making it difficult to assess the robustness of the findings.

[1] WHAT MAKES PRE-TRAINED VISUAL REPRESENTATIONS SUCCESSFUL FOR ROBUST MANIPULATION? CoRL 24 [2] On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baseline, ICML 24 [3] Masked visual pre-training for motor control, CoRL [4] Robot Learning with Sensorimotor Pre-training, CoRL