Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets

w3: Unnecessary discussion on how to use the human video datasets. The authors spend a considerable amount of efforts to explain the policy learning from human videos. But in the end, the conclusion is to not use human data. Human action data is abundant and contains rich and diverse behavior. If the authors' proposal is to not use them and use robot demonstrations, which in fact is the standard robot learning setup. I would prefer the paper to focus on the studied setting.

We would like to clarify that our contribution extends beyond simply training a pretrained model on robotic datasets, which would indeed only require a discussion of robot data. One of the key contributions of our work is proposing a manipulation-centric metric for evaluating all visual encoders, as other reviewers specifically recognize this contribution. Through our discussion of models trained on human data, we analyze how baseline models insufficiently capture manipulation-related features. This analysis demonstrates that training on robot data may be more effective in enhancing manipulation centricity. Such analysis is essential to establish our key insight and justify our approach to answer how to pretrain on robotic data more effectively for manipulation tasks, rather than simply assuming this choice. We hope this clarification helps better convey our motivation and contribution.

w4: Needs a more comprehensive ablation study. RPM is pre-trained with the DROID dataset, a large and relatively clean robot dataset, while many baselines that RPM compared against are not trained with DROID.

Thank Reviewer vJK6 for this important suggestion about ablation studies. To address this concern, while R3M's performance on DROID was already included in our paper, we have further expanded our comparisons by including results from MAE (used in MVP and VC-1) pretrained on DROID. Our experiments show that MAE achieves limited performance when trained on DROID due to reduced image quantity and lack of robotics-specific design. This further highlights the importance of designing representation learning methods specifically for robotics datasets.

Regarding the HRP baseline, it requires hand pose and object labels for dexterous manipulation, which are not available in DROID, making direct training on DROID infeasible. We have included these additional discussions and detailed results in Appendix A.7 for a more comprehensive comparison.

We sincerely appreciate Reviewer vJK6's positive feedback regarding both our paper's presentation and our conceptual contributions. Below, we provide detailed responses to address the reviewer's concerns:

w1: The technical contribution is limited. The authors spent 4 pages between the introduction and experiment section to discuss their method. But, in fact, the only new technical point that the authors proposed is equation (1), which uses InfoNCE loss to bring visual features and proprio state/aciton closer. This looks hand-wavy.

We respectfully disagree with the assessment of limited technical contribution, and would like to highlight two key technical aspects of our work:

• Following the introduction, we present a detailed analysis of how the manipulation-centricity of pretrained encoders impacts downstream task performance, introducing a clear and practical metric. This novel contribution has been specifically recognized by Reviewers wcpD, 3aRj, and cSN9. We emphasize that we are the first to systematically examine and quantify this critical problem in robotic manipulation.

• Beyond the technical contribution of state-action alignment mentioned by the reviewer, in our knowledge, we are the first to incorporate action prediction loss in visual encoder pretraining for manipulation tasks. We appreciate any relevant prior work Reviewer vJK6 may suggest that we have missed, as this would help us better contextualize our contribution and strengthen our related work discussion.

Thank you for your feedback. We believe these technical contributions constitute significant components of our work and hope this clarification addresses your concerns.

w1: Even this innovation raises questions, as the proprio state/action only describe the robot trajectory but not the visual scenes. This is very likely to make the model overfit to the learned scenarios.

Our approach intentionally leverages robot trajectory information to capture effective visual features. As emphasized in our paper, our training utilizes a highly diverse dataset comprising 36,000 trajectories across more than 500 distinct scenes and 80 different tasks. This extensive data diversity substantially reduces the risk of overfitting. Moreover, our empirical evaluation demonstrates robust generalization across various robot embodiments (beyond the Franka robot used in the DROID dataset) and diverse manipulation tasks in both simulations and real world, providing strong evidence that our pretrained encoder successfully avoids overfitting issues.

w2: The evaluation is questionable. RPM is evaluated on 20 tasks, 10 from MetaWorld, 4 from DexArt, 3 from RoboCasa, 3 from RoboMimic. However, these benchmarks combined include hundreds of tasks. Why just select these 20 particular tasks? Moreover, the "Can" task in robomimic is mentioned as challenging in this paper, but it is just a simple pick-and-place task of a moderate-size can.

We would like to clarify our task selection and address the concerns about task difficulty:

• For MetaWorld, we selected the five most difficult tasks according to the benchmark's difficulty ratings, plus five tasks evaluated in R3M.

• For DexArt, we evaluated all four available tasks in the benchmark.

• The limited evaluation on RoboCasa (3 tasks) was due to practical constraints: very slow inference speed in their environment and incomplete demonstration data.

• For RoboMimic, most tasks are challenging, with near-zero performance across existing algorithms. We selected the most commonly evaluated tasks in the literature.

In summary, our evaluation suite is more comprehensive than all baselines, spanning multiple major manipulation benchmarks and real-world tasks. ​​While the "Can" task may appear simple as a pick-and-place operation, its difficulty is evidenced by the poor performance of most methods. As widely discussed in manipulation works [1][2], pick-and-place tasks are far from trivial in robotics, because task difficulty isn't solely determined by motion complexity but also by environmental factors and perception challenges.

[1] Kim et al. "OpenVLA: An Open-Source Vision-Language-Action Model"

[2] Octo Model Team et al. "Octo: An Open-Source Generalist Robot Policy"

Dear Reviewer vJK6,

As the discussion period draws to a close soon, we are reaching out to request your feedback on our response kindly.

We are delighted to have received positive feedback from the other three reviewers and are very eager to ensure that our response has adequately addressed your concerns as well.

We believe that the detailed clarification of our contributions, along with the additional experiments in rebuttal could solve your concerns. Your feedback is crucial for us to further enhance the quality of our paper. We eagerly await your response!

Warm Regards,

Paper 793 Authors

We extend our sincere thanks for your perceptive review and acknowledgment of our technical novelty, empirical contributions and presentation. We have carefully considered and responded to each of your points below.

w1 & q1: It wasn't clear if they finetune all parameters on a given task (in the simulation and real-world policy learning experiments). In general, this may be relevant as policy performance can improve with finetuning of the entire architecture. I may have missed this in the paper but it would be good to see if the pretraining results include finetuning of the overall architecture on the robot manipulation task.

Thank you for this important question. We maintain a frozen visual encoder during visuomotor control, aligning with baseline evaluation protocols. This approach offers several advantages: it ensures fair comparisons with existing methods, reduces computational costs during policy training, and effectively tests the generalization capacity of our pre-trained features. We have also conducted additional experiments with full model finetuning for all methods, where our approach maintains its superior performance advantage while showing significant improvements over the frozen encoder results reported in the manuscript. We have added these results in Appendix A.11.

w2: It seems like the manipulation centricity is a good proxy for rudimentary manipulation tasks but it doesn't necessarily seem to be a metric that will generalise to more complex settings. The metric also relies on annotations or knowledge of the task relevant features within the image.

Thank Reviewer cSN9 for this insightful suggestion. Our evaluation suite covers a broad range of complexity, as detailed in Section 2. The annotation process is quite efficient since each task needs to be annotated only once. Leveraging SAM2, we can segment one video in approximately 1 minute, completing all required annotations in under an hour. To support the research community, we plan to release our annotated dataset for efficient pretrained model comparisons. While we acknowledge the limitations for long-horizon tasks with complex sequential planning, utilizing language instructions with transformers for metric annotation could be a promising future direction. We have incorporated this discussion in the limitation and future work section.

w3: The pretraining dynamics alignment term incorporates chunks of state-action pairs with a history of 3 being cited as optimal. When it comes to highly dynamic tasks it is not clear how the alignment between individual images and such a history will necessarily be beneficial. I question whether the improvements observed are optimal relative to other methods of incorporating dynamics information. If the dynamics varied quite a lot between demonstrations on the same robot platform I question how this will effect performance. Clearly this term is working as demonstrated in the paper but I do have reservations over its usefulness and whether it is the optimal approach to incorporating dynamics information.

It is a good question. We would like to clarify the dynamics alignment mechanism: The center of each state-image dynamics chunk corresponds to the same state as the static image, and with our minimum chunk length design incorporating intervening actions, this creates meaningful positive pairs for alignment. Importantly, the inserted actions serve as crucial bridges between potentially disparate states, enabling our approach to capture universal manipulation-related features through this alignment. While several approaches exist for pre-training robotic features, our work most closely relates to Ilija et al. [1], who explored dynamics through MAE. Our method offers a distinct and efficient approach to robotic dynamics incorporation. We acknowledge that exploring optimal dynamics integration for highly dynamic tasks is an interesting direction for future work. We have incorporated this discussion in the limitation and future direction section.

[1] Radosavovic et al. "Robot Learning with Sensorimotor Pre-training"

We hope our explanations and additional experiments have resolved your concerns. If you have any further questions, we look forward to discussing them. Thank you again for your insightful feedback.

Dear Reviewer cSN9,

Thank you so much for your time and effort in reviewing our work and providing insightful feedback. As the review discussion phase concludes in two days, we kindly request you review our response and confirm if it addresses all your concerns.

We welcome any further discussions and value the opportunity for continued improvement of our work.

Warm regards,

Paper 793 Authors

We sincerely thank Reviewer 3aRj for providing detailed and constructive feedback. Our responses to all of the concerns are addressed below. We look forward to further discussion if any points would benefit from additional clarification.

w1: 1DROID is chosen as the robot dataset of choice instead of other larger dataset in OXE.

DROID stands as one of the largest robotic datasets in Open-X-Embodiment, encompassing more than 500 distinct scenes and 80 different tasks. Due to computational resource constraints (RPM training utilized only 50 hours on a single RTX 3090), we focused our experiments on the DROID dataset. Notably, our method achieved strong generalization results across diverse robot embodiments beyond the Franka robot data provided in DROID. While RPM can certainly be trained on other robotic datasets, our current results demonstrate its effectiveness within our computational constraints. Additionally, using the full OXE dataset would require addressing differences in action spaces across embodiments, which presents an interesting direction for future research. We have discussed it in the limitation and future direction section.

w2: RPM is a ResNet-based model instead of directly using / comparing to pretrained models like SAM2, etc.

q3: R3M has been shown to be a poor visual representation model for robotic tasks, as compared to visual datasets [1] and pretraining image distribution matters. Why none of the baselines involve vision only representation backbones for comparison?

Thank you for this insightful suggestion. We have expanded our baseline comparisons in Appendix A.7 including SAM2 and other pretrained models. The results below demonstrate that RPM consistently outperforms all baselines, including both robotics-specific and vision-only approaches:

w3: The "manipulation centricity" does not seem to improve at similar scale across different simulators, with respect to success rate.

This is an insightful question. While manipulation centricity correlates with performance within each simulator, direct comparisons of these values across different simulators are inappropriate due to varying visual characteristics. For example, RoboCasa's high-resolution environments result in proportionally smaller target annotations, which naturally affect the absolute scale of manipulation centricity measurements.

Despite these inherent differences in scales, our aggregated results demonstrate a consistent trend: higher manipulation centricity generally corresponds to better performance within each simulator's context. This internal consistency validates the relationship between visual attention and task success.

w4: A concise definition of manipulation centricity and how it is calculated should be present in the main paper, rather than existing only in appendix.

This is a good suggestion. We have expanded Section 3 to include detailed definitions and formulas for manipulation centricity, with Figure 3 illustrating the calculation process. The metric requires two inputs per representation for each task: a Grad-CAM video and an annotated segmentation video. Since the segmentation video is task-specific and shared across all representations, it only needs to be annotated once. We have provided the technical details for generating these inputs in Appendix B.

We greatly appreciate the reviewer's thoughtful feedback. We have addressed each point and comment raised in the review below. Should any aspects require further clarification, we welcome continued discussion.

w1: There could be more empirical analysis and intuition regarding how each loss affects manipulation centricity. Authors could explain intuitively how each loss contributes to better MC and visualize the gradcam of an encoder trained with each loss

We have expanded the qualitative and quantitive analysis in Appendix A.9 to illustrate how each loss influences manipulation centricity. Our analysis reveals that each loss serves a distinct purpose: The dynamics alignment loss guides the encoder to capture rich manipulation-related features by focusing on the most dynamic elements of the scene. The action prediction loss directly enhances the model's action prediction capabilities by leveraging the pre-trained features. The time contrastive loss facilitates the learning of temporal visual relationships. Our experimental results confirm that all three losses contribute meaningfully to both improved manipulation centricity and higher downstream task success rates.

Moreover, we would like to emphasize two other key conclusions: (1) the dynamics alignment loss provides the most substantial benefit to representation learning, as it significantly enhances the encoder’s focus on manipulation-centric features; and (2) the strong positive correlation between manipulation-centricity and downstream task success rates still hold, strengthening the key contribution of this paper. Thanks for the good suggestion!

w2: The paper lacks comparison to more state-of-the-art baselines, such as VIP/LIV, or object-centric pre-trained representations such as POCR/HODOR.

We appreciate these constructive suggestions. Regarding baseline selection, we note that VC-1 (NeurIPS 2023) has already demonstrated its superiority over VIP, while both VC-1 and HRP (RSS 2024) represent the current state-of-the-art in robotic representation learning. While HODOR addresses tasks outside the robotics domain and POCR is not open-sourced, we have added discussions of these relevant methods in the related work section in the manuscript. Additionally, we have expanded our evaluation to include comparisons with LIV, ImageNet, SAM2 and MAE trained on DROID on Robomimic (Appendix A.7). The results below demonstrate that our method significantly outperforms all baselines.

w3: Have the authors considered directly using the masks as a form of supervision for training the representation?

This is a good question. While utilizing a dataset with segmentation masks for manipulation-related regions would be an interesting direction to explore, it is currently impractical due to the substantial annotation costs associated with labeling our large-scale robotic dataset. Moreover, segmentation-based supervision alone may not sufficiently capture the complex state-action dynamics that are critical for learning effective and generalizable representations, as demonstrated by our approach. We have incorporated this discussion in the limitation and future direction section.

w4: The paper could benefit from more extensive real robot experiments

We completely agree that additional real-world evaluations, specifically extending to more complex tasks with various objects such as Fold Clothes and Pick Place AAA Battery, would further strengthen our work. We have incorporated this discussion in the limitation and future direction section. We are currently collecting additional expert demonstrations for these extended tasks, as open-source demonstrations are insufficient for imitation learning. While resource and time constraints limit our ability to show more real-world results during the discussion period, we plan to expand these evaluations in the camera-ready version. Meanwhile, we emphasize that our current three tasks already demonstrate considerable complexity across diverse objects and manipulation skills, effectively validating our method's capabilities.

q1: Why is it still necessary to attend to the manipulator region when there is already proprioception input?

This is an insightful question. Visual information of the manipulator region provides richer contextual cues that proprioceptive input alone cannot capture, including the relative spatial relationships between the end effector and objects, as well as the manipulator's approximate pose in the world frame. By explicitly attending to the manipulator during pre-training, our encoder develops an enhanced capacity to extract critical manipulation-relevant features, making this approach both distinctive and advantageous.

We have enhanced our manuscript in several key aspects:

Main Text:

• Added detailed definitions and formulas for manipulation centricity (Section 3)
• Enhanced related work discussion of recent representation methods (Section 6)
• Added overview figure illustrating our methodology and metric (Figure 1)
• Clarified cross-simulator evaluation protocols (Figure 2)

Appendix:

• Expanded comparisons with 4 new baselines(A.7)
• Analysis of visual generalization under perturbations (A.8)
• Comprehensive ablation studies on loss components (A.9)
• Study of camera parameter effects on manipulation centricity (A.10)
• Additional finetuning experiments and results (A.11)
• Quantitative analysis of manipulation centricity for Table 1 tasks (A.12)
• Discussion of limitation and future directions (Section LIMITATION AND FUTURE DIRECTIONS)

These updates are temporarily highlighted in "blue" for your convenience to check. We strongly believe that our paper makes a great contribution to the ICLR robotics community, particularly because reviewers’ constructive comments enhanced the manuscript.

We sincerely thank the reviewers and the AC for their time and thoughtful feedback on our paper during the discussion period. We hope that our responses have effectively addressed all the questions and concerns, and eagerly await further discussion and feedback.