Human-oriented Representation Learning for Robotic Manipulation

We would like to thank the reviewer for the positive feedback and constructive comments. We are glad that the reviewer found our approach innovative.

Q1. [Recent progress in VLM has showcased vast capabilities in reasoning about state changes, object detection, and identifying irreversible states. Could these representations or features from a pre-trained VLM be harnessed to refine R3M, MVP, or EgoVLP, rather than the method proposed?]
We agree that this is a very interesting suggestion! Indeed, the recent advances in VLM have demonstrated a strong representation of learning ability. One potential approach to harnessing the representation from VLM is via distillation where we distill the foundational VLM’s representation extraction ability into smaller models such as R3M. We are very excited about investigating this open problem in our future work!

Q2. [An observation suggests that tasks across various environments and models tend to underperform in toggle on/off tasks. Is there an underlying reason for this trend?]
We follow DETR for the finetuning of our network. In the object detection area, the DETR is more fit for large object detection. We hypothesize the toggle is too small, so our representation is not sensitive to that.

Q3. [The ablation studies indicate that both spatial and temporal tasks, when treated independently, don't yield impressive results. However, their combination surpasses the baseline. Can the authors provide an intuition behind this phenomenon?]
We hypothesize this is attributed to our task fusion decoder, which explicitly builds feedback paths between tasks and helps each task to learn to query useful information from other tasks, thus making tasks interact with and complement each other, ultimately helping the encoder learn more useful representations compared to learning individual task.

Q4. [Figure 5 could benefit from a reduction in font size for the percentage scores. In Figure 6, it might be more efficient to omit repetitive legends across plots.]
We would like to thank the reviewer for this great suggestion! We will incorporate your suggestions in the final paper version.

Q5. [The authors might consider leveraging other task or robot demonstration datasets featuring a third-person perspective with actual robot embodiment beyond Ego4D. A few notable examples include Open X-Embodiment, RoboHive, AR2-D2, etc.]
We would like to thank the reviewer for providing these related papers and datasets! We are excited about leveraging these new datasets to test our representation learning approach in our future work!

Q6. [Have the authors evaluated their method on tasks with longer horizons or those more intricate than mere simple action primitives?]
We would like to thank the reviewer for this great suggestion! In this paper, we consider tasks with relatively short horizons, so the proposed method is fairly compared with baselines. We are excited about evaluating our learned representation for long-horizon tasks in our future work!

Thanks again for your time and effort! For any other questions, please feel free to let us know.

Thank you for your efforts to improve the quality of our papers. We thank you for pointing out your concerns, which we believe have been carefully clarified and addressed, as follows.

Q1. [The decoder outputs information about state changes. Could you define what constitutes a state change, or what the state is in these tasks?]
The state change label comes from the Ego4D hand-object interaction benchmark label, in the dataset, the state change moment is mostly defined by a contact between hand and object.

Q2. [The task fusion decoder uses 10 tokens. Why was this number chosen?]
Yes, TFD uses 10 tokens for 3 tasks. The number of the tokens is determined by the tasks that we considered: the first token is for the OSCC task, the second token is for the PNR task, and the last eight tokens are for the SCOD task, which needs to generate 8 nominated bounding boxes for hands and objects.

Q3. [The point of no return is defined as the task to localize the keyframe with state change in the video clip. Where does the "no-return" terminology come from and what does this mean?]
We apologize for the confusion. The no-return concept is also from the Ego4D dataset [A], it means that you now have to continue with what you are doing and it is too late to stop.

[A] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18995–19012, 2022.

Q4. [Can you clarify what the distributions are in the KL loss function in (4)? These aren't categorical, since I assume there can be multiple state changes in a sequence... How is this loss actually computed?]
Sorry for the confusion. The distribution is state change probability distribution, the decoder outputs a tensor with state change probability in frame dimension. The label is a tensor consisting of 0 and 1, indicating whether the state changes or not in the frame dimension. We treat the two tensors as distributions and measure the KL distance between them.

Q5. [Why are Task 1 and Task 2 different? It seems to me that predicting when a state change occurs already includes the task of predicting if a state change occurred.]
We agree the two tasks have some similarities. However, they play different roles in our framework: the first indicates whether a state changes in the whole video sequence, and the second indicates when the hand contact with the object. The first one focuses on the whole video sequence, while the second focuses more on one moment.

Q6. [The SCOD task is defined as the localization of the hand object positions during the interaction process, which seems to apply continuously, but the terminology state change object detection implies this is only when a state change occurs. Which is it?]
We apologize for the confusion. The SCOD is the task of detecting the object undergoing a state change from the given video clip.

Q7. [I appreciate the novelty in the proposed task decoding approach, but is it an overreach to claim that this paper represents a paradigm shift in representation learning? The idea of learning representations by exploiting human data is relatively well established eg. Sermanet et al. Time-Contrastive Networks: Self-Supervised Learning from Video 2018.]
Thank you for the suggestion. We have removed the overclaimed expression.

Q8. [Table 1. Please provide more detail on this - are success rates averaged over multiple repetitions (if so, how many?) Are there error bars, are these results significant?]
Currently, our experiment in Table 1, is evaluated 20 times and averages the three best successful rate results for every single environment. Also, for the kitchen and metaworld environments, the result is the average from two camera views, for the adroit environment, the result is the average from three camera views. Thus, in a total of 12 environments, there are 460 evaluations.

We do have error bars based on different views as the Table below:

W1.[This is absolutely not the same as having a scalable proxy for human evaluation. To make that claim, one would need to run a study comparing representations fine-tuned with human feedback to those fine-tuned with these hand-object interaction tasks.]
We apologize for the confusion. We did not intend to make readers interpret the proposed method as a way of achieving alignment without human labels. The key idea we aim to convey is that the proposed approach takes inspiration from cognitive science, which posits that humans learn to extract a generalizable behavior representation from perceptual input by learning about a multitude of simpler perceptual skills that are critical for everyday scenarios and formalize this idea through the lens of multi-task learning where each “task” is a perceptual skill. Not only do each of these perceptual skills define a human prior about what is helpful for manipulation tasks, but they are each associated with large and well-labeled video datasets pioneered by the computer vision community.

W2. [It would be valuable to know why one would prefer a fine-tuned video model on these interaction tasks, as opposed to directly predicting them from the outset.]
Good question, directly predicting affordance and visual representation are both major topics in robot learning. Predicting affordance can be more direct and intuitive for policy learning, which is highly correlated with action space. Visual representation learning like R3M, also has the advantage of flexibility, these representations can easily be used in any robot learning tasks with observation space, across the vision-language model, diffusion policy, and so on. We believe predicting from the outset is a good way for many scenes, but visual representation learning is also crucial, even some scenes like adroit pen task without accurate affordance, or other scenes with too many affordances in multi-task, direct predicting the outset may not cover all the affordance accurately, the hidden visual representation can also benefit.

W3. [Move the fanuc manipulation dataset to appendix]
We will move it to the appendix in our final version.

Thanks again for your time and effort! For any other questions, please feel free to let us know.

Thank you for the insightful comments and suggestions.

Q1. [Why would we expect fine-tuning on the task not to improve the performance on the task? (In relation to section 4.5)]
Sorry for the confusion. For Section 4.5 and Table 4, the first two rows' results are from ego4d pretraining and finetuning on specific results, and our results also pretraining from ego4d, but finetuning on two tasks at the same time with task fusion decoder. The results can, in some sense, show our tasks in the task fusion decoder have information exchange, which can benefit each one.

Q2. [What about ablating each loss type separately? Please provide ablations for each individual task head]
Good question. We have added experiments by each task head, by retraining the model with three single tasks and retraining the policy for different models. We observed that in most situations, adding all three tasks gets the best result.

Q3.[“Also, different vision tasks should have information interaction, for the human-like synesthesia.” This does not really add anything. What is human-like synesthesia?]
Human-like synesthesia is the way humans perceive spatial, temporal, and other information about the world simultaneously. In a task trajectory, humans gather information at the same time, like object localization, time to happen contact.

Q4. [For the real world experiments: how many test interactions were performed?]
We perform 10 interactions for every single model in every single task following r3m evaluation, so in every task, there are 3 baseline models and 3 our models, a total of 240 interactions for Fig 5.

Q5. [Can the authors commit to open-sourcing the real-world dataset?]
Sure. We open-source the dataset at data_link, code at code_link.

Q6. [This is analogous to humans who, in specific environments, focus solely on one particular perception.” Please remove this. Humans do not, in specific environments, focus solely on one particular perception. If they do, I would like to see the citation backing this up.]
Thank you for your advice. We have removed this in our paper.

Q7. [Learned sigmas: what are they after training? When you say they are learned, is that per scene, or just differentiable but global?]
We retrained a model to see the change of the sigma, and added one curve figure to Appendix A.5. The sigma for OSCC is about -1.3e-5, for PNR is about -3e-5, for SCOD is about 0.01, while the loss value of SCOD is about 1e3 times over PNR and OSCC. Sigma is just differentiable but global.

W2. [ The ablation study merely looks at the importance of the "time-related" and "spatial-related" tasks - and leaves the reader guessing which exactly these are, and what exactly the numbers in Table 3 mean. Even so, the results are not overly convincing, as one of the five environments performs worse under the TFD, and a second does not benefit.]
We apologize for the confusion. The time-related tasks are the OSCC and PNR tasks, which contain temporal information, while the spatial-related task is the SCOD task, which contains spatial information. We also make a clearer ablation study by separating the “time-related” tasks into the oscc task and pnr task, here we provide ablation studies with three different tasks in TFD.

The reason why some results are even worse in TFD is that in some environments maybe the three tasks we added are not the best combination, only one task can lead the policy learning in this simulation to the best result. Our TFD is a flexible multi-task structure, it can flexibly leverage different task combinations for the representation adjustment, but as seen in the table below, in most situations, adding all the tasks is the best combination.

W3. [The TFD comprises a lot of computational machinery and K stacked blocks (where K is left undocumented) and involves many design choices. In contrast, it would be interesting to know the impact of the two attention mechanisms (as these implement the core motivation of this work) and how many of these blocks are required.]
As shown in the caption of Figure 2, the design for self-attention in each layer is for the interaction of each task, we hypothesize each task should have information change, for cross-attention, it is a connection between vision encoder backbone and TFD, which can decode vision representation from encoder into multi-task with human labels. In our original paper, the choice for the TFD is 2 layers, each layer has one self-attention and one cross-attention. And also, we add more experiments about different choices of K, as shown in the table below.

After searching the parameter K, we find that the k doesn’t influence the results much, and K=2 with our original setting, maintains a good result in most environments.

W4. [it would be interesting to know the impact of the two attention mechanisms (as these implement the core motivation of this work)]
As the caption for Fig. 2, the self-attention is designed mostly for multi-task tokens information exchange, we hypothesize that multi-task can benefit each other during training, and Table 4 has proven it by a better result on joint task finetuning with TFD. Cross-attention is a bridge from visual representation to TFD, which can decode different tasks’ information for the visual representation during finetuning for the last 1/3 layers of the visual encoder.

W5. [Unclear expression and figures]
Thanks for pointing it out! We have deleted the strong words like “paradigm shift” and “consistently”. Fig. 2 is more important for understanding, the Fig. 9 and Fig. 10 are just specific TFD designs for different visual backbones, resnet, vit, and video transformers, which cover the main-trend computer vision backbone. We have a zoom-in for more clarity on the three general tasks. We also change the Fig. 2, making it more readable.

We hope that our response has addressed your concerns, and turns your assessment to the positive side. If you have any questions, please feel free to let us know during the rebuttal window. We appreciate your suggestions and comments!

We would like to thank the reviewer for taking the time to review our paper and provide insightful suggestions.

Q1. [If it is true that the TFD has no role after training ("fine-tuning"), I'm not convinced that this is the best possible setup. The idea is to force the latent representation formed by the backbone encoder to generalize across tasks, but it is not clear how much of this generalization power lies within the considerable computational machinery of the TFD itself and is thus no longer available after fine-tuning. I'd like the authors to comment on this.]
Good questions. We agree that TFD has no role after training, as it backward the gradients to the last few layers of the encoder to learn good representations (we froze most of the front layers of the encoder). Also, it is a common phenomenon to freeze the vision encoder in the visual-motor control community [A, B] for robotics policy learning, it can avoid finetuning a large vision model during policy training. Therefore, finding a strong enough visual representation is a core problem for this area. Our main contribution to TFD is to induce human perception style by multi-task structure for the existing vision backbones, and forming a new representation to test on policy learning. Thus, during finetuning, the last 1/3 layers of the original visual backbone are unfrozen, jointly training with TFD. The generalization is kept across tasks, as shown in the benchmark results in Fig.7. We also included a clear backbone setting in Appendix Section A.4.

[A]Suraj Nair, Aravind Rajeswaran, Vikash Kumar, Chelsea Finn, and Abhinav Gupta. R3m: A universal visual representation for robot manipulation. arXiv preprint arXiv:2203.12601, 2022.

[B]Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.06173, 2022.

Q2. [The wording in Sec. 3.2 suggests there is exactly one state change frame. Why can't there be more than one?]
In this paper, we used the hand and object interaction of the dataset in the Ego4D data set, which only has one state change frame label. However, our pipeline is not constrained by the number of state change frames: if there is more than one state change frame, we can segment the whole video sequences with several parts or with more state change frames, and feed them into our network.

Q3. [My two main concerns are lack of clarity and inconclusive results. At least the former can perhaps be fixed; I can perhaps be convinced to upgrade my ratings.]
We would like to thank the reviewer for the feedback! We have modified the method section 3.2 to make it clearer and added one additional figure in Appendix A.4 to further explain the training and testing setting of our paper. Please see our updated paper, thanks!

W1. [The number one improvement this paper should undergo is to clarify (in the main body) how exactly the backbone is connected to the TFD, how exactly the "fine-tuning" is done, and how the benchmark networks are set up, trained, and tested.]
Thank the reviewer for the feedback! The backbone is connected to the TFD by spatial and temporal features, which are extracted from the visual encoder, and these features are used for cross-attention with different task embedding in TFD. For the “fine-tuning,” we unfreeze the last 1/3 layers for visual encoders, like gulp, we train the last four layers’ video transformer blocks of a total of 12 layers and train the TFD from scratch. For policy learning, we follow the traditional visual-motor control method, freezing the vision encoder we have learned and learning a policy learning network by a total of 20000 steps, every 1000 steps we evaluate the policy results.