$\texttt{PREMIER-TACO}$ is a Few-Shot Policy Learner: Pretraining Multitask Representation via Temporal Action-Driven Contrastive Loss

Thanks Reviewer YiPn for the valuable review of our paper. We appreciate the questions you raised and are committed to delivering a comprehensive response to address the issues.

1. The introduced Premier-TACO shows incremental contribution over the baseline TACO. Specifically, the only difference is the contrastive loss adopted by TACO and the triplet loss adopted by Premier-TACO. The technique of negative sampling is quite common in the area of metric learning and widely adopted in applications other than robotics, e.g., face recognition. Moreover, the empirical comparison with the baseline TACO is very limited in this paper. The effectiveness of simply adding negative sampling is questionable.

In response to your suggestion, we have also included additional discussions on existing negative sampling strategies for contrastive learning related to Premier-TACO in Appendix A. We have also updated Table 1 and 2 with TACO as a separate column and updated Figure 9 with a comparison of Premier-TACO vs. TACO in more detail.

First, Premier-TACO is tailored toward a fundamentally different setting than TACO. While TACO is designed for task-specific representation learning in single-task offline and online reinforcement learning, Premier-TACO aims for a broader representation pretraining objective, which is to learn a universal/foundational visual representation from multitask offline datasets. To scale up to multitask offline pretraining, we propose the simple yet highly effective negative sampling strategy in Premier-TACO, which is the key to both scalability and great empirical performance. Without this modification, TACO is not competitive, as shown in the updated Table 1 and 2 as well as Figure 9.

Next, we believe that the general question of multitask representation learning as a foundational model for efficient downstream policy learning itself holds substantial significance for both the machine learning and robotics communities. We acknowledge that our approach bears some resemblance to existing hard negative sampling strategies in contrastive learning and metric learning literature. However, few of them have been successfully applied to the sequential decision making setting, which has to deal with unique challenges such as data distribution shifts between data collection policy and learned policy, task heterogeneity, and pretraining data qualities, as discussed in the introduction.

Furthermore, in our paper, we also discuss the additional aspects of generalization to unseen embodiments and camera views, pretraining dataset quality, and, importantly, how to finetune/adapt existing pre-trained visual encoders for control. These aspects are unique and also crucial to pretraining for sequential decision-making tasks and are under-explored in the existing literature. Many existing works pretrain only on vision datasets and then apply the pretrained vision foundational models directly for downstream policy learning. Thus they often overlook control-relevant considerations and thus suffer from a domain gap between pretraining datasets and downstream visuomotor tasks. Addressing this gap, Premier-TACO proposes a nice solution that could effectively adapt large pre-trained visual encoders for control purposes, as illustrated in Figure 8. Therefore, we firmly believe that this adds significant value to our contribution.

2. The authors repeated several times the infeasibility of adopting visual foundation models (such as those trained on ImageNet or Ego4D) in sequential decision-making tasks. However, it is ungrounded. The authors should evaluate these models (e.g., CLIP, DINOv2, EgoVLP, etc.) on the same benchmark for comparison.

We would like to kindly point out that in the main experimental results of Table 1 and Table 2, we have already compared with the best pretrained visual encoders (PVRs) / vision foundational models across 4 existing works on applying PVRs for control tasks. (We present the best PVR results in Table 1, and the results of all four PVRs are in Table 3 of Appendix B.2.) With a significant performance gain across 18 unseen tasks and 2 different domains, we show that existing works on adapting visual foundational models directly to downstream control tasks with large domain gaps without domain-specific finetuning are suboptimal.

Dear Reviewer YiPn,

Since today is the last day of the author-reviewer discussion period, we kindly ask you to review our previous response and revised manuscript and let us know if there are any other questions. We are happy to answer any additional questions you may have. Thank you.

Thank you for the valuable review of our paper. We appreciate the questions you raised and are committed to delivering a comprehensive response to address the issues.

1. It would be better to add an ablation of using the negative sample strategy in TACO on sequential decision-making tasks. It is important to show the effectiveness of the proposed negative sample selection strategy.

Thanks for your constructive feedback. we have made updates to Figure 9, presenting a comparison of the averaged performance across 10 unseen tasks from Deepmind Control Suite between Premier-TACO (with different batch sizes) and TACO. Additionally, we have revised Table 1 and Table 2 by incorporating an additional column for TACO in the baselines, facilitating a direct comparison with Premier-TACO, as per your suggestion. These modifications to Table 1 and 2 as well as the updated Figure 9 effectively illustrate the efficacy of the novel negative sample selection strategy in Premier-TACO, where Premier-TACO outperforms TACO with at least 40% across all batch sizes. We believe that these additions should address your concern on the empirical comparison against TACO baseline and we welcome any further discussion or suggestions from the reviewer.

2. Premier-TACO uses additional negative samples selected from a temporal window. Compared with TACO, is the batch size doubled? If it is true, what is the result when decreasing the batch size of Premier-TACO to 1/2N compared with TACO with N?

Thank you for your question regarding the batch size used in Premier-TACO. To clarify, the batch size in Premier-TACO is not doubled compared to TACO. As outlined on page 9, line 2 of our manuscript, and demonstrated in Figure 9, we compare Premier-TACO and TACO using an fixed batch size of 4096. This batch size is consistent with that used in our main experimental results (Tables 1 and 2).

To address your question further, we have updated Figure 9 to include comparisons of Premier-TACO and TACO both at different batch sizes. This additional analysis should provide a clearer understanding of how batch size variations impact the performance of both methods.

Finally, in the update to Table 1, we have included a column for TACO where it is pretrained with the same batch size of 4096, aligning it with other contrastive learning baselines. This ensures that the comparisons made are fair and consistent across different methods.

3. Does the selection number influence the performance? How about select more than one samples as negatives?

Thank you for this insightful question. Indeed, we have considered this in the original design of our learning objective. In the table below, we compare the performance when selecting a single negative sample randomly from a size W window against when using all samples within this window across 10 unseen Deepmind Control Suite tasks.

Our findings indicate that using all samples from the size W window does not significantly enhance performance compared to Premier-TACO. Moreover, this approach considerably increases the computational overhead. Given these results, we chose a more computationally efficient strategy of sampling a single negative example from the size W window. We appreciate this question that you raised, and we have incorporated it into Appendix D of the updated manuscript.

Dear Reviewer 7fxb,

Since today is the last day of the author-reviewer discussion period, we kindly ask you to review our previous response and revised manuscript and let us know if there are any other questions. We are happy to answer any additional questions you may have. Thank you.

3. In the ablation study, in order to show the effectiveness of the proposed negative example sampling strategy, the authors should compare Premier-TACO with a baseline without using a negative example sampling strategy. The related experimental results should be added.

Thank you for your suggestion on the additional empirical comparison. In our evaluation now, we have already compared with SPR, which applies a BYOL-based objective that does not leverage negative examples. Additionally, we have also compared with TACO, which uses a naive negative example sampling strategy and treats every other data point in the batch as negative examples. Furthermore, Figure 9 of our ablation study is now updated to include a more fine-grained comparison between Premier-TACO and TACO with different batch sizes. With a significant performance gain shown by Premier-TACO, the results should be sufficient to demonstrate the effectiveness of the negative sampling strategy proposed in Premier-TACO. If there are specific additional baselines that don't use negative sampling that you believe are important to compare against, please let us know.

Thanks Reviewer AXKc for the valuable review of our paper. We appreciate the questions you raised and are committed to delivering a comprehensive response to address the issues.

1. The novelty is somewhat limited. The proposed method is built on the temporal action constrastive learning (TACO) objective [1]. The overall framework is similar to [1]. The authors additionally employ a negative example sampling strategy. But the negative sampling has been widely used in constrastive learning [2][3]. Considering the above factors, I think that the innovation of the method is limited.

In response to your suggestion, we have also included additional discussions on existing negative sampling strategies for contrastive learning related to Premier-TACO in the Appendix A. Here, we would like to highlight several key contributions of our work that distinguish our method from TACO [1] and the existing works in hard negative mining [2,3].

First, Premier-TACO is tailored toward a fundamentally different setting than TACO. While TACO is designed for task-specific representation learning in single-task offline and online reinforcement learning, Premier-TACO aims for a broader representation pretraining objective, which is to learn a universal/foundational visual representation from multitask offline datasets. To scale up to multitask offline pretraining, we propose the simple yet highly effective negative sampling strategy in Premier-TACO, which is the key to both scalability and great empirical performance.

Next, we believe that the general question of multitask representation learning as a foundational model for efficient downstream policy learning itself is holds substantial significance for both the machine learning and robotics communities. In this paper, we show that with the negative example strategy employed by Premier-TACO, it achieves an impressive performance gain compared with a comprehensive set of baseline pretraining methods across two domains. We acknowledge that our approach bears some resemblance to existing hard negative sampling strategies in contrastive learning literature. However, few of them have been successfully applied to the sequential decision making setting, which has to deal with unique challenges such as data distribution shifts between data collection policy and learned policy, task heterogeneity, and pretraining data qualities, as discussed in the introduction.

Furthermore, in our paper, we also discuss the additional aspects of generalization to unseen embodiments and camera views, pretraining dataset quality, and, importantly, how to finetune/adapt existing pre-trained visual encoders for control. These aspects are unique and also crucial to pretraining for sequential decision-making tasks and are under-explored in the existing literature. Many existing works pretrain only on vision datasets and then apply the pretrained vision foundational models directly for downstream policy learning. Thus they often overlook control-relevant considerations and thus suffer from a domain gap between pretraining datasets and downstream visuomotor tasks. Addressing this gap, Premier-TACO proposes a nice solution that could effectively adapt large pre-trained visual encoders for control purposes, as illustrated in Figure 8. Therefore, we firmly believe that this adds significant value to our contribution.

2. The detailed experimental comparisons and discussions with TACO are missed.

Thank you for your feedback regarding the comparison with the baseline TACO. In Figure 9 of the original manuscript, we have already presented a comparison of the averaged performance across 10 unseen tasks from the Deepmind Control Suite between Premier-TACO (with different batch sizes) and TACO. We appreciate your suggestion to highlight this comparison in the paper, and so we have updated Table 1 accordingly. This revision includes an additional column for TACO in the baselines, enhancing the clarity of our empirical comparison. For the results of MetaWorld in Table 2, the experiments are still running. We will incorporate the results into our manuscript as soon as they are available.

====== Update: results of TACO for MetaWorld (Table 2) are now updated ===========

Dear Reviewer AXkc,

In our earlier response and revised manuscript, we have conducted additional experiments and provided detailed clarifications based on your questions and concerns. As we are ending the stage of the author-reviewer discussion soon, we kindly ask you to review our revised paper and our response and consider adjusting the scores if our response has addressed all your concerns. Otherwise, please let us know if there are any other questions. We would be more than happy to answer any further questions.

Dear Reviewer AXkc,

Since today is the last day of the author-reviewer discussion period, we kindly ask you to review our previous response and revised manuscript and let us know if there are any other questions. We are happy to answer any additional questions you may have. Thank you.