TVTSv2: Learning Out-of-the-box Spatiotemporal Visual Representations at Scale

W1: The paper is a bit disorganized.

W1.1: It would read better if the approach description was in one place.

A: Thanks for the advice. We have reorganized the paragraphs and put the approach description together in the revised version.

W1.2: If the empirical degradation study calls for modification of the training objective, that should be spelled out more explicitly.

A: The empirical degradation study is disentangled from the training objective, and it corresponds to the partially frozen pre-training strategy. Sorry for the confusion, we have reorganized the paragraphs to make readers easier to follow.

W2: It is not clear what the ground truth attention mask is. Could it be that in the video clips the objects/actions of interest were the only moving parts in the scene causing the attention grab?

A: That's an insightful query. Indeed, selectively masking objects or actions that are semantically linked could potentially enhance model performance. However, creating accurate ground truth masks for this purpose is a labor-intensive task, especially considering the scale of pre-training that involves millions of videos. Therefore, we've opted for a more feasible approach: employing a random masking strategy that does not depend on ground truth data. Despite its simplicity, this method proves quite effective due to the vast amount of training data we use. Our models demonstrate excellent out-of-the-box performance, highlighting the capability of this straightforward technique to learn robust representations (Appendix D Table 18).

W3: The joint attention module is not described clearly. Is it the same as in CLIP?

A: No, we adopt the divided space-time attention instead of the joint attention. An illustration of the detailed architecture is provided in Appendix C Figure 3. For the intra-frame tokens, i.e., spatial-related tokens, we add the same spatial positional embeddings, and for tokens at the same position across different frames, i.e., temporal-related tokens, we add the same temporal positional tokens. For each token, it first attends to the temporal-related tokens, then attends to the spatial-related tokens. Note that the [CLS] token is attended in both temporal and spatial self-attention.

In addition, we also compared the performance between CLIP-style joint attention and the divided space-time attention. Please refer to Appendix D Table 19 for details.

W4&Q4: More extensive evaluation of fine-tuning performance on diverse downstream tasks would be desirable for supporting the "out-of-the-box" claim.

A: Thank you for your insightful feedback. In our paper, the phrase "out-of-the-box" refers to creating versatile video representations that can be immediately applied to various downstream tasks without the need for further fine-tuning. Our goal is to offer such foundational models to the community, potentially accelerating its progress by eliminating the need for costly pre-training and fine-tuning.

We value your advice and recognize that a strong out-of-the-box representation should yield good fine-tuning performance. To this end, we conducted basic fine-tuning of our pre-trained models on DiDeMo and LSMDC datasets, deliberately avoiding complex techniques like optimal parameter searching. The findings, presented in Appendix D Table 15, indicate that our straightforward approach does enhance fine-tuning performance to a certain extent. Furthermore, we have included linear classification results where a classifier was trained on top of the frozen model backbone. Our models outperform both self-supervised and language-guided models in these tests, as detailed in Appendix D Table 13. This further demonstrates the effectiveness and versatility of our proposed training methodology.

W1: Concern about technical contribution. The degradation-free training strategy is more like an engineering trick.

A: Thanks, we are willing to highlight the technical merits of our work, both from experimental results and the underlying motivation.

(i) Experimentally, to demonstrate that our degradation-free training strategy goes beyond a mere engineering workaround, we introduced an alternative baseline. This baseline involves a ViT-B/32 model equipped with adapters [1] (M$_{\text{adapter}}$) inserted between layers of the text encoder, where only the adapter-introduced parameters are adjustable during training. The retrieval results on MSR-VTT, detailed in Appendix D Table 16, show that this approach is inferior to our method. This reinforces our assertion that end-to-end tuning can lead to style overfitting, even when the text encoder remains unchanged, proving that our partially frozen strategy is not a simplistic fix but a considered solution.

(ii) From a motivational standpoint, scalability remains a significant and challenging research area, as recognized by Reviewer hjUh. This field is marked by several pioneering works featured in leading conferences, like VideoMAE V2 [2] and FLIP [3]. In our view, it is the simplicity and efficacy of these methods that render them ideal as foundational models, requiring minimal code adjustments for integration into existing frameworks. Upon acceptance, we plan to release all relevant codes, pre-trained models, and well-crafted APIs for immediate application. We are confident that these foundational models will greatly benefit the broader community, particularly academic institutions with limited resources for large-scale experiments. Our contributions, we believe, merit recognition at prestigious conferences such as ICLR.

[1] Parameter-Efficient Transfer Learning for NLP, ICML 2019

[2] VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking, CVPR 2023

[3] Scaling Language-Image Pre-Training via Masking, CVPR 2023

W1: Minor difference between TVTSv1 and TVTSv2

W1.1: The partially frozen tuning strategy lacks novelty.

A: Thanks, we are willing to emphasize the novelty of the proposed partially frozen tuning. Note that it plays a vital role (rather than a trick) in training on the easy-to-scale-up ASR-video pairs, which is not trivial given that few previous papers consider tackling the text distortion issue. We carefully break down the experiment step and pinpoint the problem. We identify the problem via comprehensive experiments and figure out a simple yet effective solution that is in favor of practical use and can be well transferred to different datasets (Table 6) and vision models (Table 7). The logic, motivation and ablation studies are mostly acknowledged by all reviewers.

To further underscore our contribution, we introduce a ViT-B/32-based alternate baseline, integrating adapters [1] between text encoder layers, denoted as M$_{\text{adapter}}$. During training, only the adapter-introduced parameters are trainable. The MSR-VTT retrieval results, detailed in the subsequent table, show that this alternative method yields comparatively lower performance. This outcome reinforces our assertion that end-to-end tuning can cause style overfitting, even when the text encoder remains unaltered. It also confirms that our approach of partially frozen tuning is not just a superficial fix, but a substantive and effective solution to the identified problem. We have added this discussion in our revised manuscript (Appendix D Table 16).

We believe that a paper's contribution should not be limited to proposing novel methods. Extensive empirical studies (i.e., analyzing why performance degrades), useful findings/insights (i.e., the overfitted text model), scalable training schemes (i.e., the simple yet practical partial freezing), and foundation models (i.e., TVTSv2 models that can generalize to various tasks) are also essential to the community, especially in the era of big models. Such contributions are also acknowledged by Reviewer hjUh (Task-Agnostic Focus, Novel Pre-training Strategy, etc.) and Reviewer A6Wr (Effective Pre-training Strategy).

[1] Parameter-Efficient Transfer Learning for NLP, ICML 2019

W1.2 TVTSv1's poor performances may be caused by weak initialization (Appendix D).

A: We admit that a strong initialization yields better performance, as reported in Appendix D Table 20. However, naively relying on stronger initialization leads to degraded performance when pre-training on large-scale ASR-paired videos. As shown in Table 1 in our initial manuscript (the subsequent Table) , after equipping CLIP weights, all fully tuned variants still lag behind the partially tuned competitors (L3 vs L7, L5 vs L8), and they even degrade compared to the CLIP baseline (L1 vs L3, L5), indicating the importance of the proposed partial frozen tuning strategy for producing general-purpose video representations.

W1&Q1: Elaborate on the choice of tasks for empirical evaluation and envision the model's performance on other types of spatiotemporal tasks like action recognition or anomaly detection.

A: Thanks for your advice! We have conducted relative experiments to envision the model's performance in the revision.

For action recognition, the results are reported in Table 3 in our initial manuscript. We use the prompt template "a person [CLASS]" (e.g., a person running) for HMDB-51 [1], UCF-101 [2], Kinetics-400 [3], and Kinetics-600 [4], and evaluate SSV2 on a multi-choice setting, namely SSV2-MC. The detailed settings are stated in Appendix A. Our model brings significant improvements, i.e., 8.9%, 9.1%, 11.6% and 12.9% absolute gain on HMDB-51, UCF-101, Kinetics-400, Kinetics-600 and SSV2-MC respectively. The prompt-based method, i.e. ActionCLIP [5], also degrades on this test, possibly due to overfitting to manually constructed templates. When it turns to larger models, we even surpass X-Florecnce [6], a single-stream model with comparable video encoder parameters, by a large margin, revealing the superior scalability of our training paradigm. Even for the challenging SSV2-MC that contains various motion dynamics, our model still achieves promising results, solidifying the contribution of masked transcript sorting that promotes fine-grained spatiotemporal representation learning. In addition, we provide the liner classification results in Appendix D Table 13, where we surpass either self-supervised or language-guided models.

For anomaly detection, we evaluate the performance on UCF-Crime [7] test split, which aims to recognize 13 anomaly events. According to Table 3 in our revised manuscript, our models also made remarkable improvements compared to the CLIP baseline and CLIP-ViP, unveiling their adaptability and generalizability.

[1] HMDB: A large video database for human motion recognition, ICCV 2011

[2] UCF101: A Dataset of 101 Human Actions Classes From Videos in The Wild, arXiv 2012

[3] The Kinetics Human Action Video Dataset, arXiv 2017

[4] A Short Note about Kinetics-600, arXiv 2018

[5] ActionCLIP: A New Paradigm for Video Action Recognition, arXiv 2021

[6] Expanding Language-Image Pretrained Models for General Video Recognition, ECCV 2022

[7] Real-world Anomaly Detection in Surveillance Videos, CVPR 2018

W2&Q3: Providing a detailed computational and memory complexity analysis during both training and inference.

A: Nice suggestion. We've updated our manuscript to include a comprehensive analysis of computational and memory complexity, which you can find in Appendix D Table 12. During the training phase, we optimized the batch sizes to fully utilize available GPU memory. Our analysis, especially the GFLOPs comparison between the video and text encoders, reveals that the most computationally intensive part is video encoding, particularly the spatiotemporal attention module. However, the use of an efficiently designed divided space-time attention mechanism ensures that even our largest model, i.e., ViT-H/14, can be run on consumer-level GPUs like the RTX 3090, thanks to its relatively modest memory requirements during inference.

Q2: How does the model perform under conditions of noisy or incomplete data? Consider evaluations that specifically test the model's robustness.

A: Thanks for the constructive advice! To test the robustness of our model, we randomly mask some patches in each frame during inference to simulate corrupted data. The retrieval performance under different masking ratios on MSR-VTT is reported in Appendix Table 18. Thanks to the masked contrastive pre-training, our models are robust to data corruption. Impressively, even minor data distortions, such as 10% or 20% masking, do not significantly impact performance. More notably, our models demonstrate commendable robustness, maintaining considerable effectiveness even under substantial data incompleteness, with a masking ratio as high as 50%. This clearly underscores the reliability and adaptability of our models in handling corrupted data scenarios.