ResidualViT for Efficient Zero-Shot Natural Language Temporal Video Grounding

[W1] Additional tasks and datasets. Thank you for raising this issue. We would like to clarify the choice of datasets and tasks. We purposely do not evaluate TaCoS and MSR-VTT for the following reasons. As also described in the answer to Reviewer y3Uz ([Q3]), we develop ResidualViT purposely for tasks requiring dense frame extraction, for which features are extracted in nearby frames with a certain degree of visual redundancy. Text-to-video retrieval is usually addressed by sparsely sampling 4-6 frames in videos, trying to reduce the redundancy among frames to obtain a more discriminative video representation [D]. This setting does not reflect the relevant use case for ResidualViT.

Additionally, while TACoS has historically been popular, it no longer adequately captures the complexities needed to evaluate video grounding solutions. The TACoS dataset consists of only 27 videos for testing, totaling about 2.2 hours, all recorded with a static camera with an identical background, which limits its diversity. Additionally, the dataset focuses solely on cooking activities. The UniVTG [E] manuscript also notes that evaluating zero-shot approaches on TACoS is not very informative (Section 5.2.2).

Instead, we have chosen to evaluate our model on three well-established datasets: ActivityNet Captions (4885 videos, total of 162 hours), Charades-STA (1334 videos, total of 12 hours), and MAD (112 videos, total of 210 hours). In particular, the challenging, long-form, large-scale, and highly diverse MAD dataset provides a robust benchmark for assessing our approach. By focusing on these datasets, we aim to provide a comprehensive evaluation that better reflects the diversity and dynamic nature of real-world scenarios, offering a more meaningful assessment of our model's capabilities.

Additionally, with this rebuttal, we present results in two new tasks, namely: Action Recognition and Temporal Activity Localization. The results for Action Recognition are detailed in Appendix L (pages 30-31 of the updated manuscript), while results for Temporal Activity Localization are detailed in Appendix M (page 31 of the updated manuscript). For both tasks, we show that ResidualViT features provide an excellent accuracy vs. cost tradeoff, closely following the upper bound accuracy provided by CLIP but at a lower computational cost. Moreover, these results also show that our features can be successfully used for vision-only tasks.

[D] Luo, Huaishao, et al. "Clip4clip: An empirical study of clip for end to end video clip retrieval." arXiv preprint arXiv:2104.08860 (2021).

[E] Univtg: Towards unified video-language temporal grounding. In ICCV, 2023.

[W2] Metric. We thank the reviewer for finding this typo. We would like to clarify that none of the weakly supervised, pseudo-supervised, and zero-shot methods cited in our work, except for CPL and U-VMR, report R@5 metrics. In line with previous works, we chose to focus on reporting the top-1 predicted moment, as it is considered the most relevant and important metric.

However, for completeness, we have included the R@5 performance metric for the MAD dataset in Table 4 and commit to providing the R@5 results for Charades-STA and ActivityNet-Captions in the final version of the manuscript.

[W3] Additional References. We thank the reviewer for these additional citations. As a result of this comment, we have added these references to the related work in Section 2.

We would like to point out that these methods deal with model weights pruning/reduction, while our design aims at avoiding any modification to the weights of existing large-scale pre-trained models while reducing their inference cost by manipulating the input data fed to them. Therefore, the methodology proposed in these papers is complementary to our setting. Additionally, [1] and [3] do not do not address video understanding tasks.

[Q1] Computational costs clarification. We thank the reviewer for the opportunity to discuss this key aspect. The discrepancy between the previous and current versions of our computational cost analysis stems from our misinterpretation of the prior work, MR-FLVM. Prior to our paper resubmission to ICLR’25, we conducted a careful re-analysis of MR-FLVM and engaged in direct discussions with the MR-FLVM authors via email.

MR-FLVM represents video data as 32 snippets, each containing a few frames, which are initially encoded using 3D networks (C3D at 38.5 GFLOPs for Charades-STA and I3D at 148.4 GFLOPs for ActivityNet-Captions). In addition, each snippet is re-encoded using InternVideo for snippet-text similarity calculations—contrary to our initial assumption that this step used CLIP. The use of InternVideo, which applies full self-attention over all patches in 16 frames using a large model (InternVideo-MM-L-14), incurs a significant computational cost of 1.332 TeraFLOPs per feature (measured by us on the official InternVideo implementation). Once similarity scores are obtained, each snippet feature is refined through a weighted sum of all snippet representations with high similarity scores to the text query, effectively smoothing the snippet representations over time. The snippets are then combined into proposals and re-scored using InternVideo.

Based on this information, we estimated the average feature extraction rate for Charades-STA to be ~1 feature per second (as the average video duration is 30 seconds) and ~0.25 features per second for ActivityNet-Captions (with average video durations of 2 minutes).

The total average computational cost per feature per second, which we use as a metric in Table 2, is calculated as the cost of encoding one feature multiplied by the extraction rate of features per second. For MR-FLVM, this results in a computational cost of (38.5 + 1332) GFLOPs × 1 = 1370 GFLOPs for Charades-STA, and (148.4 + 1332) GFLOPs × 0.25 = 370 GFLOPs for ActivityNet-Captions.

[Q2] Fair comparison. We appreciate the opportunity to address this concern. The primary purpose of the distillation loss (Equation (2)) is to train the residual tokenizer to approximate the original CLIP features. This process focuses on feature alignment rather than task-specific optimization, i.e., our training loss does not explicitly encourage the feature representation to learn new information beyond the original CLIP representation. Note that the accuracy of our approach is upper-bounded by the original CLIP representation.

Further evidence is provided in the ablation regarding the distillation strategy presented in Appendix E. In detail, we remove the language features from the distillation approach and replace the loss to be the mean-squared error (MSE) between the visual representation of the ViT teacher and the visual representation of the ResidualViT student. We illustrate the new distillation pipeline in Figure 11(a). Our investigation results are reported in Figure 11(b) where we showcase that this approach provides near identical grounding accuracy as compared to the distillation that adopts language features and cross-entropy loss (Equation (2)). Notice that for a fair comparison, all training and inference hyperparameters are kept exactly the same. The key takeaway is that ResidualViT is capable of closely approximating the CLIP features and retaining the alignment with the original CLIP language model even in this setting.

[Q3] More pre-training data and scalability. We would like to clarify that our method involves only one stage of training of the residual tokenizer alone, i.e., there is no post-training stage. It is conceivable that incorporating additional data points could further enhance the quality of distillation. However, our accuracy already approaches the upper bound established by CLIP, which suggests that further scaling of the training data may yield diminishing returns. One of the key reasons is that the number of learnable parameters is small (~400K for B/32 and B/16 models and ~800K for L/14), as only the residual tokenizer weights are learned.

[Q4] Increasing N to a larger value. We would like to clarify that the accuracy vs. cost trade-off for different values of the interleaving factor N is analyzed in Figure 4 of the main paper, where we report accuracy for N=1,2,3,5, and 10. Additionally, in response to requests from reviewers cDMc (Q1) and jGpQ (Q3), we are actively conducting experiments on training with varying N values and will make every effort to complete these experiments within the rebuttal period.

[W1] Clarifications: Thank you for your comments. We would like to address the points raised by the reviewer and clarify several aspects of our work:

1. We focus on transformer-based architectures as they have become the de-facto standard architecture in the vast majority of computer vision and natural language processing applications. The ViT architecture has proven to be highly scalable and well-performing. In this regard, we base our work on the most widespread backbone to date.

2. The reviewer's assessment is correct that ResidualViT does not explicitly model temporal relationships between frames. Nonetheless, our token reduction plus residual strategy is agnostic to the backbone type, allowing for flexibility in its application. For example, our approach could be extended to a spatiotemporal video foundation model by applying tokenization to 3D spatiotemporal volumes while still applying the token reduction module (i.e., dropping patches over volumes) and utilizing the residual tokenizer. Our primary objective remains to develop a computationally efficient model through distillation that approximates the accuracy of the foundation model (CLIP in our study). Our research aims to provide an efficient framework for video frames encoding, complementing methods focused on temporal dependency modeling. Moreover, as discussed in Section 2 (lines 104-115), frame-based video encoding remains popular due to its computational efficiency. While dedicated temporal modeling can enhance accuracy, it often increases computational demands, which must be carefully balanced against possibly modest accuracy gains.

3. We would like to draw the reviewer’s attention to Appendix E, where we present an ablation study of our distillation approach without requiring language. In detail, we remove the language features from the distillation approach and replace the loss to be the mean squared error (MSE) between the visual representation of the ViT teacher and the visual representation of the ResidualViT student. We showcase the new distillation pipeline in Figure 11(a). Our investigation results are reported in Figure 11(b), where we showcase that this approach provides near identical grounding accuracy as compared to the distillation that adopts language features and cross-entropy loss. Notice that for a fair comparison, all training and inference hyperparameters are kept exactly the same. This result further demonstrates the flexibility of our methodology.

4. Finally, as requested, we investigate the quality of our ResidualViT features as opposed to CLIP features on the task of Temporal Activity Localization. For this experiment, we identify ActionFormer [B] as a recent, well-performing, and easy-to-use baseline for the task. We extract features using the CLIP B/32 and ResidualViT B/32 (N=2, p=85%) visual backbones at one frame per second and train the ActionFormer baseline from scratch for both sets of features. We find that CLIP features can obtain a mAP of 34.05% while ResidualViT closely follows with a mAP of 33.25%. This experiment confirms that our features, which can be computed with only 44% of the CLIP cost, can achieve very competitive accuracy with the CLIP upper bound on vision-only tasks. We include this new result in Appendix M (page 31 of the updated manuscript).

We also would like to point out that we performed a comparative study for the task of Action Recognition following reviewer y3Uz question 3, and we report the results in Appendix L (pages 30-31 of the updated manuscript). In this evaluation, we showcase how ResidualViT provides competitive performance with respect to CLIP for the same amount of encoded frames. Moreover, when analyzing the accuracy versus total encoding cost, ResidualViT demonstrates a clear advantage: with four frames and a total cost of 12.8 GFLOPs, it outperforms CLIP with three frames and a total cost of 13.2 GFLOPs for both Accuracy@1 and Accuracy@5.

Moreover, following the request of reviewer cDMc, we employ the ResidualViT features in a fully supervised method for natural language temporal video grounding. We present the result in Appendix K, where we showcase that the ResidualViT features induce a cost saving of over 50% at the cost of an average relative accuracy degradation of about 2%.

All of these experiments point to the conclusion that our methodology has proven successful in reducing the video encoding cost while retaining close performance to the upper bound provided by CLIP for several video recognition tasks that now include Natural Language Temporal Video Grounding, Automatic Audio Descriptions, Action Recognition, and Temporal Activity Localization on four datasets, namely: Charades-STA, ActivityNet-Captions / ActivityNet, MAD, and Kinetics-400.

[B] Zhang, Chen-Lin, Jianxin Wu, and Yin Li. "Actionformer: Localizing moments of actions with transformers." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.

I would like to thank the authors for their detailed response. I acknowledge that some of the requested ablations were previously included in the appendix and am glad to see the results further highlight the strength of the proposed method. For this reason, I have raised my rating and recommend acceptance of the paper. In the meantime, I encourage the authors to reconsider the title of the paper. As the other reviewers correctly pointed out, the proposed approach is not limited to video grounding and has been shown to be effective for other video understanding tasks. Please revise the text accordingly to not overemphasize ResidualViT as a video grounding method.

[W1] Distillation through the soft target. Thank you for pointing this out. We would like to emphasize that our work primarily addresses the practical challenge of efficiently deploying a trained model (forward inference), with training efficiency being outside the primary scope of this study. The result in Section E is encouraging, as it demonstrates that training can be conducted without the need for language annotations (reducing the training cost by approximately 10% for ViT-B/32, 3% for ViT-B/16 and 1.5% for ViT-L/14). Importantly, this observation does not diminish our main contribution: the effectiveness of learnable residual connections combined with token dropping during inference. If requested by the reviewer, we would be happy to revise the description of our secondary contribution (training via distillation) to clarify that the training process can be carried out with or without the use of language annotations.

[W2] Training Objective. We thank the reviewer for this insightful suggestion. Incorporating frame-language or video-language interactions into the learning objective presents an exciting avenue for future research. We have included this suggestion in the manuscript to encourage the community to build upon our approach and explore further enhancements.

In detail, we have added the following sentence to our conclusions: “We believe that our work opens up the possibility of extending the distillation objective to incorporate richer interactions between visual and language representations, as well as exploring additional large-scale pre-trained models that natively model temporal relationships.”

[W3] Missing references. We thank the reviewer for highlighting these related works; each reference makes a great addition to our list of citations. In response, we have updated section 2 of the manuscript accordingly. We invite the reviewer to read the updated version of the manuscript. All additions have been highlighted in blue.

[W4] Additional issues. We thank the reviewer for providing constructive feedback for improving the clarity of our work.

1. Training details. We include in “Appendix H Implementation details” the specifications of the training setting used in our manuscript. In particular, our method is trained on video-text pairs from the WebVid-2.5M dataset for five epochs. Our batch size is 2048 for ViT-B/32 and ViT-B/16 models and 1536 for ViT-L/14. At training time, we set the interleaving factor N_{train}=3. We use a constant learning rate of 0.0005 while weight decay and warmup are disabled. All model training is performed on 4 V100 GPUs, while inference only requires 1 V100 GPU. To improve the clarity of our manuscript, we modified the introductory paragraph of Section 4 to explicitly mention that the training details are provided in Appendix H. All changes to the manuscript are highlighted in blue.
2. Inference details. The inference process is detailed in Appendix C as part of the zero-shot grounding algorithm description. This section explains how video frames and textual queries are encoded into feature representations and then processed for the grounding task. Additionally, we provide the implementation details, such as FPS settings, the smoothing window, and the watershed threshold used, in Appendix H. We modified the introductory paragraph of Section 4 with a pointer to the inference details.
3. We appreciate the suggestion regarding citation formatting. We have updated the manuscript to use \citep{} for improved readability and consistency.

[Q1] Language description. We thank the reviewer for this question. We use the WebVid-2.5M dataset for distillation, which contains language-video pairs. As far as we are aware, there are no annotations for the quality of the language descriptions for this dataset, making an analysis of our approach according to the descriptions’ quality difficult. If the reviewer has suggestions for how to conduct such an analysis, we are happy to attempt it.

Regarding long-form videos, we would like to clarify that we are not training using long-form videos. The dataset used for distillation is WebVid-2.5M whose average video duration is approximately 17.6 seconds. Additionally, in this work, we focus on frame-level feature embedding cost reduction. In this context, we apply the token reduction module for cost reduction and exploit the residual tokenizer for providing information from temporally neighbouring frames. Therefore, the approach is agnostic to the video duration, as our architecture remains a frame-based encoding backbone. Additionally, our zero-shot grounding baseline only exploits frame-text similarities for predicting relevant moments and is again agnostic to the video duration (the reason why we can adopt the same algorithm for short videos (Charades-STA / ActivityNe-Captions datasets) and long videos (MAD dataset)).