Enhancing Video Understanding with Vision and Language Collaboration

W1: Unmatched title: Thanks for your helpful suggestions. The experiments have been extensively conducted on popular action recognition datasets, and we have revised our title accordingly for improved clarity.

W2: marginal improvements: Tab. 3 displays the comprehensive ablation studies by removing or adding the proposed losses, with using cross entropy alone as baseline. The absence of either $\mathcal L_{text}$ or $\mathcal L_{image}$ leads to a performance drop (Tab. 3). In contrast, the table blow shows that incorporating both loss terms enables our method to surpass the baseline by 1.56% and 1.48% on UCF101 and Kinetics-400, respectively, demonstrating that $L_{text}$ and $L_{image}$ provide orthogonal and substantial performance gains.

Q1, Q4, Q5: Sorry for the confusion. We have corrected these mistakes in the manuscript.

Q2: Experiments on SSV2. The table below presents the comparison on SSV2, including recent video-based and VLM-based transfer methods using the same ViT-L backbone. Our method achieves 74.9% Top-1 accuracy and 94.8% Top-5 accuracy. Notably, SSV2 places greater emphasis on temporal understanding compared to Kinetics-400, which explains why video-based transfer methods (e.g., VideoMAE) typically outperform VLM-based approaches. By leveraging a video pre-trained model to retain temporal understanding while integrating knowledge from VLM, our method achieves state-of-the-art performance.

[1] VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training, NeurIPS2022.

[2] Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning, ICCV2023.

[3] Enhanced Motion-Text Alignment for Image-to-Video Transfer Learning, CVPR2024.

[4] Rethinking Image-to-Video Adaptation: An Object-centric Perspective, ECCV2024.

Q3: Ablation study on weight coefficients for the loss function We conduct grid search on the weight coefficients to determine the optimal values. The results in the table below demonstrate that our method is robust to variations in the two hyperparameters. For all experiments, we select $λ_{t}=1$ for $\mathcal L_{text}$ and $λ_{i}=10$ for $\mathcal L_{image}$.

Q1: Ablation study on number of negative descriptions, i.e., k.: Sorry for the any confusion caused by Fig. 3. To fully investigate the irregular fluctuation especially when k ranges from [200,1000], we conducted experiments with varying k using different random seeds. Results of k ranges from [200,1300] are presented in Fig.3. The box plot reveals slight fluctuations when k $<$1000 and a noticeable increase as k$>$ 1000, which can be attributed to the inclusion of more hard negative descriptions. Based on these results, we selected the optimal k $=$ 1200 for all experiments. Fig. 3 in the manuscript has been updated accordingly.

Q2: Visualization to validate the effectiveness of gate mechanism: We include additional visualization results provided in Fig. 4. These results show that training without the gate mechanism leads to diffused attention, while incorporating it focuses attention more effectively on the foreground, thereby enhancing overall performance.

Q3: Number of the generated descriptions and working principle for positive text: Our description candidates for each class includes 3 kinds of descriptions: 10 different descriptions generated by decompose-expand prompting method, 2 different detailed descriptions generated by GPT 4-o and the original action name. For each sample, we choose the description from the ground-truth description candidates with the highest image-text similarity score as positive. This strategy ensures that the selected positive description has a similarity score equal to or higher than that of the original action name, thus making the positive description is conceptually matched to the sample.

W1: Insufficient experiments prove the core motivation in Fig. 1(c): We add additional visualization results in Appendix A.4 to further demonstrate the general spatial understanding capabilities of Vision-Language Models (VLMs), such as CLIP, which excel at focusing on foreground objects. Based on these observations, it is reasonable to leverage VLMs to enhance the spatial understanding capabilities of video pre-trained models.

W2: The table below compares our method with UMT(baseline) in terms of training time on Kinetics-400. All experiments are conducted on the same GPUs using identical training configurations, including batch size, training epochs, and learning rate. Despite comparable training hours, our approach achieves a significant improvement in accuracy.

Q1: Relations between lines42-44 and Fig1(c): Sorry for the confusion. We would like to emphasize that video pre-trained models often lack spatial understanding due to their video-centric pre-training objectives. Specifically, as shown in Fig. 1(c), the video pre-trained model is noticeably distracted by the background, whereas CLIP focuses more on the foreground person. This observation motivates us to leverage CLIP's strong spatial encoding capabilities to enhance the spatial understanding of video pre-trained models, thereby improving video understanding.

Q2 and Q4: Comparison with the state-of-the-art on Kinetics-400: Tab. 1 presents a comparison of all methods using ViT-B as the backbone, whereas the works mentioned by the reviewer employ ViT-L to achieve 88.5% [1] and 89.7% [2]. For a fair comparison, we provide an additional experiment using ViT-L in the table below. The results demonstrate that our method achieves the highest performance in both top-1 and top-5 accuracy, surpassing the second-best method, DiST, by 1.0% in top-1 accuracy. In summary, our method outperforms others with both ViT-B and ViT-L as backbones.

[1] Rethinking Image-to-Video Adaptation: An Object-centric Perspective, ECCV2024.

[2]Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning, ICCV2023.

Q3: Correct usage of generated descriptions by the decompose-expand prompting: To select the conceptually correct positive description, we consider the union of generated descriptions and the original action names of the ground-truth label as text candidates. From these, we choose the one with the highest image-text similarity score as positive. This strategy ensures that the selected positive description has a similarity score equal to or higher than that of the original action name. For creating negative descriptions, we select the bottom-k candidates with the lowest similarity scores, excluding the ground-truth descriptions. This ensures that the selected negative descriptions are irrelevant to the training sample. We have updated the manuscript to clarify this process.

W1: unclear contribution: Our main contributions stand in enhancing video understanding by equipping pre-trained models with Vision-Language Models (VLMs), which is under-explored in the literature. However, adapting the visual and textual knowledge of VLMs to video tasks is non-trivial due to modality gaps. To the end, we propose attentive spatial and contrastive textual knowledge transfer modules to unveil the VLM’s power. These modules, combined with simple yet effective loss functions, are extensively validated through ablation experiments, as shown in Tab. 3.

W2: In-depth discussion of the cross-attention method: To explore the cause of the suboptimal performance of the cross-attention method, additional experiments are conducted in Appendix A.3. Specifically, we visualize the segmentation mask and Grad-CAM of a few typical examples in Fig. 6. It reveals the suboptimal performance of the Cross-Attention method arises from unstable segmentation results, particularly when the objects are either too small or obscured.

W3: Unclear description: Thanks for your helpful suggestions. We rely on text-image similarity scores to determine whether descriptions are positive or negative. High scores indicate a strong correlation between the video/frames and the text descriptions, while low scores suggest that the descriptions do not accurately reflect the video content and thus serve as negatives. We have updated it in the manuscript.

W4: Sensitivity analysis of hyperparameter k: Sorry for the any confusion caused by Fig. 3. To evaluate the sensitivity to the number of negative descriptions, namely k, we conducte experiments with varying k values using different random seeds. The box plot reveals slight fluctuations when k $<$ 1000 and a noticeable increase as k $>$ 1000, which can be attributed to the inclusion of more hard negative descriptions. Based on these results, we select the optimal k $=$ 1200 for all experiments. Fig. 3 in the manuscript has been updated accordingly.

Q1: The performance gain: We thank the reviewer for recognizing the motivation behind our work. Our method has been extensively evaluated on three video recognition datasets: UCF-101, Kinetics-400, and Something-Something-v2 (SSv2), achieving consistent improvements. Specifically, on UCF-101, we achieve a notable 1.56% gain in Tab. 2(b) compared to the strong UMT baseline using ViT-B. On the challenging Kinetics-400 and SSv2 datasets, we obtain improvements of 1.0% and 0.6% by using ViT-L, respectively. Results are shown below.

[1] VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training, NeurIPS2022.

[2] Disentangling Spatial and Temporal Learning for Efficient Image-to-Video Transfer Learning, ICCV2023.

[3] Enhanced Motion-Text Alignment for Image-to-Video Transfer Learning, CVPR2024.

[4] Rethinking Image-to-Video Adaptation: An Object-centric Perspective, ECCV2024.

Q2: Ablation study in Tab. 3: In Tab. 3, we present comprehensive ablation studies on the proposed loss functions $\mathcal L_{text}$ and $\mathcal L_{image}$. Notably, these two loss functions provide orthogonal performance gains. Different configurations of these loss functions consistently enhance the baseline performance. Further validation is provided by the ablation study on Kinetics-400 in Tab. 5, which demonstrates that introducing the proposed loss functions yields the best performance.

We sincerely thank all the reviewers for their insightful comments, which have significantly enhanced the quality of our manuscript. We look forward to receiving further feedback and are eager to engage in discussions to address any concerns to the best of our ability.