FROSTER: Frozen CLIP is A Strong Teacher for Open-Vocabulary Action Recognition

Thanks for your precious time reviewing this paper and writing meticulous and constructive feedback. We are delighted that this paper is found to be simple but effective, and the experimental results are promising.

Q1: The authors touch upon "Text rephrasing" in Sec. 4.2, yet there's an absence of any ablation study related to it.

R: The impact of text rephrasing is shown in the last two rows of Tab. 4 of the paper, which is presented below. It can be found that rephrasing the class names into descriptions can improve performance since they provide more detailed information about the actions. The corresponding experimental analysis is given in Sec. 4.3 of the paper.

Table 1: Impacts of text rephrasing under the cross-dataset evaluation setting.

Q2: In the caption of Table 1, the authors mentioned that 'The results of most of the other papers are taken from Open-VCLIP and ViFi-CLIP.', where are the results for Open-VCLIP? From Table 2 and Table 3, if my understanding is correct, the reported results use the Open-VCLIP as the backbone. Hence, showcasing Open-VCLIP's results in Table 1 becomes critical to underscore the effectiveness of the suggested method.

R: Please refer to Tab. 1 of our paper. Thanks for the suggestion and we have included the suggested comparison. Below, we present the results of VCLIP. The results demonstrate that our method can improve the performances for both the base and novel categories for all datasets.

Table 2: Performance (top1-acc (%)) under the base-to-novel evaluation setting. B, N, and HM denote the base set, novel set, and harmonic mean, respectively.

Q3: It would be beneficial to incorporate columns for "encoder" and "pretrained dataset" within the table for clearer comprehension.

R: We have followed this suggestion to indicate this information in the table.

Thanks again for your valuable comments and looking forward to your reply.

[1] Chen, Shizhe, and Dong Huang. "Elaborative rehearsal for zero-shot action recognition." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[2] Lin, Ji, Chuang Gan, and Song Han. "Tsm: Temporal shift module for efficient video understanding." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.

We are grateful for your thoughtful and thorough review. Your insightful comments and constructive suggestions are of great help to us. We are delighted that our paper is found to be well-written and easy to follow, the proposed method is interesting, and the performance improvement is significant. Please refer to the detailed response below regarding the raised several weaknesses.

Q1: It would be better to report the finetuned results on all datasets (including K-400) for a better comparison.

R: Thanks for the kind suggestion and we have included the additional results in Tab. 5 of Appendix A.3. Since this paper focuses on the open-vocabulary recognition setting, we did not include the closed-set recognition results (fine-tuning and evaluation are conducted on the same dataset) in the initial submission. In Tab. 1 below, we include the closed-set evaluation results on K-400. We can see that our method achieves comparable performance to the state-of-the-art closed-set models (AIM and ViFi-CLIP). At the same time, our method consistently outperforms them by a large margin under the cross-dataset evaluation, i.e., UCF101, HMDB51, and K-600 datasets. The results indicate that the state-of-the-art closed-set methods fit the seen categories well but cannot be satisfactorily generalized to unseen ones. To evaluate the overall performance under the closed-set and open-vocabulary settings, we report the harmonic mean of them in Tab. 2 below. From Tab. 2, we find that our method achieves the best trade-off on both the closed-set and open-vocabulary settings.

By the way, inspired by this suggestion, developing a more comprehensive protocol to evaluate overall performance on both closed-set and cross-dataset scenarios is a promising direction for our future work.

Table 1: Performance (top1-acc (%)) of different methods under the closed-set (K-400) and cross-dataset (UCF,HMDB, and K-600) evaluation settings. Except for ER-ZASR (TSM), all the methods are based on the CLIP-ViT-B/16 model.

Table 2: Harmonic mean (top1-acc (%)) of different methods under the closed-set (K-400) and cross-dataset (UCF,HMDB, and K-600) evaluation settings.

Q2: The performance gains compared to baseline methods (in Tables 1 and 2) are indeed surprising, and it would be better to visualize the learned embeddings on both base and novel datasets to see the effects of RFD on feature learning.

R: Thanks for the kind suggestion. We have included the t-SNE visualizations of the learned embeddings on both base and novel datasets in the supplementary material. Interestingly, we observe that despite the superior performance of our method compared to the VCLIP model (or superior performance of VCLIP over frozen CLIP), there does not appear to be much visual discrepancy in the t-SNE projection of different methods. The possible reason is that the fine-grained spatial-temporal information that is essential for discriminating actions may not be well preserved in the low-dimensional space of t-SNE.

However, from the visualizations of spatial-temporal attention maps (refer to Fig. 4 in our paper and Fig.6, Fig. 7, Fig. 8, and Fig. 9 in the Appendix), our method can better focus on parts related to informative patterns and actions compared to frozen CLIP and VCLIP.

Q3: The results in Table 1 are a little bit confusing, as the "Top1 accuracy" is only 77.8 on base categories. It would be better to further clarify the training and evaluation protocol, and the number of clips/crops used.

R: Under the base-to-novel evaluation setting, each class consists of only 16 samples in the training set. This setting was first introduced by ViFi-CLIP [1], where the model is trained on base (seen) classes in a few-shot manner and evaluated on a set of novel (unseen) classes. Due to the limited training samples, the performance on the base categories is relatively low.

For the training and evaluation protocol, we use videos with 8 frames for training and sample 3 clips (8 frames each) with 1 crop (3x1 views) for evaluation.

The above protocol details have been given in Appendix A.1 and A.2 of the paper, please refer to them.

Q4: As the paper employs rephrased action descriptions, instead of action names, it would be interesting to show the video-text retrieval results.

R: We appreciate this suggestion and conducted the video-text retrieval experiment. We followed the evaluation setting in Open-VCLIP [3], where the model is initially trained on the Kinetics-400 dataset and then evaluated on the video-text retrieval dataset, MSR-VTT [4], to evaluate the generalizability. Compared to Open-VCLIP, our method consistently achieves better performance in all cases.

Table 3: Video-text retrieval performance on MSR-VTT.

Thanks again for your valuable comments and looking forward to your reply.

[1] Rasheed, Hanoona, et al. "Fine-tuned clip models are efficient video learners." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PMLR, 2021.

[3] Weng, Zejia, et al. "Transforming CLIP to an Open-vocabulary Video Model via Interpolated Weight Optimization." arXiv preprint arXiv:2302.00624 (2023).

[4] Xu, Jun, et al. "Msr-vtt: A large video description dataset for bridging video and language." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.

Q2: The authors didn't employ a compact student model as in normal knowledge distillation. I am wondering if the performance is better if we use ViT-B/32 as teacher and ViT-B/16 as student.

R: Thanks for the suggestion. We suppose the reviewer meant using ViT-B/16 as the teacher and ViT-B/32 as the student, because '/32' and '/16' indicate the patch size for patch embedding, making ViT-B/32 a more compact model than ViT-B/16. We conduct the comparison in Tab. 1 below. We observe that using a compact student model can also consistently improve performance, which further demonstrates the effectiveness of our proposed method.

Table 1: Performance (top1-acc (%)) of using different models for distillation under the cross-dataset evaluation setting. --- indicates only conducting model finetuning but not involving knowledge distillation.

Q3: In Fig. 1, why CLIP performs more stable on larger Kinetics dataset than HMDB51 or UCF101?

R: We are not completely sure what the reviewer means by "stable" here. We suspect that it refers to the fact that the frozen CLIP achieves better performance than the fine-tuned methods on the Kinetics-600 dataset, instead of the HMDB51 and UCF101 datasets. We clarify it as follows.

The experiments in Fig. 1 of the paper were conducted under a cross-dataset evaluation setting, where the models were fine-tuned (except for the frozen CLIP) on the Kinetics-400 dataset and then evaluated on the UCF101, HMDB51, and Kinetics-600 datasets. Please note that in the Kinetics-600 dataset, we excluded the classes that are identical to those in the Kinetics-400 dataset. In this way, the average text-feature similarity (see Fig. 1 of our paper) of all categories between Kinetics-600 and Kinetics-400 (0.721) is smaller than that between UCF101 and Kinetics-400 (0.805) and that between HMDB51 and Kinetics-400 (0.771). Therefore, compared to the UCF101 and HMDB51 datasets, recognition performance on the Kinetics-600 dataset relies more on generalizability. Since the frozen CLIP has not been fine-tuned on video data, its pre-trained generalizability is better maintained than the fine-tuned methods. Hence, the frozen CLIP achieves better performance than fine-tuned methods on the Kinetics-600 dataset, instead of the HMDB51 and UCF101 datasets.

Q4: What is the contribution of text rephrasing in Section 4.2?

R: In the last paragraph of Sec. 3.1.1 in the paper, we mention that the text rephrasing is introduced to rephrase the action names into descriptions for enriching the text diversity, which serves as text data augmentation during training, due to the limited template text provided.

The impact of text rephrasing is evaluated in the last two rows of Tab. 4 of the paper, which is shown below.

Table 2: Impacts of text rephrasing under the cross-dataset evaluation setting.

We have added the above description in 3.1.1 in the paper to further clarify.

Thanks again for your valuable comments and looking forward to your reply.

[1] Pei, Renjing, et al. "CLIPPING: Distilling CLIP-Based Models with a Student Base for Video-Language Retrieval." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2]Dai, Wenliang, et al. "Enabling multimodal generation on CLIP via vision-language knowledge distillation." Findings of Association for Computational Linguistics 2022: 2383-2395.

Thanks for your time and efforts in reviewing this paper and your detailed and insightful feedback. Your constructive suggestions are of great help to us. We are pleased that our paper is found to be well-motivated, the proposed method is practically useful and easy to implement, and the experiments are comprehensive.

Please refer to our detailed response below, regarding the concerns about novelty, experiments of knowledge distillation using a small student model, and additional explanation of experimental results.

Q1: I have concerns on the novelty. Knowledge distillation on CLIP is not new in this field. The authors didn't discuss related references as follows. Besides, the authors almost follow the classic knowledge distillation framework except for residual feature distillation. [a] CLIPPING: Distilling CLIP-Based Models with a Student Base for Video-Language Retrieval, CVPR 2023. [b] Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation, ACL 2022.

R: Thanks for pointing out the CLIP-based distillation method (CLIPPLING [1] and VLKD [2]), which helps improve the literature analysis in our paper. Below, we compare our method with CLIPPLING [1] and VLKD [2]:

• (a) CLIPPING focuses on model compression, which aims to fully transfer the knowledge of the Clip4clip (a big teacher model) to the MobileViT-v2 (a small student model). In contrast, we aim to efficiently adopt a pre-trained CLIP model, such that the resulting model maintains the strong generalization capacity of a pre-trained CLIP while being video-specific. (b) The key technical contributions of CLIPPING are a layer-wise alignment scheme and a distillation strategy based on video-text distributions. Both the two components are to ensure the student model can fully absorb the knowledge of the CLIP model. Differently, we design a residual feature distillation module emphasizing the balance between generalizability maintenance and video-specific feature learning, which goes beyond just mimicking the CLIP model. $c$ Besides, CLIPPING is only designed for a specific student model (MobileViT-v2), while our method can be widely used for student models with different architectures (refer to Tab. 3 of our paper).

• (a) VLKD concentrates on aligning the features of the language model (BART) to the CLIP model and subsequently integrating them to be a multi-modal generator. In contrast, our method is more than feature alignment. Since the tuned model in our framework is well-aligned with the CLIP model at the beginning of fine-tuning, our goal is to maintain its generalizability while learning video-specific knowledge. (b) The key technical contribution of VLKD is proposing to align the features of BART to CLIP using feature distance minimization and contrastive loss. On the contrary, we propose to balance feature learning in two different objectives (generalizability and video-specific learning) by a residual feature distillation module.

To demonstrate the superiority of our method, we evaluate the effectiveness of the VLKD method for knowledge distillation in our task. After carefully reading the CLIPPING paper, we found that its design choice is highly optimized for the specific MobileViT-V2 model in the conventional knowledge distillation setting, making it non-trivial to be modified to handle the open-vocabulary action recognition task. In contrast, our method can be easily integrated with different methods to improve their performance on open-vocabulary action recognition (please refer to Tab. 3 in the main paper).

As shown in Tab. 1 below, our method achieves superior performance than the VLKD method.

Table 1: Performance (top1-acc (%)) of different distillation methods under the cross-dataset evaluation setting.

We have included the above analysis in the section of related work of our revision.

Besides, we agree that knowledge distillation has been a widely studied topic in the field, which is part of the motivation of our method to study open-vocabulary action recognition from a knowledge distillation perspective. However, we would like to highlight that conventional knowledge distillation methods aim to distill the same knowledge from one model to another, while differently, in our case, we have two objectives, one is maintaining the generalization capability of a pre-trained CLIP and the other is effectively adapting from image to video task. Simply applying the conventional distillation methods is not sufficient for our tasks (refer to Tab. 4 of our paper), while our method achieves superior results.

Dear Reviewer zWh1,

Thanks for your valuable time and efforts in reviewing our paper. We understand that you may be rather busy during this period. However, as the discussion period will end in less than one day, we would like to kindly request your feedback on our responses. We would be happy to discuss with you in detail if you have additional comments about our paper.

Look forward to your reply.

Best regards,

Paper1809 Authors

Thanks for your time and efforts in reviewing our paper. We are delighted that our paper is found to be well-written, well-motivated, and presented a reasonable solution, along with extensive experiments.

Please find the response below addressing the novelty concern related to previous works, as well as additional experimental results provided in Tab. 1.

Q1: The proposed feature distillation falls into the framework of knowledge distillation, which has been extensively explored for adapting pre-trained models. While I agree that many prior works are insufficient for the objective of this paper, there are a few very recent papers that share the key idea of using feature distillation to adapt CLIP models for video tasks (see e.g., CLIPPING). With this prior work, the technical innovation of the paper seems a bit incremental.

R: Thanks for pointing out the CLIP-based distillation method (CLIPPING [1]), which helps us improve the reference comparison in our paper. Below, we compare our method with CLIPPING [1] and an additional CLIP-based distillation method, i.e., VLKD [2]:

• (a) CLIPPING focuses on model compression, which aims to fully transfer the knowledge of the Clip4clip (a big teacher model) to the MobileViT-v2 (a small student model). In contrast, we aim to efficiently adopt a pre-trained CLIP model, such that the resulting model maintains the strong generalization capacity of a pre-trained CLIP while being video-specific. (b) The key technical contributions of CLIPPING are a layer-wise alignment scheme and a distillation strategy based on video-text distributions. Both the two components are to ensure the student model can fully absorb the knowledge of the CLIP model. Differently, we design a residual feature distillation module emphasizing the balance between generalizability maintenance and video-specific feature learning, which goes beyond just mimicking the CLIP model. $c$ Besides, CLIPPING is only designed for a specific student model (MobileViT-v2), while our method can be widely used for student models with different architectures (refer to Tab. 3 of our paper).

• (a) VLKD concentrates on aligning the features of the language model (BART) to the CLIP model and subsequently integrating them to be a multi-modal generator. In contrast, our method is more than feature alignment. Since the tuned model in our framework is well-aligned with the CLIP model at the beginning of fine-tuning, our goal is to maintain its generalizability while learning video-specific knowledge. (b) The key technical contribution of VLKD is proposing to align the features of BART to CLIP using feature distance minimization and contrastive loss. On the contrary, we propose to balance feature learning in two different objectives (generalizability and video-specific learning) by a residual feature distillation module.

To demonstrate the superiority of our method, we evaluate the effectiveness of the VLKD method for knowledge distillation in our task. After carefully reading the CLIPPING paper, we found that its design choice is highly optimized for the specific MobileViT-V2 model in the conventional knowledge distillation setting, making it non-trivial to be modified to handle the open-vocabulary action recognition task. In contrast, our method can be easily integrated with different methods to improve their performance on open-vocabulary action recognition (please refer to Tab. 3 in the main paper).

As shown in Tab. 1 below, our method achieves superior performance than the VLKD method.

Table 1: Performance (top1-acc (%)) of different distillation methods under the cross-dataset evaluation setting.

We have included the above analysis in the section of related work of our revision.

Q2: I am a bit confused by how the results were reported for the proposed method in Table 1. Sec 4.2 (implementation details) states that “Otherwise stated, we use VCLIP for conducting experiments.” Is the proposed method (FROSTER) also built on VCLIP in Table 1? If so, VCLIP should be included as a baseline here. If not, a description should be included.

R: Thanks for the suggestion. We have included the results of VCLIP in Tab. 1 of the revised manuscript. Below, we present the results, which demonstrate that our method can improve the performances for both the base and novel categories for all datasets.

Table 2: Performance (top1-acc (%)) under the base-to-novel evaluation setting. B, N, and HM denote the base set, novel set, and harmonic mean, respectively.

Thanks again for your valuable comments and looking forward to your reply.

[1] Pei, Renjing, et al. "CLIPPING: Distilling CLIP-Based Models with a Student Base for Video-Language Retrieval." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[2]Dai, Wenliang, et al. "Enabling multimodal generation on CLIP via vision-language knowledge distillation." Findings of Association for Computational Linguistics 2022: 2383-2395.

Dear Reviewer ZdtY,

Thanks again for your great efforts and insightful comments in reviewing this paper! With the discussion period drawing to a close, we expect your feedback and thoughts on our reply. We look forward to hearing from you, and we can further address unclear explanations and remaining concerns if any.

Regards,

Authors

We thank the reviewer for the response and address the additional comments as follows:

Q1: There is a fairly strong conceptual similarity, as also pointed out by zWh1.

R: The aim of previous methods is to mirror the representation capability from a pre-trained CLIP model as much as possible. However, in our work, the model needs to maintain the feature generalization capability as the pre-trained CLIP model and more importantly, also adapt to the downstream video action recognition task, which can not be achieved by previous knowledge distillation approaches. The model needs to learn to balance between these two distinct objectives. For instance, if we enforce the features to be overly close to the frozen CLIP features, it may hinder the video-specific learning to fit the video data. Conversely, if we overemphasize video-specific learning, the generalizability in the tuned model might be lost. Our residual feature distillation method is specifically developed to achieve both goals simultaneously, which has not been well investigated in the field.

Q2: It is perhaps not surprising that the proposed solution can work, yet it is somewhat a question if the proposed work brings new knowledge to the field.

R: It is also important to note that for the task of open-vocabulary action recognition, knowledge distillation from a pre-trained CLIP model is not enough as the model has never been trained to adapt to this task (see Fig. 1 of our paper), while directly finetuning will hurt the generalization capability of the pre-trained CLIP (see Fig. 1 of our paper). Therefore, our method addresses this problem and presents a simple but effective approach to it, achieving new state-of-the-art performance on all five popular video benchmarks, which we believe can be a very helpful reference for future research in this direction.

Thanks for your reading and hope you can take the above information into consideration for assessment.