VideoGLUE: Video General Understanding Evaluation of Foundation Models

We hope our response addresses your concerns. Should you have additional questions, we are readily available for further discussion.

• Re W1 "The proposed VideoGLUE score focuses solely on the computation-performance tradeoff."

As reviewer 1CiA commented, “The suggested VideoGLUE score can be an interesting initial way to start a comparison and discussion of the strength of foundation models”. We acknowledge that it is only an initial attempt to score FMs by their video understanding capabilities, and it is not comprehensive as we discussed the limitations (more details in Appendix A).

Nonetheless, it captures the essence and can help the design of new metrics in future work. In particular, we design VideoGLUE score (VGS) as the model performance weighted by the number of tunable parameters, which are arguably the two most important factors when evaluating an FM [1, 2].

[1] Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E. and Brynjolfsson, E., 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.

[2] Li, C., Liu, H., Li, L., Zhang, P., Aneja, J., Yang, J., Jin, P., Hu, H., Liu, Z., Lee, Y.J. and Gao, J., 2022. Elevater: A benchmark and toolkit for evaluating language-augmented visual models. Advances in Neural Information Processing Systems, 35, pp.9287-9301.

• Re W2 "Selection of tasks are limited to three tasks."

We appreciate the reviewer’s suggestion, and we have to leave them to future work from the community. As you may notice, completing the evaluation on one set of metrics in our paper requires more than one hundred experiments (six foundation models, four adaptation methods over eight datasets), not to mention the hyperparameter tuning and other explorations. Hence, we indeed made hard decisions about which to include in VideoGLUE.

We are working on a new video task/dataset. Should bandwidths permit, we will work on the second version of VideoGLUE by including the new task and investigating the tasks the reviewer suggested.

• Re W3 "Selection of motion-focused benchmarks."

We did consider Something-Else against SSv2 and FineGym against Diving and decided to go with SSv2 and Diving because they are well received by the community. We can swap them in the future if their drawbacks become an obstacle for fair evaluations.

• Re Q1 "Is there any reason why image-native FM CoCa outperforms other FMs on D48"

We note that although image-native CoCa and CLIP perform well on D48 in the end-to-end fine-tuning setting and the adapter setting, respectively, video-native FMs perform better in the frozen feature setting and the multi-layer attention pooler setting. Because D48 is small, it is unlikely to provide sufficient supervision for end-to-end fine-tuning the spatiotemporal weights in the video-native FMs. However, it still signifies the video-native FMs’ motion understanding capabilities when the backbones are frozen.

As reviewer 1CiA commented, “The suggested VideoGLUE score can be an interesting initial way to start a comparison and discussion of the strength of foundation models”. We acknowledge that it is only an initial attempt to score FMs by their video understanding capabilities, and it is not comprehensive as we discussed the limitations (more details in Appendix A). Nonetheless, it captures the essence and can help the design of new metrics in future work. In particular, we design VideoGLUE score (VGS) as the model performance weighted by the number of tunable parameters, which are arguably the two most important factors when evaluating an FM [1, 2].

I appreciate the efforts put into developing the VideoGLUE score as an initial step in evaluating video foundation models (FMs) as the Reviewer1CiA mentioned. However, IMHO, VideoGLUE requires further refinement to establish a comprehensive and clear benchmark, especially considering its role not just as a method but as a benchmark and evaluation protocol. For example, GLUE(NeurIPS’19) and SuperGLUE(ICLR’19), which is an improved version of GLUE, were published in the same year. The significant difference in their citations underscores the importance of a benchmark’s initial comprehensiveness and reception. As the first of its kind, VideoGLUE must ensure a robust and comprehensive foundation to garner significant attention and use.

We appreciate the reviewer’s suggestion, and we have to leave them to future work from the community. As you may notice, completing the evaluation on one set of metrics in our paper requires more than one hundred experiments (six foundation models, four adaptation methods over eight datasets), not to mention the hyperparameter tuning and other explorations. Hence, we indeed made hard decisions about which to include in VideoGLUE.

Concerning the extensive requirements of video experiments, I understand the constraints of computational resources. However, this limitation raises concerns about the benchmark’s ability to assure the generalizability of video FMs. If not, the authors at least should’ve justified how the three chosen tasks can guarantee the measure of generalizability of video FMs and why other video tasks, e.g., repetition counting, or fine-grained motion understanding, are excluded.

We did consider Something-Else against SSv2 and FineGym against Diving and decided to go with SSv2 and Diving because they are well received by the community. We can swap them in the future if their drawbacks become an obstacle for fair evaluations.

I respectfully disagree with the author's statement, especially, “We can swap them in the future if their drawbacks become an obstacle for fair evaluations.” at two points. First, the problem of spurious correlation of SSv2 has already been raised a couple of years ago, and thus, VideoGLUE should reflect current understanding and criticisms. Second, changing datasets in benchmarks is not trivial, as practitioners developing video models might rely on existing VideoGLUE scores for benchmarking, making any alterations challenging for fair comparisons.

In summary, while acknowledging VideoGLUE’s initial contribution, I urge consideration of its comprehensiveness, justification of dataset selection, and the implications of changing benchmarks in the future for a fair and effective evaluation of video FMs. I thus keep my rating.

We hope our response addresses your concerns. Should you have additional questions, we are readily available for further discussion.

• Re W1 "The diversity of these settings compromises the comparability of the experimental results, rendering the conclusions less reliable."

We acknowledge the reviewer's comment. In our experiment design, we try to compare FMs fairly, so that we strictly control our comparison in the main paper to the same model size, ViT-B, and resolutions and other hyperparameters like augmentation strategy. Pre-training data and objectives are not controllable and we note that it is also happening to the evaluation of large language models. While we cannot control the FMs' diverse pretraining settings, this work tries to standardize the evaluation so that we can examine "which FMs are good at solving video tasks, what makes them better than others in the video domain, and how to best adapt them to video understanding" (quoted from Section 3.1 in the paper).

• Re W2 "design of VideoGLUE score":

As reviewer 1CiA commented, “The suggested VideoGLUE score can be an interesting initial way to start a comparison and discussion of the strength of foundation models”. We acknowledge that it is only an initial attempt to score FMs by their video understanding capabilities, and it is not comprehensive as we discussed the limitations (more details in Appendix A). Nonetheless, it captures the essence and can help the design of new metrics in future work. In particular, we design VideoGLUE score (VGS) as the model performance weighted by the number of tunable parameters, which are arguably the two most important factors when evaluating an FM [1, 2].

[1] Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E. and Brynjolfsson, E., 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.

[2] Li, C., Liu, H., Li, L., Zhang, P., Aneja, J., Yang, J., Jin, P., Hu, H., Liu, Z., Lee, Y.J. and Gao, J., 2022. Elevater: A benchmark and toolkit for evaluating language-augmented visual models. Advances in Neural Information Processing Systems, 35, pp.9287-9301.

• Re Q1 "score on the MIT dataset":

Thanks for pointing this out. We will modify the statement to “Foundation models underperform task-specialized models on video tasks in general — MIT is the only exception.”

• Re Q2 " Why Figure 3 and Table 5 CoCa scores are consistent?"

Thank you for catching this! We made a typo in Table 5. Figure 3 is the correct one. We have updated the manuscript accordingly.

We thank the reviewer for the detailed comments and valuable suggestions. We hope our response addresses your concerns. Should you have additional questions, we are readily available for further discussion.

• Re "How does VideoGLUE address the differences among these diverse datasets?"

We choose a diverse set of video datasets/tasks on purpose. These diverse video datasets/tasks help us probe into FMs’ video understanding capabilities from different perspectives. For example, SSv2 and Kinetics emphasize motion and appearance, respectively. Video classification and temporal action localization test an FM’s holistic understanding versus temporal reasoning.

• Re "is there any comprehensive analysis or comparison involving 3D-CNN or even 2D-CNN based FMs?"

We followed the following definition of foundation models: “any model that is trained on broad data that can be adapted (e.g., finetuned) to a wide range of downstream tasks” [1] to choose the most representative image- and video-based foundation models. Interestingly, the resultant FMs happen to be all transformer-based, potentially because the model developers chose the transformer architecture considering its better scalability than CNNs [2, 3].

[1] Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg, J., Bosselut, A., Brunskill, E. and Brynjolfsson, E., 2021. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258.

[2] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S. and Uszkoreit, J., 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.

[3] https://twitter.com/giffmana/status/1717995379034071483

We thank reviewer for the detailed comments and valuable suggestions. We hope our response addresses your concerns. Should you have additional questions, we are readily available for further discussion.

• Re W1 "no novel models, modifications, or datasets presented in this paper":

We appreciate the reviewer for this invaluable suggestion as we are actively working on a new video task/dataset. We cannot include it here because of the high workload involved. As one may notice, completing this work's evaluation on one set of metrics in our paper requires more than one hundred experiments (six foundation models, four adaptation methods over eight datasets), not to mention the hyperparameter tuning and other explorations. Including new models, modifications, or datasets would be great complements to this project but also require extensively more work. We will highlight this limitation in the revised paper.

• Re W2 "The list of studied foundation models is not as comprehensive as potentially can be":

Regrettably, we have lost the priority of computing resources for this project, and we are unable to provide further results for other models. Nevertheless, the paper has included two “video-based models developed on top of CLIP paradigm”: InternVideo and VATT. Especially, VATT is more or less a “superset” of VideoCLIP suggested in the review because VATT’s pretraining data and method cover VideoCLIP’s. Similarly, InternVideo can be viewed as a “superset” of the second paper mentioned in the review.

• Re W3 "other adaptation techniques":

Thanks for pointing these out. In our exploration phase, we tried multiple adapters, including the “ST-Adapter” suggested by the reviewer. However, we were unable to reproduce ST-Adapter’s results reported in the original paper. In contrast, LoRA was the most stable and widely embraced one with superior performance, so we chose to report LoRA in the paper.