VideoEval: Comprehensive Benchmark Suite for Low-Cost Evaluation of Video Foundation Model

1. Clarity Needed: How are authors defining “Beyond Action recognition” “Task Diversity” and “Domain Diversity”, e.g. VideoGLUE Yuan et al. (2023) is restricted to action recognition while it is task and domain-diverse. (Table 1).

2. Comparison with previous work is misleading:
   a. VideoGLUE (Yuan et al. (2023)) is restricted to action recognition only, although it consists of other tasks like temporal localization, and spatiotemporal localization, while this work counts “Medical Surgery” (medication action recognition) as separate task from normal action recognition.
   b. BEAR (Deng et al. (2023)) is counted as only action recognition while it does “anomaly (violence) classification” which this work counts as separate content moderation. Similarly, their “Instructional Video” classification is counted as a separate task as AI for Science via “Medical Surgery”. Their “Gesture” recognition is similar to “Emotion Analysis” and “Animal Behavior” which are counted as separate tasks in this work.
   c. VidTAB is classification only (albeit different datasets). Temporal action localization is a much harder and a diverse task compared to classification. [Line 87-88] “Focus on action recognition, overlook other video understanding scenarios” This motivation is weak. Maybe reword this?

3. Figures are difficult to understand: Figure 1 can be easily summarized and it is occupying too much space. The same applies for Figure 2. Comic San font is hard to read and difficult to differentiate what part of the figure is important and what part belongs to the setup environment. Generally, these fonts denote dataset specific / setup, while methodology / proposed technique is in default font (but its opposite here?). Table 2 (and Table 3) are unreadable. So small font size. Put it in supplementary and expand it?

4. Choice of words ‘Evaluation’ vs ‘Comparison’: [Line 020-021] “evaluating the task adaptability of VFMs under few-shot conditions”. Benchmarks traditionally evaluate the models on a variety of tasks. Comparison among models is the byproduct of these benchmarks.
   (a)[line 232-233] “exclude datasets with videos that have low resolution”,
   (b) [line 250-251] “the number of categories is too high, models often perform no better than random guessing”.
   These preprocesses are mainly done for model comparison and not in the spirit of true evaluation.

5. Authors seemed to have missed the technique of “Visual Prompt Tuning” (Jia, Menglin, et al. "Visual prompt tuning." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.) for “Identifying efficient adaptation method for evaluation”? It just adds 50 spatial tokens to adapt to models for various fine-tuning tasks.

6. I’m doubtful if [Line 107] Low-cost is truly a contribution. This study deals with foundation models which are especially good in zero-shot evaluation. It's unfair to count it as a contribution when dealing with such models, and comparing traditional benchmarks with traditional models. Additionally, What’s “’K710ft” Kinetics-710 (0.66M) is this few shot? 0.66M is not low cost as indicated in Figure 1.

7. Space allocation is unconventional. For a benchmarking paper, the setup for benchmarking takes the majority of space, while the benchmarking analysis starts at the 8th page (out of 10 pages) with minimal analysis. It's not well-reasoned why image models are included when benchmarking video models.

8. [LINE 424-425] “The adaptation performance of models generally increases with the growth of data and model size” This is not necessarily true and requires a deeper analysis.
   (a). Within similar models (i) “VideoMAEv1-H” has 651 M parameters with superior performance to “VideoMAEv2-g” (1037M). (ii) “ViCLIP-L-200M” (200 M pretraining) is inferior to ViCLIP-L-10M (10 M pretraining) on “Harmful Content”, and “Quality Assess”.
   (b). Outside the model the number of contractions is far greater. E.g. VideoMAEv2-g < UMT-Lstage1.

It seems to me that there is a disconnect between the discussion of the results and the actual results. Conclusions drawn are not supported by the data while some clear trends in the data are not discussed. While I would not recommend this paper for acceptance in its current state, I would be in favor of accepting it if the weaknesses above can be addressed sufficiently.

• The main weakness of the paper is that many of the conclusions it draws are not fully supported by the experimental results.
  • On VidTAB:
    • l. 421: “current vision FMs struggle to adapt to unseen video tasks”. What is the evidence for this statement? A comparison of VFM performance to SotA on these datasets would help support this.
    • l. 422: “VFMs outperform IFMs”: The best IFM beats 9 out of the 12 evaluated VFMs, so this statement doesn’t seem to be generally true, and also not in action and behavior tasks where some IFMs show stronger performance than most VFMs. So this statement should be weakened and results discussed in more nuance. This is actually an interesting negative result that warrants further discussion.
    • l. 431: “Similar findings are observed with ViCLIP-L, where post-pretraining on a large-scale video dataset improves Action-related tasks but diminishes performance in other domains (Science, Safety, Quality, Emotion)”: This statement is not supported by results in Tab. 5: VICLIP-L performance actually increases on science tasks, only slightly drops on one safety task while improving on the other, and increases on emotion.
  • On VidEB:
    • Conclusion (1): “The contrastive learning (CL) based approach consistently excels in embedding evaluation.”: This does not seem to be the case as two MVM methods perform better than the best CL method.
    • Conclusion (2): “The effectiveness of masked video modeling is closely tied to the targets it reconstructs or aligns with.”: This statement is hard to understand without context about the training data. What are the targets? How is “higher semantic richness” introduced?
• Results also suggest conclusions that are not discussed in the paper. These should be addressed and discussed in more detail:
  • VidTAB: A missing insight is that VFMs show no gain over IFMs on spatial tasks (Safety and Quality).
  • Image-to-video methods do better than VFMs on most tasks.
• From the evaluation results on both benchmarks it is not clear what the gap between VFMs and state of the art on each dataset is since results on specialist models on each task are not given.
• There are significant differences in results between the VidTAB and VidEB benchmarks that are not explained.
  • Why does stage 2 training improve UMT-L and InternVideo2 performance on VidTAB but degrades performance on VidEB?
  • Similarly, why does fine-tuning on K710 increase performance on VidTAB but hurts performance on VidEB?
• The related work section does not discuss image foundation models and image-to-video adapter approaches. It would be good to at least discuss those approaches that are evaluated. This would allow readers who are less familiar with the literature in this area to better interpret the results.
• Evaluation of task adaptation is limited to one method (attentive probe).
• Future research directions are not clearly pointed out, though some ideas are given in the results discussion. A separate section on this could help direct future research.

The paper did not evaluate any of the large multimodality models such as GPT-4V, Gemini, video-llava, etc. It seems that the paper regards "video foundation models" as "video encoders", and the benchmark can only be used to evaluate video encoders.   The paper focuses on classification tasks. Modern video understanding tasks have shifted from classification to more fine-grained understanding such as (dense) text description, video question answering, etc.

Only a few (8 vidTab, 16 vidEB) frames are selected from each video. The benchmark won't be able to evaluate long video understanding capabilities, and it won't be able to capture the speed of motion.

• Justification of dataset selection 1: In VidTAB, why "Action Recognition in Special Scenarios" are focused instead of accessing general action recognition? Furthermore, it is not clear to me why "dark scene" is representative in action recognition in special scenarios. More justifications like reference to related works are needed.
• Justification of dataset selection 2: I have concerns regarding the validity and rationale behind the dataset choices in VidEB, particularly the focus on scene retrieval tasks. The superior performance of image-based foundation models on these benchmarks suggests a potential bias towards static scene understanding, which may not fully capture the desired motion and dynamic understanding capabilities expected in a video context.
• Following data selection 2: as the evaluation sets are towards static scene understanding, I found it hard to ground the observations and conclusion in VidEB section on video foundation models. Maybe one way to improve the task is to re-purpose some video-centric dataset for embedding retrieval task, e.g. using nearest neighbor retrieval for zero-shot classfication.
• Model sizes in the comparison in the paper: in the main table (Tab 5, 6), foundation models of different sizes are put together to compare. As it largely follows the intuition that the larger the model sizes the better the performance, I found it hard to get insights when comparing models of different size in the table.
• It is strange to see the "zero-shot performance of visual languange models" in table 5. With TA-score proposed in eq. 1, how the score on zero-shot V-L model performance is calculated? as I understand V-L retrieval does not involve in few-shot examples.
• [minor] Table 2 characters are too small to read.