VaQuitA: Enhancing Alignment in LLM-Assisted Zero-Shot Video Understanding

1. Recently, some video datasets have been released to test long or short video understanding and reasoning, such as video-mme, video-vista, and MVBench. These datasets can better test the overall performance of models.

2. Some video-llms should be compared, such as Video-CCAM-v1.0, CogVLM2-Video-Chat, LLaMA-VID, MiniGPT4-Video, and others.

1. The experiments are insufficient. The VideoChatGPT Benchmark also includes TGIF-QA and Video-based Text Generation Performance Benchmarking to evaluate the performance of MLLMs across five aspects, such as correctness, orientation，temporal understanding, and so on. However, the paper lacks these experimental results.
2. The experimental results are not impressive. Furthermore, the LLM employed in this paper is based on LLaMA2, while VideoChatGPT utilizes Vicuna. The inherent performance disparities between different LLMs undermine the persuasiveness of the experimental results due to the absence of a consistent LLM. It would be beneficial for the experimental tables to specify the exact LLM used, along with the number of parameters.
3. The paper proposes utilizing a Video Perceiver and a Visual-Query Transformer to extract text-guided video embeddings. Is there any way to verify that this approach effectively aligns visual and textual features as expected? Additionally, the experiments lack comparative results with similar methods, such as the QFormer of BLIP2.

1. The Method section is confusing at times. E.g. Figure 1. - it is unclear from the diagram that in the Data Alignment stage the frames are first being encoded through CLIP to compute Clip similarity to the question.
2. Tables in Sec. 4 are hard to read, especially the numbers in Table 4.
3. The discussion on CLIP sampling efficiency is missing, e.g. I’d be interested to see what the computational cost is.
4. Multi-round conversation tasks is very interesting but lacks quantitative evaluation.
5. Regarding prompt engineering - the suggested prompt might just be effective for one particular model, hence leaving the contribution of the paper less significant.

1. The evaluation dataset/metrics seem to be sub-optimal and might not be sufficient. The proposed method only focuses on three old open-ended video question-answering datasets with ChatGPT-based evaluation. Some datasets like MSVD/MSRVTT have a strong single-frame bias as shown in some previous studies [1], so evaluating those benchmarks might not be sufficient, while recent video-llms more focus on video QA datasets with longer/more complex video, like video-mme/next-qa. Also, the ChatGPT-based evaluation is problematic, even it has been adopted by some previous works, it might lack solid human agreement study, thus it is not a very reliable/stable metrics, and will even give different responses with the same input according to some related research [2]. So multi-choice video QA tasks (e.g. video-mme, MVLU, Nextqa, star, TVQA) might be necessary to fully understand and validate the proposed method.

2. According to my understanding, the proposed model still pre-trains/fine-tunes the alignment modules on instructional video data, so why this can be claimed as a zero-shot video understanding? The zero-shot is only for downstream video QA tasks, so this claim is bit weird from my perspective. Could you make me clear on this?

3. What is the trade-off between efficiency (running speed/memory) and effectiveness (accuracy) for using CLIP selected frames during inference

4. Is the conclusion in the 'Prompt Engineering Design' section supported by experiments with different LLM backbones? or it is only specific to llama2/llava 1.5?

5. missing baselines, including but not limited to [3]

[1] Revealing single frame bias for video-and-language learning. ACL23.

[2] Freeva: Offline mllm as training-free video assistant.

[3] Slot-VLM: Object-Event Slots for Video-Language Modeling. NeurIPS24.

Unclear Rationality of the Technical Solution: While the concept of adaptive frame sampling is reasonable, I question the technical approach of inferring the distance between a query and a video using the CLIP score. This concern arises for several reasons:

• Lack of Semantic Grounding: A significant portion of queries are not semantically grounded in video frames. For example, a query like “describe the video” does not directly correspond to specific visual elements.
• Influence of Nonexistent Objects: The similarity measure may be affected by objects mentioned in queries that do not exist in the video. For instance, given a video without a cat and a query “Does a cat exist in the video?”, the selected frame might be misleading.
• Limitations of Pretrained CLIP: A pretrained CLIP model functions similarly to a visual bag-of-words model, capable of recognizing objects, attributes, and simple interactions. However, it struggles to model words related to video dynamics, potentially reducing the accuracy of the CLIP score in such contexts.

Insufficient Evaluation: Building on the previous point, I am concerned about evaluating the accuracy of the CLIP-based frame selector using questions that involve the existence or nonexistence of certain elements, such as those question format in POPE[1]. A more comprehensive evaluation is necessary, incorporating queries that reflect realistic scenarios (e.g., conversations, open-ended QA) rather than solely relying on complex, specially designed evaluation queries. Additionally, including more visualization results for long videos would be beneficial. It is also puzzling that the performance on longer videos (e.g., ActivityNet-QA) shows less superiority over state-of-the-art methods compared to shorter videos (MSVD/MSRVTT).

[1] Li et al., Evaluating Object Hallucination in Large Vision-Language Models, 2023.