Q-Bench-Video: Benchmarking the Video Quality Understanding of LMMs

1. In my opinion, the work’s full impact is not felt in the current form of the paper. Specifically, the key observation of humans > LMMs > random choice is neither surprising nor unexpected. The same can be said of the observation that proprietary models outperform open-source models. In light of this, the study should have been extended to show how the benchmark can help improve LMM performance to reduce the gap with human performance. Without this, a practitioner would be left wanting for the utility of Q-Bench-Video.

2. Given that the original content sources contained a much larger set of videos, the size of Q-Bench-Video (1800) is too small to qualify as a representative benchmark. For example, the LSVQ dataset has 39000 videos, and the MaxWell dataset has 4500 videos. Further, several user-generated content datasets, such as YT-UGC, Konvik-1K, etc., have in excess of 1500 videos. Given that Q-Bench-Video is proposed as a benchmark composed of not only real videos but also AIG and CG videos, a representative dataset should contain at least 4500 videos.

3. Some of the key details of the work have been relegated to the appendices. For example, the video selection and evaluation strategy are both detailed in the appendix.

4. The fact that only three human subjects were used for human performance evaluation does not inspire confidence in the work. There is no information on statistical consistency tests on the subjective experiments.

5. The presentation quality needs to be improved.

Testing only small-sized LMM models. There is no indication of how the scaling law affects the model's ability to understand video quality.

The length of the VLM answer is not taken into account. Thus, if the model gave a long answer, it can receive higher score.

Technical details regarding the operation of LMM are lacking:

• Details on how exactly videos were input into each LMM are not thoroughly described, such as resolution and the number of frames (see the questions section).

1. While Q-Bench-Video includes diverse video types, there is limited exploration of how LMMs generalize across these domains. Testing LMMs on specific domains, such as medical or surveillance videos, would enhance the benchmark’s relevance by showing how well models handle domain-specific quality variations, which are critical in many real-world applications.

2. The benchmark results indicate that LMMs struggle significantly with open-ended questions, but the paper lacks a detailed breakdown of common error types or patterns in these responses. A more granular analysis could provide clearer insights into where models fail in nuanced video quality understanding and offer guidance on specific areas for model improvement.

3. The reliance on GPT-assisted evaluation for scoring open-ended responses may introduce subjectivity or bias, as the evaluation depends on the alignment of the assistant model’s judgments with human interpretation. Including human evaluations as a baseline for open-ended questions would provide a stronger reference and help validate the reliability of the GPT-assisted scores.

4. The benchmark primarily focuses on a balanced quality distribution but lacks an emphasis on evaluating LMMs in scenarios with challenging low-quality or noisy data, which are common in real-world settings. Incorporating more degraded video samples could better test the robustness of LMMs and highlight areas where models may require improvement for real-world deployment.

5. Some of the highly relevant video quality assessment works should be mentioned in the related work section, like [R1][R2][R3].

[R1] UGC-VQA: Benchmarking Blind Video Quality Assessment for User Generated Content, TIP 2021 [R2] RAPIQUE: Rapid and accurate video quality prediction of user generated content, OJSP 2021 [R3] FAVER: Blind Quality Prediction of Variable Frame Rate Videos, SPIC 202

1. Although the benchmark includes annotations reviewed by three additional participants, the performance on human evaluation is only 81.56%. This raises concerns about whether the annotations meet a robust confidence interval and whether there may be inherent biases in the data. The reliance on subjective assessments for video quality (such as aesthetics) could introduce inconsistencies.
2. The technical and aesthetic evaluations in the dataset are annotated by eight experts, but the paper does not validate these annotations against established subjective scores, such as those from the MaxWell dataset.
3. The paper provides no details on the video format and resolution used in the benchmark. It is unclear if any downsampling was applied, which may affect the quality assessment and limit the applicability of the results to different resolutions.
4. The benchmark’s evaluation of open-ended questions relies on GPT-based scoring, repeated five times for consistency. However, this method may introduce biases, as GPT may favor responses with longer outputs or struggle with certain scenarios where perfect answers are difficult to generate. This raises concerns about the objectivity and reliability of the scoring for open-ended questions.
5. The paper does not provide a comparison with other established video quality datasets to validate the reliability of its annotations.

• Motivation of the Benchmark: The paper's motivation for building the benchmark is not sufficiently strong. While the introduction states the importance of video quality for viewer experience and high-quality video generation, what viewers primarily care about is the classification of video quality (e.g., low, medium, high). A well-trained classifier could fulfill this need more directly. Additionally, for assessing video generation quality, designing robust evaluation metrics is more crucial than simply building a benchmark to test LLMs.

• Question Design Limitations: The paper’s three question categories are not comprehensive enough. It would be beneficial to include overarching questions such as "What is the quality level of this video?" with clear, concise answers like "low/medium/high." This would better align with practical applications and user expectations.

• Evaluation Metric Weaknesses: The evaluation metrics used to assess LLMs' QA performance lack robustness. The current scoring prompt only accounts for completeness, accuracy, and relevance, which are insufficient for open-ended questions. Recent studies (e.g., RAGchecker [NeurIPS 2024]) suggest including metrics like faithfulness (to detect hallucinations), truthfulness, correctness, etc. Additionally, prompts should be specifically designed with definitions and in-context examples to help LLMs understand each metric. Without refining the prompt and using comprehensive evaluation metrics, the reported performance scores are likely unreliable.