TVBench: Redesigning Video-Language Evaluation

Thank you for the time and effort spent in reviewing our paper. In the following, we reply to all the concerns raised in the review.

W1 – Discussion on TVBench and future evaluation benchmarks: Thank you for highlighting this. We believe that new benchmarks must be created every few years to keep pace with the rapid advancements in AI. However, our benchmark, TVBench, is far from being solved. Even the best-performing method, Tarsier-34B, achieves only ~20% above the random baseline, while many recent models, such as mPLUG-Owl3, GPT-4o, VideoGPT+, and PPLaVA, perform close to random chance, with less than a 9% improvement. In TVBench, we source videos for each task from various existing datasets (see Section 5.2), including Perception Test, CLEVRER, STAR, MoVQA, Charades-STA, NTU RGB+D, FunQA, and CSV. TVBench thus includes a diverse range of scenarios, featuring both first- and third-person perspectives, indoor and outdoor environments, and real and synthetic data. It comprises 2,654 QA pairs across 10 different tasks, ensuring robust coverage of various temporal challenges. This diversity is essential for creating a benchmark that truly assesses temporal reasoning capabilities. For video QA examples of TVBench, please refer to Appendix A.4.

W2 – Details on Task Selection: Our benchmark, TVBench, draws inspiration from existing datasets [1, 2] to cover different skill areas—Memory, Abstraction, Physics, and Semantics—through 10 selected temporally challenging tasks: repetition counting (Action Count); properties of moving objects (Object Shuffle, Object Count, Moving Direction); temporal localization (Action Localization, Unexpected Action); sequential ordering (Action Sequence, Scene Transition, Egocentric Sequence); and distinguishing between similar actions (Action Antonyms). We finalized these 10 tasks by verifying that they are free from spatial and textual biases, unlike previous benchmarks.

--- continued in next comment ---

Clarification of Figure 1

Figure 1 presents a summary of our paper’s main results. We have updated this figure by incorporating additional models (see W2) and clarified the right side by showcasing only the top-performing video model, Tarsier 34B, for comparisons between MVBench and TVBench under video, shuffle, and reverse settings.

Right side: Examines how performance changes when videos are shuffled or reversed on MVBench versus TVBench. On MVBench, altering the frame order results in only minor performance variations, indicating that frame order is not critical for solving MVBench tasks. In contrast, TVBench performance significantly drops when videos are shuffled and drop even further below random chance when videos are reversed. This demonstrates that TVBench requires strong temporal understanding to be solved.

Left side: TVBench serves as a strong temporal benchmark, with only a few frontier models surpassing the random chance baseline. In contrast, MVBench exhibits a continuous performance progression across all models. Notably, models that rely solely on a single random frame achieve nearly 50% accuracy, while those based only text (question) reach approximately 40%. These findings raise concerns about what MVBench truly measures, as strong performance can be achieved without genuine temporal understanding.

W2 – Adding More Video Models to the Benchmark: We agree with the reviewer that TVBench can benefit from incorporating additional models. Therefore, we have included the suggested models such as GPT-4o with videos, VideoLLaVA, mPLUG-Owl3, VideoLLaMA2 7B, VideoLLaMA2.1 7B, VideoLLaMA2 72B, Qwen2-VL 7B, and Qwen2-VL 72B with more models, e.g. PandaGPT and LLaVA-Next, in the coming days. To ensure the quality of our benchmark, we have also included a human baseline. For more details, see Appendix A.2.2.

W3 – Presentation: Thank you for the detailed feedback on our paper. In response to your suggestions, we have made the following updates to our revised PDF:

• Minor Adjustments: Added the axis to Fig. 1 and corrected the reference to Table 1 to improve clarity and accuracy.
• Reducing MVBench examples: We removed one of the three examples of MVBench in Fig. 2
• Human Baseline: Included a human baseline to verify the quality and solvability of our benchmark.
• Benchmark Examples: Added examples of TVBench for all tasks in Appendix A3 to give more insights.
• Model Consistency: Ensured consistency by including the same models across all modalities in Tables 1, 2, and 4, facilitating direct comparisons.
• Benchmark Creation Details: Clarified the strategies for creating the benchmark in Section 5.1, including the addition of Figure 7 to visually represent our process.
• Extended MCQA Analysis: Expanded our analysis beyond MVBench by also evaluating NextQA, as detailed in Appendix A.3, exhibiting the same problems of spatial and temporal biases.

Importance of Section 4 (Open-Ended VQA): Currently, video-language models are evaluated using two methods: multiple-choice and open-ended VQA. After examining the widely used MVBench benchmark for multiple-choice tasks—and extending our analysis to NextQA in Appendix A.3—we found that spatial and textual biases also exist in open-ended VQA. Additionally, relying on closed-source proprietary LLMs for evaluation introduces unreliability. This discovery motivated us to focus on designing a multiple-choice VQA benchmark rather than a new open-ended VQA. As appreciated by Reviewer TPmi, we believe that highlighting these issues in open-ended VQA is beneficial for the community.

Thank you again for reviewing our paper and providing constructive feedback. We hope that our responses have addressed your concerns and that you will consider increasing your score.

Thank you for the time and effort spent in reviewing our paper. In the following, we reply to all the concerns raised in the review.

W1 – Experiment consistency: Thank you for addressing this. Below, we respond to your concerns in detail:

Model consistency:

Tables 1 and 2 examine the spatial and temporal biases of MVBench. We have updated these tables in response to your feedback to include the same models across all settings, enabling direct comparisons.

Table 1: Examining the spatial bias of MVBench using the same models

With the updated table, the message becomes clearer. Models that receive only a random frame as input demonstrate strong performance across all four tasks, surpassing the random baseline. Notably, GPT-4o achieves the highest average performance of 62.8% across these tasks, nearly matching its video-based performance of 65.8%. Overall, GPT-4o gets an average accuracy of 47.8% across all 20 MVBench tasks, which is 20.5% higher than the random baseline of 27.3% (see Table 4). This suggests that a significant portion of the benchmark is influenced by spatial bias. Additionally, shuffling the videos has minimal impact on the performance of all video-language models, with an average difference of only 2.3%, indicating that frame order is not crucial for solving these tasks. This problem goes beyond the four tasks analyzed here. As shown in Table 4, Gemini 1.5 Pro and Tarsier achieve average accuracies of 60.5% and 67.6% across all 20 MVBench tasks, respectively. Shuffling video frames results in performance drops of merely 3.8% and 6.4%, respectively, highlighting that spatial bias affects not only the tasks discussed in this table but the entire dataset. Additionally, we verify the agreement between the correct responses of Tarsier 34B across modalities: 91.0% between image and video inputs, and 93.9% between video and shuffled video. This confirms that current models heavily rely on spatial biases to solve MVBench.

Table 2: Examining the textual bias of MVBench using the same models

With the updated table, our findings show that models based on text-only can effectively eliminate incompatible candidates, significantly outperforming the random baseline. Notably, models using only text achieves results comparable to video-language models across these four tasks. For example, Gemini 1.5 Pro attains an average performance of 62.3% with text-only input, versus 63.7% when using videos. This trend extends beyond the four tasks, as Gemini 1.5 Pro reaches an average performance of 38.2% across all 20 tasks, which is 10.9% higher than the random chance baseline of 27.3%. We have identified three key sources of this textual bias in the paper. Additionally, we verify the agreement between the correct responses of Tarsier 34B across modalities: 85.3% between text and video inputs. This confirms that current models like Tarsier heavily rely on textual biases to solve MVBench.

Furthermore, we have expanded Table 4 to include Gemini 1.5 Pro and Tarsier 34B across all settings (text-only, image, video shuffle, video reverse, and video) for both MVBench and TVBench. Additionally, we examined the spatial and textual biases on another multiple-choice QA benchmark, Next-QA, confirming the same biases observed in MVBench. For more details, see Appendix A.3.

--- continued in next comment ---

Textual Bias:

• MVBench exhibits a strong textual bias as models using only text (question) achieve competitive results compared to those with video input across the four tasks in Table 2. For instance, Gemini 1.5 Pro (text-only) achieves an average performance of 62.3% nearly matching its video-based performance of 63.7%. This issue goes beyond the four tasks, as Gemini Pro 1.5 (text-only) achieves an average performance across all 20 tasks of 38.2%, which is 10.9% higher than the random chance baseline of 27.3%, see Table 4.
  • On TVBench, models with only text, such as Gemini 1.5 Pro, only improve 0.3% above the random chance baseline, verifying that questions cannot be answered without the corresponding videos.

The new tables can be found in the updated PDF. We further improved our benchmark by evaluating the newest video language models such as Qwen2-VL, Video-LLaMA2, Video-LLaMA2.1, Video-LLaVA, mPLUG-Owl3 on our benchmark.

L1 Standard template QA: In the era of modern LLMs, providing unnecessary context in questions or answer candidates can lead to mis-evaluations. We observed this in MVBench, where LLMs were used to create answer candidates, some of which are non-sensical. Hiring annotators to propose answer candidates is however expensive. Therefore, we decided to adopt standard templates specifically designed for each of the 10 tasks to eliminate any spatial or textual bias. With this approach, we created a temporally challenging benchmark in which the most recent methods achieve only 53.7%, representing an improvement of ~20% over the random chance baseline.

Q2 Reminder of MVBench:

MVBench is not yet saturated, why TVBench?

Thank you for pointing this out. While MVBench is not saturated, it is unclear what it truly measures, as it exhibits both strong spatial and textual biases. For instance, in Fig. 3, example 5, the Episodic Reasoning task requires extensive world knowledge to answer the question about the TV show but lacks any temporal component. Similarly, the Fine-grained Action Recognition task in Fig. 2 only requires image understanding of the bathtub, with no need for temporality. This issue extends beyond individual examples, as image-based models achieve nearly 50% accuracy on MVBench. In summary, MVBench assesses various aspects of multi-modal understanding but fails to specifically measure temporality. This limitation motivated us to develop TVBench, explicitly designed to evaluate temporal understanding, as confirmed through experiments such as shuffling, and analyzing image and text accuracies.

Cleaning MVBench for a larger benchmark?

The problems with MVBench are more structural than simply cleaning bad examples. To address this, we analyzed MVBench and selected tasks suitable for a temporal benchmark—those that are unsaturated and temporally challenging—but required new templates for generating question-answer pairs to mitigate spatial and textual biases. We also sourced videos from original datasets like Perception Test, CLEVRER, and STAR, designing new task-specific templates based on the original annotations. Below are the tasks we retained from MVBench but redesigned:

• Object Shuffle (OS):This task effectively evaluates temporal understanding. QA pairs are sourced from the Perception Test and supplemented with additional pairs to ensure balanced answer distributions, avoiding correlations that might bias models.
• Action Count (AC): Similar to OS, this task evaluates temporal reasoning but suffers from imbalanced answers (e.g., one answer appears 45% of the time). We balance the dataset by adding more QA pairs from the original source.
• Action Localization (AL): QA pairs from MVBench are reused but filtered to remove correlations between question verbs (e.g., "open") and answers (e.g., "At the beginning of the video"). The set is rebalanced to minimize textual bias.
• Moving Direction (MD): Additional QA pairs are sourced from CLEVRER, excluding stationary object answers to ensure temporal reasoning is required. MVBench's ChatGPT-based QA generation strategy is avoided.
• Scene Transition (SC): Videos are reused, and QA pairs are rephrased to include only challenging candidates. Options are restricted to scenes that appear in the video, reducing spatial bias.
• Action Sequence (AS): Videos are sourced from STAR, as in MVBench, but answer options are limited to actions that actually occur in the video to address spatial bias.
• Unexpected Action (UA): Videos from FunQA are used with a new QA generation strategy. Instead of describing actions, the task requires localizing unexpected actions, avoiding biases from ChatGPT-generated QA pairs.

--continued--

For a comprehensive temporal benchmark, we introduced the following tasks to expand the benchmark:

• Object Count (OC): Videos are sourced from CLEVRER, similar to the Moving Count tasks in MVBench. However, unlike MVBench, ignoring the temporal aspects of the question leads to incorrect answers in TVBench. Answers are also balanced to ensure fairness.
• Action Antonym (AA): While MVBench includes an Action Antonym task, we have completely reformulated it. Videos are sourced from a different dataset (NTU RGB-D instead of PAXION), and temporally opposed candidates (e.g., "sitting down" vs. "standing up") are generated, replacing textually opposed ones, as shown in examples 1 and 2 of Fig. 3.
• Egocentric Sequence (ES): Videos are sourced from the CSV dataset (not utilized in MVBench), featuring detailed action sequences recorded from a first-person perspective. To evaluate temporal understanding, negative candidates are created by reordering the correct sequence of actions.

L2 Absence detection: Thank you for pointing this out. Indeed, correctly classifying the presence or absence of an object requires analyzing all frames of a video. However, this can be achieved by treating each frame independently, as this task does not require temporal understanding of the video. State-of-the-art models like Tarsier-7B and Tarsier 34B effectively solve this task, achieving accuracies of 95.0% and 96.5% on the Object Existence (OE) task in MVBench, respectively. Thus, in order to track progress in challenging video understanding cases, we have chosen to not incorporate this task into our benchmark.

Thank you again for your feedback! We hope that our answer addresses your concerns about experimentation and that you will consider raising your score.

Thank you for the time and effort spent in reviewing our paper. In the following, we reply to all the concerns raised in the review.

W1 - Human baseline: Thank you for pointing it out. We used 14 annotators to establish a human baseline for TVBench. Each annotator labeled around 30 videos. The overall accuracy of the human baseline for our benchmark is 95%. In our updated Appendix we have also computed the statistics for the mean error for this task: 4.5% using finite population correction, thus verifying the quality and solvability of our benchmark.

W2 & W3 & Q1 Model consistency: Thank you for your suggestion. We updated Tables 1, 2, and 4 using the same models (GPT-4o, Gemini 1.5 Pro, and Tarsier 34B) across all modalities enabling a direct comparison.

For convenience, we attach the updated tables here:

Table 1: Examining the spatial bias of MVBench using the same models

Table 2: Examining the textual bias of MVBench using the same models

Table 4: Benchmark overview with the same model. For a full table with all models and all TVBench tasks please see the updated PDF. We omitted GPT-4o for shuffle and reverse for cost reduction.

With that, our message becomes even clearer:

Spatial Bias:

• On MVBench, models that receive only a random frame as input demonstrate strong performance across all four tasks, surpassing the random baseline (Table 1). Notably, GPT-4o achieves the highest average performance of 62.8% across these tasks, nearly matching its video-based performance of 65.8%. This issue extends beyond these four tasks, as GPT-4o attains an average accuracy of 47.8% across all 20 MVBench tasks, which is 20.5% higher than the random baseline of 27.3% (Table 4). This indicates that a significant portion of the benchmark is influenced by spatial bias. Similarly, for Gemini 1.5 Pro and Tarsier.
  • In contrast, on TVBench, GPT-4o with a random frame performs close to random chance, achieving only a 1.7% improvement over the baseline. This verifies that TVBench is a strong new benchmark that cannot be solved with a single random frame.
• When videos in MVBench are shuffled or reversed, the performance of top models like Tarsier-34B and Gemini 1.5 Pro remains largely unchanged. Specifically, Tarsier-34B maintains performance levels of 61.2% and 67.7%, which are 33.9% and 40.4% above the random baseline, respectively (Table 4). This consistency indicates that the order of frames does not matter for MVBench. -In contrast, shuffling and reversing videos in TVBench result in substantial performance declines of 38.0% and 27.2% which is only 4.7% better and even 6.1% worse than the random baseline respectively. These results demonstrate that frame order is crucial for TVBench, as reversing the sequence of frames leads to incorrect answers.

--- continued ---

Thank you for your comment.

We redesigned the tasks in TVBench from the ground up to ensure they are inherently temporally challenging, rather than relying on state-of-the-art methods to identify “hard” samples in existing benchmarks. Consequently, some methods perform at random baseline levels, while only those with robust temporal reasoning, like Gemini, outperform this baseline. Additionally, reversing the frame order causes models like Gemini to perform below random levels, indicating that our benchmark requires temporal understanding. This highlights a limitation of the filtered MVBench subset, where performance reflects random chance rather than the method’s (e.g., Gemini’s) capabilities.

Thank you for the time and effort spent in reviewing our paper. In the following, we respond to your comments and how they have further strengthened our submission.

W1 & Q3 - Problems also present in other benchmarks?: In Section 3, we focused on MVBench as it is widely used for multiple-choice video question-answering. In Section 4, we broadened our analysis to three other benchmarks (MSVD-QA, MSRVTT-QA, ActivityNet-QA) for open-ended video QA. We found these benchmarks exhibit similar issues as MVBench (e.g. video shuffle and video perform almost similarly well on ActivityNet-QA (see Table 3). Moreover, we show that these can be unreliable due to reliance on closed API LLMs for evaluation. Additionally, we have now analyzed the NextQA [1] multiple-choice benchmark (see Appendix A.3), which shows the same pattern of strong textual and spatial biases:

As shown, Tarsier 34B achieves an accuracy of 71.3% using only a single image, nearly matching its 79.0% accuracy with full video input, indicating a strong spatial bias, similar to what we observed with MVBench. Also, shuffling or reversing video frames does not impact performance, similarly to MVBench, demonstrating temporal frame consistency is not needed.

W2 & Q2 – TVBench Examples: Thanks for pointing this out. In Appendix A.4, we add Fig. 7-13 providing 45 examples from our benchmark. In addition, we also provide more examples of the spatial and textual bias in MVBench in Fig. 14-25.

W3 – Task Design: Thank you for highlighting this point. It is possible that selecting the correct two frames is sufficient to address the Scene Transition task. However, the model must accurately identify and interpret these frames, including their order, making it a temporal challenge. To verify this, we report the performance of the leading method, Tarsier 34B, on TVBench as follows:

We find that TVBench cannot be effectively addressed using two random frames. Although scene transition might appear to be a straightforward task, many recent models struggle with it even when given the entire video, performing nearly at chance levels—for instance, GPT-4o achieves 39.1%, which is only 8.8% better than random guessing. Additionally, we evaluate the average performance on MVBench using two random frames with Tarsier 34B:

These results on MVBench indicate that using two random frames significantly improves performance by 11.4% compared to a single frame, approaching the video-level performance of 67.6%. In contrast, on TVBench, the performance with two random frames remains close to random chance. This further underscores the necessity of a challenging temporal benchmark like TVBench.

--- continued in next comment ---