TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models

Insufficient Text-Only Baseline in Table 5: The current text-only baseline presented in Table 5 may not fully capture the model's reasoning capabilities. Converting video content into textual descriptions for input into a model like GPT-4 would offer a more meaningful comparison with visual inputs. Including this enhanced baseline would provide deeper insights and enrich the discussion in Table 5.

We hope to respond to this comment from three aspects.

(1) In our pilot studies, we found that having models generate frame-by-frame descriptions when providing the questions and options does not yield satisfactory results, mainly because of the lack of granularity in the descriptions. An example description is like below:

Frame 1: A person is standing next to a round table with a notebook, a tennis ball, an orange, and a bottle on it. The person is wearing a striped shirt and beige shorts.
Frame 2: The person is now holding a notebook and placing it on the table.
Frame 3: The person is holding a tennis ball and placing it on the table.
Frame 4: The person is holding a notebook and placing it on the table.
[...The above line repeated for frames 5-16…]

(2) In our experiments, we instructed the models to output their explanations to their answers, serving the purpose of generating descriptions. We found that while GPT-4o often provides the correct descriptions for the transition between consecutive frames, it still fails to interpret across all frames as a continuous sequence. For example, GPT-4o outputs accurate descriptions of the movement of a moon around earth being upper-right, lower-right, lower-left, and then upper-left, but still concludes that such movement is counter-clockwise. Therefore, in response to your comment, we added an analysis paragraph in Section 6.3 titled Models lack the basic ability to interpret frames as a continuous sequence. Other example outputs can be found in Appendix I – N; the prompts used can be found in Appendix D1.

(3) We also conducted experiments in converting video content into textual descriptions first using Qwen2-VL-7B (due to limited compute to run Qwen2-VL-72B), and the results from this "enhanced text-only baseline" indicate that model performance is significantly worse across all reasoning types compared to using video input:

---

Will the benchmark be publicly available?

Yes, the benchmark will be made publicly available upon publication.

---

How do you generate the queries for each task in the benchmark? Please include this information in the main paper, which may be inspiring to the readers.

Each task is composed of a video, a question, and 4-6 options. Section 3.3 (Section 4.3 in the updated manuscript) illustrates the question-answer annotation and quality check process. We have now added additional clarifications to this section.

---

Why not reporting the results of open-source models after fine-tuning them on the temporal reasoning questions. e.g, a training set of the proposed benchmark? It is possible that the model learns to reason from dynamics in the video after fine-tuning. Please justify this.

We’d like to clarify that our benchmark is evaluation only, following the approach adopted by many contemporary benchmarks [1][2][3]. Traditional training-testing benchmark setups are less common, as models are increasingly evaluated in zero-shot or few-shot settings to better reflect real-world applications and the current state of foundation model development. Introducing a training set and then fine-tuning models on that set would shift the focus from evaluating general and out-of-the-box reasoning capabilities to testing model performance after domain-specific adaptation, which is outside the scope of our benchmark.

[1] Wang, Zirui, et al. "Charxiv: Charting gaps in realistic chart understanding in multimodal llms." arXiv preprint arXiv:2406.18521 (2024).
[2] Yue, Xiang, et al. "Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[3] Lu, Pan, et al. "Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts." arXiv preprint arXiv:2310.02255 (2023).

Thank you for taking the time to review our work and providing such insightful feedback. We would like to address your concerns and questions as follows:

---

Sensitivity in Mathematical Formulation: The mathematical formulation of the three principles appears sensitive to certain parameters, particularly the choice of human-picked frames and the number of input frames. For instance, using 16 frames as the multiple-frame input seems arbitrary; different frame counts could significantly impact evaluation scores and model comparison across benchmarks. This choice requires further justification, along with ablation studies on multi-frame input numbers within the three principles, to confirm robustness and fairness.

In response to this common concern, we have now extended our analysis to study m uniformly-sampled frames (rather than just m=16) to better assess variations in video content and duration. Given the relatively short video duration and homogenous content within each second, it is very likely that a highly informative frame is sampled as the number of sampled frames increases. Our study across m = 1, 8, 16, 24, and 32 demonstrates that all performances stabilize as m approaches 16, as shown in Figure 4 in Appendix E.3 of the updated manuscript. Therefore, our choice of m = 16 is an outcome of experiments rather than an arbitrary one. The results show that m = 16 provides a sufficient window for models to perform temporal reasoning.

Our handpicked single frame setting is to illustrate: if a model performs equally well (or nearly so) using handpicked single frames compared to how models typically process videos (uniformly-sampled multiple frames), it strongly suggests that it is not the additional frames that helped with answering and the question does not require temporal reasoning.

---

Data Imbalance Across Tasks: The sample sizes across the six tasks are imbalanced, with some categories, like Visual Cues, containing only 70 questions—significantly fewer than other categories. This imbalance could impact the benchmark's representativeness and reliability. Addressing this imbalance would improve the benchmark's overall consistency and rigor.

The Visual Cues category indeed contains fewer questions (70) compared to other categories. These questions were carefully curated through an extensive manual inspection of over 8,000 questions from the Music-AVQA dataset. Expanding this category further poses significant challenges, as identifying and verifying appropriate videos for this specific reasoning type requires substantial human effort and domain expertise. Despite the smaller sample size, these questions effectively encapsulate the core characteristics of this reasoning type.

---

Fairness in Benchmark Comparisons: Comparisons with other benchmarks may not be entirely fair due to the random sampling of 200 QAs for evaluations. This approach may inadvertently filter out complex temporal reasoning questions. To ensure fairness and maintain the benchmark's integrity, evaluating all QAs from other benchmarks would be preferable.

Thank you for your suggestion. To address fairness in benchmark comparisons, the selection of 200 QAs for each benchmark is done through proportional random sampling across all reasoning types (if applicable). This strategy is designed to provide a representative subset that reflects the overall characteristics of each benchmark.

References to the visual temporal reasoning principles are needed. Some principles discussed in the manuscript for temporal reasoning in video understandings have been studied and revealed in previous literatures, e.g. "Single Frame Insufficiency" has been discussed in [1]; "Frame Order Sensitivity" has been explored/discussed in many literature, e.g. [2]; [3][4] studied "FRAME CONTRIBUTION PARITY" as well. Having a brief literature review section for each principle, and discuss the difference and contributions of current work could strengthen the manuscript.

Thank you for your suggestions. We have added a brief literature review for each principle in the updated manuscript.

---

The author seem not mention it in the paper - will the benchmark be made publicly available at some point?

Yes, the benchmark will be made publicly available upon publication.

---

The author selectively evaluate some proprietary MFMs in Table5. It would be interesting to report the results of different model sizes in the same model family (e.g. Gemini 1.5 pro v.s. Gemini 1.5 flash; Claude 3.5 Sonnet v.s. Opus) to understand the scaling effect of current SOTA MFMs on the proposed benchmarks.

Table 5 presents evaluation results on Gemini 1.5 flash, Claude 3.5 Sonnet, and Claude 3 Opus. We have now added evaluation of Gemini 1.5 pro, and Claude 3 Haiku. We also attach results of different model sizes in the same model family below, which highlights the scaling effect of the current SOTA MFMs:

Thank you for taking the time to review our work and providing suggestions for our analysis. We would like to address your concerns and questions as follows:

---

Clarification on Table 6. It is not very clear to me what "True" & "False" stand for in the table. More explanation and details on the experiment setups are needed. This leads to the question of how the conclusion "More Capable Models Are More Likely to Exploit Shortcuts Through Common Sense." is drawn. Does "True" and "False" mean different split of datasets? If so, those videos and their QAs are different, how could we compare their relative performance drop and reach the conclusion?

In Table 6 (Table 7 in Appendix B in updated manuscript), “True” and “False” indicate different splits of whether the questions meet the criteria specified in the merged cell above the respective column. For instance, in the “Counterfactual” column, “True” denotes performance on questions that are counterfactual, while “False” denotes performance on questions that are not counterfactual.

As shown in Table 6, more capable models (GPT-4o and Qwen2-VL-72B) show much lower performance in the counterfactual scenarios (i.e., those in the “True” sub-column). This observation suggests that these models are more likely to exploit shortcuts through common sense rather than performing reasoning over the video content.

Our videos are labeled after collection and generation, meaning that the video scenarios are not purposefully designed to prove such conclusions but rather to suggest general trends displayed in our experiment results. We have now added clarifications and downgraded these analysis to Appendix B Scenario Analysis.

---

L479 "Models Excel in First-Person Over Third-Person Perspective Temporal Reasoning Video Understanding." The conclusion of model excels FPV over TPV videos on comparing experiment results of 88 FPV QA v.s. 668 TPV QA. Since these are two set of different videos and QAs, it is hard to compare the final score side by side and draw the above conclusion in my opinion. More rigorous study or careful experiment design is needed, for instance, creating a matched set of FPV and TPV videos covering the same scenarios would make more sense.

Thank you for this comment. We acknowledge that a comprehensive comparison of FPV and TPV requires a more rigorous study. However, by including the comparison of FPV and TPV, our goal is to explore the general trends in model performances based on the presence or absence of a main subject in the video, rather than a comprehensive study. The results suggest that not having a main subject does not negatively impact the model performance. We acknowledge that it needs a better video collection process to rigorously prove the effect of perspectives where the scene variables are more strictly controlled. We have carefully rephrased our conclusion of these results to reflect this point.

Many of the proposed question types already are part of existing works, so it’s unclear how this benchmark would be any different in satisfying the principles. Is the difficulty coming from the annotation process and counterfactuals?

Thank you for your question. As illustrated in Table 7 in Appendix A (Table 6 of the updated manuscript), among the six reasoning types included in TVBench, only Visual Cues (4.7%) and Action Counts (19.7%) are sourced from existing datasets. For Visual Cues, we repurpose and reannotate all videos that we sourced from Music-AVQA. For Action Counts, we acknowledge that this is an existing reasoning type within some existing works and we edit and reannotate a part of those videos. In inclusion of the Action Counts reasoning type as one of the six reasoning types, TVBench aims to comprehensively evaluate all aspects of visual temporal reasoning of current models.

---

The benchmark may contain a lot of unrealistic questions because of (i) counterfactuals and (ii) simulated data that are used to artificially make the benchmark harder. These may affect the usefulness in real-world applications.

We hope to respond to this question from two aspects.

(1) There are 233 questions from simulated data and 100 counterfactual questions (not mutually exclusive), which constitute 22.3% of the benchmark QAs. This small portion does not detract from its primary focus on real-world visual temporal reasoning.

(2) While the benchmark does include questions that may be considered “unrealistic” (e.g., counterfactual and questions derived from simulated videos), these questions are intentionally designed to test models’ capabilities in handling out-of-distribution tasks. Such tasks provide valuable insights into models’ robustness and generalizability, which is crucial for real-world applications. To better illustrate our point, we now included two analysis paragraphs in Section 6.3, titled Models fail to truthfully leverage the visual input while being over-reliant on common sense and Models are highly susceptible to noisy information in the input.

---

Audio inputs (speech) are not discussed in this benchmark. What happens if audio (speech) is included as part of video inputs to the models? Will the results on principles and and on TVBench change?

Thank you for raising this interesting point. Since this work focuses primarily on visual temporal reasoning, audio modality is beyond the scope of this work. All inputs in our evaluation do not include audio. If audio (speech) were to be included, among the six reasoning types, models are likely to take shortcuts in answering questions from Visual Cues, Action Counts, and Velocity & Frequency, leading to an overestimation of model performance on our benchmark.

Thank you for taking the time to review our work and providing such insightful feedback. We would like to address your concerns and questions as follows:

---

The main weakness in my opinion is that the defined principles and ways of measuring benchmarks' adherence to these principles is not very rigorous.

For example, a benchmark that simply contains state transition questions requiring exactly 2 frames will get perfect scores on these three principles. However, such a benchmark is not very valuable. In effect, these principles can be considered necessary conditions (second one is a bit questionable), but not sufficient conditions for a benchmark to be considered as requiring temporal reasoning.

We’d like to first clarify a potential misunderstanding about the purpose of the principles we have developed. Our goal is not to develop principles that define the entire spectrum of what constitutes temporal reasoning. Rather, these principles serve as diagnostic tools to reveal loopholes that (1) exist in current benchmarks and (2) are exploited by current foundation models.

Our results in Table 1-4 show that our current principles are sufficient to demonstrate several of these loopholes. By alerting the research community to these loopholes, we believe that our work is both impactful and timely.

We agree that there are likely many other prerequisites for a question to truly “require” temporal reasoning. We hope that, by being the first to pose such principles, our work can inspire others to continue to discover more.

---

1.1 Single frame insufficiency: Why is the definition limited to a single frame? What about QA pairs that can be answered in just a few frames, say 2 frames? In the abstract, it's argued that such a scenario is bad for temporal reasoning benchmark, however, this principle will consider such a question as good.

As you have suggested, our definition of “single frame insufficiency” serves as a deliberately minimal baseline—a lower bound for exposing the lack of temporal reasoning in current benchmarks and models. Specifically, if a model performs equally well (or nearly so) using single frames compared to multiple frames, it strongly suggests that it is not the additional frames that helped with answering and the question does not require temporal reasoning.

This is what our results in Table 1-2 show—existing benchmarks and models don’t even meet this low standard. While our principles may not directly expose all such cases (e.g., two-frame sufficiency), they are still very useful for identifying common deficiencies. Realistically, temporal reasoning should require more frames than 2 frames. Coming up with a higher standard for frame sufficiency will be important once models become more capable of temporal reasoning. Furthermore, the exact number of frames used for this standard may be dependent on the number of separate events depicted in the video.

---

1.2 Single frame insufficiency: In the second implementation, a single hand-picked frame is compared with uniformly sampled 16 frames. This seems a bit unfair to the 16 frame case. Shouldn't we compare hand-picked 16 frames with hand-picked 1 frame to get a true sense of single frame insufficiency?

Models typically process a video as a set of uniformly-sampled frames. If a model’s performance increases when it is given more samples, one might infer that it is performing temporal reasoning (i.e., that it is reasoning over the relationship between frames). However, there is an alternative explanation: as the number of samples increases, it becomes more likely that one of those samples is highly-informative. Thus, the model may not be considering the relationship between frames at all, but rather, is reasoning over them individually.

The purpose of our test (1 hand-picked frame vs 16 uniformly-sampled) is to detect whether a model may be exploiting this loophole. The hand-picked frame is our approximation of a “best-case” scenario; i.e., a single frame that is maximally representative of the entire video. If a model achieves similar performance in both conditions, then it may be relying on the information within a single frame and is not leveraging all 16 frames available to it.

We have now extended our analysis to study m uniformly-sampled frames (rather than just m=16) to better assess variations in video content and duration. Given the relatively short video duration and homogenous content within each second, it is very likely that a highly informative frame is sampled as the number of sampled frames increases. Our study across m = 1, 8, 16, 24, and 32 demonstrates that all performances stabilize as m approaches 16, as shown in Figure 4 in Appendix E.3 of the updated manuscript. Therefore, our choice of m = 16 is an outcome of experiments rather than an arbitrary one. The results show that m = 16 provides a sufficient window for models to perform temporal reasoning.

2. Frame order sensitivity: The definition assumes that frame order sensitivity is essential for a good temporal reasoning benchmark. However, a powerful model human can easily figure out the temporal order in a lot of cases. So, I don't know if this needs to be a hard requirement.

We hope to clarify this from two aspects. (1) Table 3 shows a significant performance drop on TVBench when frames are shuffled, indicating even advanced models struggle to infer order. (2) While it is true that a sufficiently powerful model or human could infer temporal order in some cases, Figure 1 highlights that the common issue with existing benchmarks lies not in shuffled frames being easily solvable, but in QAs that do not necessitate reasoning in temporal order. Thus, we believe our metric effectively identifies tasks that fail to require reasoning with the correct frame order.

---

3. Frame contribution parity: Implementation-wise, again why only a single frame is considered? The proposed implementation will not give any signal for a QA pair requiring 2 frames.

We propose through Frame Contribution Parity that each frame should contribute equally to answering the question. Thus, evaluating with the exact number of frames needed to answer the question is unnecessary. By demonstrating a significant performance gain from random to handpicked single frames (even for questions that require 2 frames to answer), we show that the informativeness of frames is uneven.

---

Lastly, the way of evaluating these principles using pre-existing models creates a chicken and egg problem, as the models themselves may not be capable of answering the said questions. For example, a very hard benchmark that GPT-4o cannot answer will fail all the principles if we use GPT-4o to evaluate the principles.

Given the potential misunderstanding that these principles guarantee a question requires temporal reasoning, we agree that relying heavily on GPT-4o to qualify good temporal reasoning tasks does create a chicken and egg problem. However, as mentioned above, the goal of these principles is to expose common loopholes in existing benchmarks. GPT-4o only helps in rejecting the bad examples and is not used in creating the valid questions.

We acknowledge that even GPT-4o will fail all the principles on a very hard benchmark. As shown in Table 5, the performances of the SOTA models are still above the baselines, meaning that while our questions are challenging, they are still feasible for models to perform a certain degree of valid reasoning.

Dear Reviewer rS5B,

Thanks again for your insightful and thoughtful comments!

With the author-reviewer discussion period ending soon (Dec 2nd at 23:59 AoE), we wanted to take this opportunity to thank you again for your suggestions and to ask whether you had any remaining questions after our responses! We especially hope that you have a chance to review the additional clarifications we provided regarding our metrics.

If we've addressed your concerns, we'd be grateful if you'd consider updating your score!

Best regards,

Authors

Thank you for your feedback!

Principles developed in this paper are diagnostic tools and not comprehensive. If that is the case, it's best to describe them as diagnostic tools in the paper for clarity.

We appreciate the reviewer’s suggestions and would like to clarify that the introduction and Section 3 both indicate the principles are intended to evaluate the effectiveness of tasks in visual temporal reasoning, underscoring the diagnostic nature of these principles rather than presenting them as a comprehensive definition of the entire spectrum of visual temporal reasoning. We will further clarify this in our updated manuscript.

There are several studies that discuss the drawbacks of existing benchmarks in terms of temporal understanding, spread across different works [e.g. 1, 2, 3].
[1] Revealing Single Frame Bias for Video-and-Language Learning
[2] Verbs in Action: Improving verb understanding in video-language models
[3] PAXION: Patching Action Knowledge in Video-Language Foundation Models (NeurIPS Spotlight)

We appreciate the reviewer highlighting these related works. As suggested by reviewer NXCv, to strengthen our claims, we have now included a brief literature review for each principle in our updated manuscript. As you suggested, [1][2][3] are also relevant in the context of temporal reasoning. Specifically, [1][2] relate to Principle 1: Multi-Frame Gain, considering “static appearance bias”[1] and “bias towards singleframe concepts”[2]. [3] relates to Principle 2: Frame Order Sensitivity, considering video reversal, a special case of frame shuffling. We will include these works as parts of our literature review in our updated manuscript.

While these works address individual aspects of temporal reasoning, to the best of our knowledge, our work is the first to unify them as principles with corresponding metrics to: (1) quantitatively evaluate the effectiveness of visual temporal reasoning tasks, and (2) expose loopholes in existing temporal reasoning video understanding benchmarks. By alerting the research community to these loopholes and by presenting TVBench, we hope that our work can inspire others to propose more rigorous tasks as well as principles and metrics for the advancement of multimodal foundation models.

Thanks for your response!

The authors suggest that the principles developed in this paper are diagnostic tools and not comprehensive. If that is the case, it's best to describe them as diagnostic tools in the paper for clarity.

We hope that, by being the first to pose such principles, our work can inspire others to continue to discover more.

There are several studies that discuss the drawbacks of existing benchmarks in terms of temporal understanding, spread across different works [e.g. 1, 2, 3].

[1] Revealing Single Frame Bias for Video-and-Language Learning
[2] Verbs in Action: Improving verb understanding in video-language models
[3] PAXION: Patching Action Knowledge in Video-Language Foundation Models (NeurIPS Spotlight)

Thank you for taking the time to review and recognizing the strengths of our work. We would like to address your concerns and questions as follows:

---

There are sourced YouTube videos in the dataset, which might have some implications in terms of privacy, security and safety.

All videos sourced from Youtube are licensed under Creative Commons, which states that “This license enables reusers to distribute, remix, adapt, and build upon the material in any medium or format, so long as attribution is given to the creator”. We also included further clarification in Appendix G (Appendix F in the updated manuscript): “We do not hold the copyright to any of the YouTube videos; we ask that you respect the rights of the original video creators. If you believe that any content in TVBench infringes on your rights, please contact the authors immediately, and we will remove the video accordingly.”

---

If the authors could include an experiment to test the effect of language prior - i.e. the model's accuracy when only given the text part of the dataset questions, and report the difference, that might also help reinforce the soundness of their benchmark design.

Our text-only baseline (Random (GPT-4o)) in Table 5 is intended to address this. In this setting, GPT-4o was prompted to answer the questions using only the textual input, without access to the videos. The overall performance in this text-only setting is 23.1%, resulting in a difference of 4.6% with the random baseline (Random Choice (42)), which shows that we introduce limited bias in the textual input.

---

If the authors could provide some information on the resolution and scale of the videos, that might be helpful especially because the authors discuss models not benefiting from more than 8 frames.

Thank you for your suggestion. We have now included the resolution details of the videos in Table 9 (Appendix E.1) of the updated manuscript. The average resolution is 1332x1076, and the maximum resolution is 1080x1920. However, some models might preprocess video inputs to its required resolutions.

---

What would happen if corresponding accuracy is zero in the denominator for the three metrics? Might need an epsilon term, or it might just be sufficient to subtract the two accuracies.

Thank you for your suggestion. We now incorporate an epsilon term to prevent division by zero in the updated manuscript.

---

Why might there be a performance plateau beyond 8 frames? Is it in the design of common foundation models? Is it likely due to there being too much information to process (for example, the way the models process more than 8 images) ? Do various models show the plateau behavior exactly at 8 frames? What about at smaller frames?

We hypothesize that the performance plateau is likely due to current foundation models being commonly trained on datasets with a fixed number of frames, such as 8 frames [2] or 16 frames [1]. This training design might limit foundation models’ capability in effectively processing additional temporal information with more frames than they were originally trained on.

We have performed additional analyses with smaller frame counts: 0, 1, 2, 4, 6, and 8 frames. As shown in Figure 5 (Appendix E.4) of the updated manuscript, the performance shows a general increase in this range. However, we also hypothesize that the plateau is also dependent on the difficulty level of the task.

[1] Fei, Jiajun, et al. "Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos." arXiv preprint arXiv:2408.14023 (2024).
[2] Wang, Yi, et al. "Internvideo2: Scaling video foundation models for multimodal video understanding." arXiv preprint arXiv:2403.15377 (2024).

---

How does the performance of zoomed-in views differ when only tested on simulation videos with less noise?

To clarify, only human videos have zoomed-in views. All simulated human videos are generated from zoomed-in real human videos. This generation from zoomed-in views only is to ensure a more accurate conversion from the real human motions to 3D skinned human models. Thus, Figure 3 (Figure 2(b) of the updated manuscript) presents, given zoomed-in views, the results of real humans compared to simulated humans.