Temporal Reasoning Transfer from Text to Video

Dear Reviewer c8nY,

We are greatly encouraged by your recognition of our well-defined motivation, the innovative fine-tuning technique that bypasses the need for visual data, and our thorough experimental analysis across multiple benchmarks.

We address the questions you pointed out as follows:

1. Examples of captions generated by GPT-4o

Thanks for raising this important question! We have demonstrated the GPT-4o generated frame captions in Tables 9, 10, and 11 in the appendix. Below are two examples of captions for the temporal aspects of brightness change and blooming.

Brightness Change

• Frame 1: The image shows a very dark scene with a black and white cat lying on top of a laptop keyboard. The cat’s face and whiskers are slightly visible, but the overall image is hard to see due to the low brightness.
• Frame 3: The image shows a black and white cat lying on top of a laptop keyboard. The overall scene is very dark, making it difficult to see details clearly.
• Frame 5: The image shows a black and white cat lying on top of a laptop keyboard in a slightly dark environment.
• Frame 7: The image shows a black and white cat lying on top of a laptop keyboard. The cat appears to be comfortably settled in the space between the laptop screen and keyboard. The overall brightness of the image is normal, allowing clear visibility of the cat’s fur and the laptop’s keys.

Blooming

• Frame 1: This frame shows a blue bud at the end of a green cotton swab positioned vertically against a plain white background. The cotton bud appears clean and unused, with no visible movement or action taking place.
• Frame 3: This frame shows the same blue bud at the end of a green cotton swab, still positioned vertically against a plain white background. The cotton on the bud appears slightly more compressed and misshapen compared to the preceding frame.
• Frame 5: This frame shows a blue flower with five petals on a green stem, set against a white background. The flower appears to be blooming mid-stage with slightly spread petals, and the stem is straight and upright, positioned centrally in the frame.
• Frame 7: This frame features the same blue flower with six petals on a green stem, now slightly to the right against a white background. The petals are still widely spread, and the stem remains upright, suggesting no noticeable movement or change from the previous frame.

As we can see, these captions provide sufficient and accurate per-frame information to comprehend the temporal dynamics of brightness change and blooming. We will update the paper to make references to Tables 9, 10, and 11 clearer in the main text. Thanks again for the constructive advice.

2. LLaVA-ReCap-558K is derived from LCS-558K rather than BLIP558K

Thank you for bringing this issue to our attention. Our information was initially based on Section 3.2 of the blog you referenced (https://llava-vl.github.io/blog/2024-05-25-llava-next-ablations/), which stated, "We used the model to generate new captions for the images from the following datasets: COCO118K, BLIP558K, and CC3M." However, it appears that "LCS-558K (a subset of the LAION/CC/SBU dataset)" is a more accurate description of the original source of the images. We will since update our paper to reflect this information accurately. Thank you again for this careful suggestion.

Thank you for your response. Most of my concerns are addressed so I increased my ratings (5->6). I'm looking forward to your results on at least one of larger models.

Dear Reviewer SHtr,

We greatly appreciate your thoughtful feedback on our work! We are encouraged by your recognition of our clear writing and the effectiveness of our method. We address your questions and concerns as follows:

Weakness 1. Is LLM the major bottleneck in video temporal reasoning?

Thanks for raising this important question. Indeed, the vision encoder, vision-language alignment and the LLM are all essential for video temporal reasoning. However, in our work, we argue that while the vision encoder and alignment components already provide sufficient visual information, the LLM remains the major bottleneck due to its insufficient temporal reasoning ability. This claim is supported by the following evidence:

• Even when provided with per-frame textual captions as context, 7B-scale LLMs (e.g., Qwen2 and LLaMA3) fail to correctly answer a significant portion of questions in basic temporal probing tasks. This highlights a critical limitation in their ability to handle temporal reasoning.
• Classifier probes trained on visual features projected into the LLM embedding space achieve over 90% accuracy in most temporal aspects. This demonstrates the high quality of both the vision encoder and vision-language alignment components, further isolating the LLM as the bottleneck.
• Our study involves two Video LLMs, LongVA-7B and VILA-8B, built on Qwen2-7B and LLaMA3-8B, respectively. These models exhibit similar trends in our probing study, reinforcing the robustness of our conclusion.
• TempCompass [1] reveals that while advanced Video LLMs achieve high accuracy in recognizing coarse-grained and fine-grained actions, their performance is substantially weaker in temporally reliant dimensions such as event order, attribute changes, and directional reasoning. This finding aligns with our claim that the LLM’s temporal reasoning ability is the major bottleneck.
• In TOMATO [2], a recently proposed benchmark for video temporal understanding, it was observed (in Section 6.3) that “GPT-4o correctly generates captions for each consecutive change in the moon’s movement, showcasing its ability to reason at individual time steps. However, it fails to infer based on the captions that the overall sequence represents a clockwise rotation.” This finding suggests that vision-language alignment is not a significant bottleneck for the most advanced Video LLMs.

In summary, our claim that “LLM is the major bottleneck of video temporal reasoning” is well-supported by our rigorous empirical results & previous/concurrent studies.

[1] TempCompass: Do Video LLMs Really Understand Videos?

[2] TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models.

Dear Reviewer GraR,

We greatly appreciate your thoughtful comments on our work. Below we address each concern in detail:

1. Can our method enhance the understanding of motion-related temporal concepts?

Thanks for raising this insightful question! Indeed, the temporal pattern of motions such as rotation, direction and velocity differ from concepts like order and temporal grounding, making them less straightforward to represent via frame-by-frame captioning. We acknowledge that fully understanding motion in videos may require approaches beyond our proposed textual temporal transfer. However, it is important to note that the temporal patterns of motion are inherently recognizable from discretely sampled video frames. For instance, by observing the relative position changes between two individuals across frames, we can infer which one is running faster. To interpret such information, a Video LLM must have a foundational understanding of temporal relationships between frames (e.g., order), a capability that can be supported by our method.

To assess whether our method benefits the understanding of motion-related temporal concepts, we evaluate Video LLMs on the recently introduced TOMATO benchmark [1]. Interestingly, as shown in the table below, our method yields noticeable improvements in temporal concepts such as velocity & frequency (+16.8), shape & trend (recognizing the shape drawn by a person in the air) (+8.1) and action counting (+1.7). Notably, these temporal concepts are unseen in our training set. Table 2 and Table 3 in our paper also demonstrate that our T3 training data improves the accuracy of LongVA by 3.9 points in MLVU action counting and 2.4 points in TempCompass direction.

We attribute these improvements to the enhanced ability to capture temporal relationship between video frames, which is the foundation for understanding velocity, direction, counting and shape & trend. These findings suggest that our T3 training data, while primarily focused on basic temporal concepts, can generalize to more complex and unseen temporal concepts. We will incorporate the TOMATO benchmark results and the above discussion into the next version of the paper.

[1] TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models.

Performance on the TOMATO Benchmark.

Dear Reviewer hvnd,

Thank you for the thoughtful review of our work on improving temporal reasoning in Video LLMs. We sincerely appreciate your recognition of the novel perspective, the quality of our empirical analysis, and the clarity of our writing. Our responses to your questions are below.

Weakness 1: Can our method generalize to downstream tasks with subtle temporal cues?

We fully understand your concern that our template-based text tasks may not generalize effectively to real-world videos with subtle temporal cues. To address this, we evaluated the models on the recently released Vinoground benchmark [1]. Vinoground comprises carefully curated real-world video-caption pairs, with each pair sharing identical static content but differing in subtle temporal nuances. For example, "the watermelon is cut and then turned" versus "the watermelon is turned and then cut" is a pair of positive/negative captions. In this benchmark, models are challenged to identify the correct caption given a video (Text) or the correct video given a caption (Video).

As shown in the results below, our T3 dataset significantly enhances the performance of LongVA-7B on Vinoground. Furthermore, our approach demonstrates noticeable improvements in tasks such as Action Counting (+3.9) and Plot QA (+1.3) on MLVU, even though these tasks are not present in the T3 dataset. These findings highlight that while our method relies on template-based text tasks, it generalizes effectively to downstream tasks involving real-world videos and subtle temporal cues.

Furthermore, we have identified two promising directions to enhance our approach to better address the subtle temporal cues in real-world videos:

• Diversifying text formats by leveraging MLLMs to automatically derive natural temporal descriptions from real videos and images. This would help capture more organic temporal expressions and relationships beyond templated patterns.

• Mixing modalities by combining text with images/videos to better capture subtle temporal nuances. This multimodal approach could help bridge the gap between templated and natural temporal expressions.

We would like to explore these ideas in the future and incorporate the discussion into our revision.

[1] Vinoground: Scrutinizing LMMs over Dense Temporal Reasoning with Short Videos.

Weakness 2: Regarding a broader spectrum of temporal tasks.

Thanks for the constructive feedback! We agree that incorporating more diverse and complex forms of temporal reasoning could enhance the robustness of our method, and we look forward to exploring this in future work. However, we would like to emphasize that the four temporal aspects in our paper are very basic and serve as the foundation for more complex temporal reasoning skills.

To validate this hypothesis, we evaluated our method on the newly released TOMATO benchmark [1], which tests a variety of motion-related temporal reasoning abilities, including direction, velocity, and rotation.

Our findings reveal promising generalization to these temporal concepts unseen in our T3 training dataset: For instance, our method significantly improves performance in velocity and frequency understanding (39.9% vs 23.1% baseline) and Shape & Trend (30.5% vs 22.4%). While there is room for improvement in categories like Rotation, we plan to explore enhanced approaches using the directions discussed above. These new results will be included in our revision.

[1] TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models

Dear Reviewer UiVz,

Thank you for your comments! Our responses to your questions are below.

W1-1: Why choose LongVA as the backbone?

We chose LongVA for two compelling reasons:

1. Superior Performance and Rigorous Evaluation

   • LongVA demonstrates exceptional long video understanding capabilities (52.6 on Video-MME vs. 50.8 of Phi-3.5-vision). Using a strong baseline model allows us to rigorously validate our method's effectiveness. Besides, the shared Qwen-2 architecture between LongVA, Qwen2-VL, and LLaVA-OneVision ensures our findings are transferable across these models

2. Consistency with our Claim: Temporal Reasoning Transfer without Video Data

   • LongVA's training exclusively on image-text data, with no video-language datasets. This setup is more consistent with our claim and provides a stronger validation of our findings.

Regarding the alternative models (Qwen2-VL and LLaVA-OneVision) you mentioned, their recent release dates (Qwen2-VL: Sep. 19, LLaVA-OneVision: Sep. 25) were too close to the submission deadline (Oct. 1) for the comprehensive evaluation. Therefore we did not include the experiments with these models in our submitted version.

W1-2 and W3: Results on Larger Models and Video-trained Models

Thank you for raising this important question. Following your suggestion, we evaluated our method on Qwen2-VL-72B, a larger model pre-trained with video data. We used the ms-swift training framework following Qwen2-VL fine-tuning best practices. The evaluation results on MLVU and Video-MME using 16 and 32 frames demonstrate that T3 consistently improves performance even on large video-trained models:

These results show that T3 provides significant improvements on key temporal reasoning tasks (e.g., Action Count: +9.2% and Action Order: +8.1% with 32 frames on MLVU) even when applied to a SOTA 72B video-trained model. This demonstrates that our approach is not merely a temporary solution for small models, but rather provides complementary benefits that enhance temporal understanding capabilities across model scales.

W4: Evaluation is limited

We respectfully disagree that our evaluation is limited. Beyond temporal ordering, our evaluations comprehensively cover various aspects of temporal reasoning through multiple established benchmarks:

1. MLVU and Video-MME are holistic video understanding benchmarks that evaluate diverse temporal reasoning capabilities including ego reasoning, action counting, plot understanding, and anomaly recognition.

2. We further validated our approach on two additional recent benchmarks with key results highlighted below (please refer to our responses for Reviewer GraR and Reviewer hvnd) :

TOMATO: Tests sophisticated motion-related temporal reasoning:

• Velocity & Frequency: T3 significantly improves performance (39.9% vs 23.1% baseline)
• Shape & Trend: Shows notable gains (30.5% vs 22.4%)
• These improvements demonstrate our method's generalization to unseen motion-related temporal concepts

Vinoground: Evaluates fine-grained temporal understanding through real-world video-caption pairs with subtle temporal differences:

• Text retrieval: +10.4% improvement (31.6% vs 21.2%)
• Video retrieval: +8.4% improvement (30.0% vs 21.6%)
• Group accuracy: +6.4% improvement (11.8% vs 5.4%)

These results demonstrate that our approach generalizes well across different temporal reasoning aspects and datasets, beyond just ordering tasks. The improvements on tasks not explicitly present in T3 (e.g., Action Counting +3.9%, Plot QA +1.3% on MLVU) further validate the broad applicability of our method.

Furthermore, while ActivityNet-QA and NextQA are valuable benchmarks, they may not be suitable for benchmarking the temporal reasoning capabilities according to [1]. Here's a detailed comparison:

Our chosen benchmarks are more suitable for evaluating temporal reasoning capabilities for several key reasons:

1. Longer Temporal Dependencies: Video-MME and MLVU feature significantly longer videos (1,018s and 720s vs 44-180s), allowing for evaluation of long-range temporal understanding.

2. Temporal Verification Complexity: The Temporal Certificate Length in Video-MME (385.5s) is substantially longer than ActivityNet (2.4s) and NextQA (2.7s), indicating that our chosen benchmarks require understanding broader temporal contexts rather than just short, isolated events.

3. Domain Diversity: While ActivityNet and NextQA focus solely on everyday activities, our benchmarks span multiple domains including knowledge-based content, entertainment, sports, and egocentric videos, offering a more comprehensive assessment of temporal reasoning abilities.

These characteristics make our evaluation suite more appropriate for assessing sophisticated temporal understanding capabilities in real-world scenarios.

[1] EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding

Dear Reviewer UiVz,

We have tried to address your concerns in our earlier response. If you have any further questions or suggestions, we are very happy to discuss with you.

Dear Reviewer UiVz,

As we approach the end of the reviewer-author discussion period, we have made efforts to address your concerns in our previous response. We would greatly appreciate it if you could let us know whether our response has adequately addressed your questions and concerns.