Seeing the Arrow of Time in Large Multimodal Models

We thank reviewer GqEN for the helpful comments and for providing thoughtful feedback on our work.

---

1. On Grounding Arrow of Time in Language

Why should an existing video LLM model understand what one means by "forward or reverse direction of the video"?

Describing direction of time in language is tricky for irreversible videos. How would one describe a reversed video of "smoke diffusing"?

We thank the reviewer for these insightful comments. They touch on the core challenge of evaluating a concept as fundamental as the Arrow of Time within a language-based framework. We address the key points below.

Describing Arrow of Time in Language. We agree that describing a physically irreversible event in reverse (like a waterfall going up) is linguistically awkward. Our evaluation is designed accordingly to ensure that it never requires a model to generate a free-form description of a physically implausible reversed video:

• The quantitative sequence direction classification task in AoTBench is a classification, not a captioning task. LMMs are required to map its visual understanding of sequence directionality to a simple linguistic label ("forward" or "reverse"), testing a fundamental aspect of multimodal reasoning.
• For qualitative examples that do involve description (e.g., Fig. 1, Fig. 7, right), we use "reversible" videos where both directions are physically plausible. We will add clarification in the paper.

To your specific question about describing reversed physically implausible events such as "smoke diffusing", our observation is that current LMMs, due to lack of temporal directionality understanding, fail to describe the implausible dynamics. They tend to focus on static objects and often repeat the forward-action verb (e.g., still saying "smoke diffusing"), confirming their temporal insensitivity.

In all, we believe the ability to recognize that a video violates the physical laws of nature and connect that visual intuition to a symbolic linguistic concept is a fundamental capability. For LMMs to progress, they need to be able to ground simple linguistic concepts like "forward" and "reverse" in their visual understanding—a task humans perform effortlessly. Our classification task provides a crucial testbed for this capability.

On Evaluating Base LMMs. About whether an existing LMM should be expected to understand this task without explicit training, we clarify that our goal in testing base LMMs is not to claim they should succeed, but rather to use the task as a diagnostic tool. Their near-random performance, as shown in Table 1, is a key finding of our paper. It provides a critical baseline that demonstrates a clear capability gap in current models and powerfully motivates the need for targeted training methods like ArrowRL.

Finally, we agree with the reviewer's insightful point that understanding when a visual phenomenon is difficult to describe is a key challenge for more advanced LMMs. While our work focuses on the foundational skill of recognizing directionality, we believe exploring the limits of linguistic description for complex physical events is an important direction for future research. We welcome any further suggestions from the reviewer on improving this task format and will revise the paper accordingly to reflect this discussion.

---

2. Control Task to Disentangle OOD Effects

Is there a control task (like in [1]) where the question is insensitive to the arrow of time (e.g., describe all objects appearing in the video) and the model does well with both forward and reverse videos? This can establish that low performance of existing models need not be because the video is OOD.

We appreciate the suggestion of adding a control task, which is well-motivated by prior work ([1] Bagad et al., cited in our paper as [2]) on disentangling temporal effects. Accordingly, we design action recognition as the control task, using videos from the UCF101 dataset and their action label. For each video, we formulate a binary-choice question with the prompt, "What action is being performed in this video?" The correct action was provided as one choice, and the incorrect option was randomly drawn from the set of all other action labels. We then evaluated several LMMs on both the original (forward) and time-reversed versions of these videos.

The results above show that all models perform at near-perfect accuracy on this control task, irrespective of video direction. This contrasts sharply with their near-random performance on our AoT-sensitive sequence direction classification task (Table 1, UCF column). This analysis provides strong evidence that the failure of existing LMMs on AoT tasks is due to their inability to perceive temporal directionality, not a general failure to process reversed videos as OOD inputs. We appreciate the reviewer for prompting this important analysis and will incorporate it into our paper.

[1] Test of Time: Instilling Video-Language Models with a Sense of Time. Bagad et al. CVPR 2023.

---

3. Justification for Using the First Token Probability and Clarification on Ln 145

A justification for the choice of measure (KLD between output distributions of first token) of arrow of time consistency would make the paper stronger. Likewise, it would be helpful to clarify L145 on filtering out samples.

First Token Probability. We use the first-token probability to estimate the model’s likelihood of selecting each option in the MCQ format. Since our prompt instructs the model to “Answer with the option’s letter from the given choices directly,” the probability of the first token (e.g., A, B) serves as a reliable proxy for the model’s chosen answer. This approach is standard and widely adopted by the LLM community [1–3]. Our novelty lies in adapting this standard evaluation practice for temporal analysis by computing the KL Divergence between the distributions generated from the forward vs. the reversed video.

Ln145 Clarification. In some rare cases, we observe that our candidate LMMs perform better on reversed videos than on the original forward ones. These are typically outlier cases where temporal direction is either ambiguous or irrelevant. We exclude these samples from further analysis to ensure that our evaluation focuses on instances where AoT reasoning is actually required. We appreciate the reviewer’s suggestion and will update the manuscript to clarify these design choices.

[1] Brown et al., Language Models are Few-Shot Learners, NeurIPS 20

[2] Hendrycks et al., Massive Multitask Language Understanding, ICLR 21

[3] Lin et al., TruthfulQA: Measuring How Models Mimic Human Falsehoods, ACL 22

---

4. General Video Benchmark Performance

On non temporal benchmarks, does the model suffer at all or retain its performance?

We thank the reviewer for this important question regarding ArrowRL’s performance beyond temporally-focused benchmarks. We reiterate that ArrowRL is designed as a targeted enhancement for an LMM's temporal sensitivity. As our analysis in Sec. 3.1 shows, many existing video benchmarks do not actually require this capability to be solved (e.g., they can be answered with single, shuffled or reversed frames). For this reason, our main results in Table 1 focus on the benchmarks we identified as temporally sensitive, where ArrowRL shows significant boosts.

The reviewer raises a valuable follow-up question: whether our specialized training harms performance on easier tasks where AoT is not required. To answer this, we have conducted experiments on the benchmarks that our own analysis identified as less sensitive (Fig. 3 right).

As shown, ArrowRL preserves or even slightly improves performance across these general video benchmarks that do not demand temporal awareness. This new analysis (same as shown for Reviewer mxwr and 6fDT) confirms that ArrowRL is a targeted enhancement: it provides great gains on temporally-aware benchmarks (as shown in Table 1 with data from 3 relevant and popular benchmarks) without degrading performance on more general benchmarks where AoT is not a primary factor (Ln 280-283). We thank the reviewer for this suggestion and will add this analysis to the final paper.

---

4. On Including Analysis of Failure Cases in Main Text

While the Supp includes failure cases, it may be worth adding a couple of lines in the main text on what kinds of samples the model fails on and if there is a pattern to it. That could also guide future work.

Thank you for your comment. We will move a concise summary of our failure case analysis (detailed in Supp. Sec. D) into main paper as suggested.

We thank reviewer 6fDT for the helpful comments and for providing thoughtful feedback on our work.

---

1. Clarification Regarding ArrowRL Response Generation

How did the authors construct response pairs where the difference between reverse responses and forward responses can reflect temporal directionality?

We appreciate this insightful question, and fully agree that ensuring the difference between forward and reverse responses reflects true temporal directionality is crucial.

First, we clarify that our method does not construct or use pre-defined "forward-reverse response pairs." Instead, as part of the GRPO framework, all responses are generated on-the-fly during training: the policy model generates a group of forward responses to the original video, and a reverse response. These responses are then used to compute the fidelity and reverse rewards in real time.

This dynamic approach means the central challenge is ensuring the reverse reward is applied intelligently, which is crucial for the method's effectiveness. We achieve this with two techniques:

High-temporality Data Curation. For captioning tasks, we leverage the curated RTime dataset [14], which consists of videos specifically selected for having meaningful and distinct forward vs. reverse actions (Ln 251-253). This provides a clean, high-quality signal for learning.

Dynamic Reward Weighting. For larger, less curated datasets, we employ a dynamic weighting scheme (Ln 207–214) that detects when the reversed response is too similar to the forward ground truth and automatically disables the reverse reward for that sample. The effectiveness of these strategies is supported by our experiments. The ablation in Table 2 (+ArrowRL (γ=1)) already demonstrates that removing the dynamic weighting hurts performance. Furthermore, to isolate the effect of data selection, we ran a new experiment focused only on the captioning task, replacing the high-quality RTime data with uncurated videos from LLaVA-Video-178K. The results below (same as shown for Reviewer tJ9G) underscore the importance of our data curation strategy.

Together, these strategies ensure our reverse reward is applied precisely and only to cases where it can encourage a meaningful understanding of temporal directionality.

---

2. General Video Benchmark Performance

Does this method still show improvements on general temporal benchmarks like VideoMME?

We thank the reviewer for this important question regarding ArrowRL’s performance beyond temporally-focused benchmarks. We reiterate that ArrowRL is designed as a targeted enhancement for an LMM's temporal sensitivity. As our analysis in Sec. 3.1 shows, many existing video benchmarks do not actually require this capability to be solved (e.g., they can be answered with single, shuffled or reversed frames). For this reason, our main results in Table 1 focus on the benchmarks we identified as temporally sensitive, where ArrowRL shows significant boosts.

The reviewer raises a valuable follow-up question: whether our specialized training harms performance on easier tasks where AoT is not required. To answer this, we have conducted experiments on the benchmarks that our own analysis identified as less sensitive (Fig. 3 right), including VideoMME as suggested by the reviewer.

As shown, ArrowRL preserves or even slightly improves performance across these general video benchmarks that do not demand temporal awareness. This new analysis (same as shown for Reviewer mxwr) confirms that ArrowRL is a targeted enhancement: it provides great gains on temporally-aware benchmarks (as shown in Table 1 with data from 3 relevant and popular benchmarks) without degrading performance on more general benchmarks where AoT is not a primary factor (Ln 280-283). We thank the reviewer for this suggestion and will add this analysis to the final paper.

The authors have comprehensively addressed my questions:

• Regarding the question of whether "the difference between reverse responses and forward responses can reflect temporal directionality," the authors have resolved it through "video source selection" and "dynamically weighting forward/reverse rewards." Overall, this is reasonable.
• The training scheme introduced by the authors can slightly improve the performance on General Video Benchmark.

Therefore, I retain my score as positive.

We thank reviewer mxwr for the helpful comments and for providing thoughtful feedback on our work.

---

1. On the Use of MCQ for Evaluating Temporal Awareness

Most of the multiple-choice questions do not require temporal reasoning and therefore do not clearly demonstrate the need for time awareness.

Thank you for highlighting this critical point, which in fact perfectly states the central motivation for our work. We agree entirely that most existing video MCQ samples do not adequately test for temporal awareness. This widespread issue is exactly what inspired us to develop a more rigorous approach. In Sec. 3.1 & 3.3, we detail our diagnostic analysis of existing benchmarks, propose the Temporal Divergence Score (TDS) metric, which sifts through these inadequate benchmarks and algorithmically identifies the set of questions that are truly sensitive to the Arrow of Time.

For the evaluation itself, as noted in Supp. Ln 344-346, we chose the MCQ format because it is the standard for objective and reproducible LMM evaluation. In contrast, open-ended QA and captioning are notoriously difficult to evaluate and often require LLM-based or human judges, which can introduce noise and reduce comparability [12].

Therefore, the combination of our targeted TDS filtering with a standard, objective evaluation format provides what we are confident is a robust and convincing test for the Arrow of Time. If there are particular aspects about our evaluation that the reviewer finds unconvincing, we would be glad to elaborate further.

---

2. Clarification on Notation $P$

what does P denote in line 167?

We follow the convention used in the original GRPO paper [47], where $P$ is used to denote the underlying data distribution of a dataset. In our context, $P$ refers to the distribution from which we draw our training samples, each consisting of a question $q$ and response $o$ (Ln 165-166). We will revise the sentence to improve clarity.

---

3. On the Necessity of GRPO

Applying GRPO (or other RL methods) may not be necessary. Augmenting the pre-training (or SFT) datasets with reversed videos could potentially address this problem.

We appreciate the comment and agree that SFT is a critical comparison point. In fact, we included this exact experiment in our paper to test the necessity of our RL-based approach. As shown in Table 2, we trained a model using SFT on the exact same instruction tuning datasets designed for ArrowRL. While this approach improves upon the base LMM (57.4% vs. 56.2%), confirming the value of our training data, ArrowRL demonstrates a significant advantage, achieving 60.7%. This result indicates that RL is necessary to unlock the full potential of the training signal.

---

4. Clarification on ArrowRL reward

Why is maximizing the dissimilarity between forward and backward responses essential? Enforcing them to be dissimilar does not guarantee that the reversed response will be correct.

We fully agree with the reviewer that maximizing the dissimilarity between forward and backward responses would not guarantee correctness. Our methodology was designed with this principle in mind. As detailed in Sec. 3.2, ArrowRL’s reward is composed of two parts: (1) a fidelity reward ensures the forward response is correct by rewarding similarity to the ground truth, (2) a reverse reward encourages this correct response to be temporally sensitive by rewarding dissimilarity from reverse response. It is important to clarify that our objective is to generate correct and temporally-aware responses for forward videos, not to ensure the correctness of responses for reversed videos. The reversed video is used solely as a training signal (Ln 202-214).

The final reward is a weighted sum of both components. Fig. 4 and Supp. Fig. 8 illustrate these two reward signals, and Table 2 (along with our response #3 to Reviewer tJ9G) analyzes the effect of varying the weight between them. We thank the reviewer again and are happy to discuss further if there are additional concerns.

---

5. Clarification on Evaluation

The evaluation is not comprehensive. It lacks comparison with other popular video benchmarks (e.g., MVBench, VideoMME).

We appreciate this comment and would like to clarify our principled approach to benchmark selection, which draws from 8 popular existing benchmarks, and then present additional results as requested.

Principled Benchmark Selection. As noted in Ln 46-54, Fig. 2 and Sec. 3.1, a core finding of our work is that many existing video benchmarks are not well-suited to evaluate temporal sensitivity. To address this, we introduced the Temporal Divergence Score (TDS) to quantify each benchmark's reliance on the Arrow of Time. Our analysis of 8 existing benchmarks (Fig. 3 right) revealed that TVBench, TempCompass, and Vinoground are the most temporally sensitive benchmarks. We notably selected TVBench over the older MVBench, as TVBench was specifically designed to address the temporal insensitivity of its predecessor while using much of the same underlying video data. We therefore focused our main evaluation (Table 1) on these benchmarks to provide the clearest measure of improvement on temporal awareness (Ln 157-161, 264-265).

Additional Evaluation on General Benchmarks. To address the reviewer's question about performance on other popular benchmarks that don’t require temporal reasoning (like VideoMME), we have conducted the requested experiments. The question is whether our method harms performance on tasks that do not require AoT reasoning. The results below show that it does not; in fact, we see neutral to slightly positive results across the board.

These results demonstrate that ArrowRL is a targeted enhancement: it provides great gains on temporally-aware benchmarks (as shown in Table 1 with data from 3 relevant and popular benchmarks) without degrading performance on more general benchmarks where AoT is not a primary factor (Ln 280-283). We thank the reviewer for this suggestion and will add this analysis to the final paper.

---

6. Clarification on Fig. 3

Why, in Fig. 3 (left), are the forward and reverse results similar? How should this be explained?

Thank you for your question. Your observation that the forward and reverse results are similar for many of the models shown is entirely correct, and this is precisely the finding that the figure is designed to illustrate.

Our Fig. 3 provides the temporal sensitivity analysis. As noted in Ln 128-136, 155-161, it shows that most existing LMMs are largely insensitive to temporal direction (left) and that current benchmarks fail to meaningfully evaluate AoT perception (right). In other words, even when provided with semantically distinct forward and reverse videos, these models often generate nearly identical responses—demonstrating a lack of temporal awareness, likely caused by an over-reliance on language priors, where the model's textual knowledge overrides the visual evidence of temporal direction (Fig. 1(b), Ln 46–64).

This observed limitation directly motivates our work: we propose ArrowRL to improve temporal sensitivity in LMMs, and AoTBench as a focused benchmark to evaluate this capability. We thank the reviewer again and are happy to discuss further if there are additional concerns.

We thank reviewer tJ9G for the helpful comments and for providing thoughtful feedback on our work.

---

1. Reverse Reward Mechanism

Forcing the model to learn from both forward and reversed videos may lead to the generation of physically implausible descriptions. It is recommended to include an additional set of experiments to analyze and illustrate this potential issue.

We agree that applying a reverse reward indiscriminately would be problematic, and emphasize that ArrowRL was designed specifically with this nuance in mind. As noted in Ln 207-214, ArrowRL incorporates two key mechanisms—high-temporality data curation and dynamic weighting—to ensure that the model learns a true understanding of event dynamics rather than superficial cues.

High-temporality Data Curation. First, at the data level, a portion of our training data is sourced from RTime [14], a benchmark of high-temporality videos where the reversed version is both physically plausible and semantically distinct from the forward version (Ln 251-253). By training on these vetted (manually verified to demonstrate high temporality) videos, ArrowRL learns the core patterns of reversible, meaningful actions without being confused by inapplicable cases. As suggested, we ran an additional experiment to isolate the effect of our data selection strategy. We train a model only for the captioning task, where the high-quality RTime data was replaced by uncurated videos from LLaVA-Video-178K [73]. As the table shows, the model trained on curated, high-temporality data performs better, underscoring the importance of our data selection strategy, and (we think) consistent with the reviewer’s expectation.

Dynamic reward weighting. Next, to ensure scalability beyond curated data like RTime, we also train on the large, uncurated LLaVA-Video-178K dataset for open-ended QA (Ln 248-250), where irreversible events are likely to appear. Our dynamic reward weighting is designed to handle precisely these cases. Consider the glass shattering example the reviewer mentioned: since the reversed action is physically impossible, a model will naturally struggle to describe it. In fact, as we see across many examples, LMMs tend to default to descriptions that are semantically very close to the forward ground truth in such cases. Our mechanism detects this by calculating a high Similarity$(\tilde o, o^*)$ that surpasses a threshold 𝛾. Therefore, if the video is not meaningfully reversible, ArrowRL will "turn off" the reverse reward for that specific example by setting its weight to zero. This ensures our model is not forced into the task of creating a plausible description for an implausible event. The importance of our dynamic weighting is already ablated in Table 2 (row +ArrowRL (𝛾=1)), which shows that removing this dynamic weighting hurts performance.

We believe this two-part strategy ensures that ArrowRL is a nuanced and targeted tool, not a brute-force mechanism, and will clarify this further in the final version of the paper.

---

2. Clarification of AoTBench Construction

It would be helpful to clarify in the paper how the questions in AoTBench were constructed, whether they were directly extracted from existing datasets or re-annotated for forward and reversed videos.

We designed AoTBench to be a dedicated test of LMM temporal sensitivity, rather than a generic collection of difficult samples. As detailed in Sec. 3.3 and Supp. Sec. B.3, AoTBench is a multi-faceted benchmark where each component is designed to probe this specific capability.

The first two tasks, sequence direction classification & directional caption matching, are inherently temporal. For these, we use videos from existing datasets and design questions that require the model to classify a video's playback direction or match it to a corresponding forward/reverse caption (Table 4 & Fig. 9). These tasks cannot be solved without understanding temporal directionality.

For the third task, AoT-sensitive VQA, we compose data samples from 8 existing video benchmarks. Rather than pulling generically "hard" questions from existing benchmarks, we develop a more principled approach. We introduce the Temporal Divergence Score (TDS), a metric that precisely quantifies how critical the video's AoT is to arriving at the correct answer (Sec. 3.1). TDS allows us to analyze samples from different benchmarks and select only those that require true temporal perception. This principled filtering guarantees our VQA subset is a true test of temporal perception, not just a collection of unrelated hard cases, as illustrated by the examples in Fig. 6 & 8.

---

3. Different values of α & 𝛾

If possible, providing a more detailed analysis of how different values of 𝛼 and 𝛾 affect training performance would make the paper more comprehensive and informative.

We appreciate the suggestion and have conducted additional experiments on the hyperparameters 𝛼 (reverse reward weight) and 𝛾 (dynamic weighting threshold), building upon the ablations already in Table 2.

Analysis of Reverse Reward Weight (𝛼). Our main paper reports results for our chosen value of 𝛼 = 0.5 and includes an ablation of 𝛼 = 0 (disabling the reverse reward). As requested, we have now evaluated a fuller range of values. The results below show the performance across different weights.

These results confirm that the reverse reward is a critical component of our method. Performance is poor at the extremes (𝛼 = 0 and 𝛼 = 1.0), but strong and stable across the range of [0.25, 0.75]. This detailed analysis identifies 𝛼 = 0.25 as the optimal weight, and we will update our model and paper accordingly.

Analysis of Dynamic Weighting Threshold (𝛾). Our main paper reports results for our chosen value of 𝛾 = 0.5 and ablates the case with no thresholding (𝛾 = 1). We have now tested additional intermediate values as suggested. Note that 𝛾 = 0 is an invalid setting as it would filter out all training data. The extended analysis is below.

This analysis confirms our dynamic weighting is a critical component, as removing it (𝛾 = 1) greatly degrades performance. The method is also robust, with stable and strong results across the [0.25, 0.75] range. This detailed analysis identifies 𝛾 = 0.25 as the optimal weight. We thank the reviewer for this valuable suggestion and will update our model and paper accordingly.

Dear Reviewer tJ9G,

The system noticed that you haven't post any discussion with the authors yet. As per the review guidelines, reviewers are expected respond to authors’ rebuttal, ask further questions (if any) and listen to answers to help clarify remaining issues before submitting Mandatory Acknowledgement. Please engage with the authors now to offer your feedback on their rebuttal.

Thank you!

AC, NeurIPS 2025