Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding

We appreciate the time and effort you invested in providing these detailed observations, questions, and comments. We have carefully considered your comments and outlined our responses and proposed revisions below. We hope these clarifications and revisions address your concerns and further enhance the manuscript.

W1: This work's post-training strategy is almost the same as GRPO.

• Time-R1 is the first work to introduce the RLVR-style post-training paradigm into temporal video grounding. We address the critical yet underexplored challenge TVG task in this domain by proposing a novel paradigm that fundamentally redefines how long-video understanding can be approached. Our method not only offers a new perspective but also achieves state-of-the-art performance.
• Our contribution lies in a complete post-training framework that includes Time-R1, TimeRFT, and TVGBench. GRPO serves as a key training objective within Time-R1, but is just one component of the overall comprehensive design.
• Moreover, we provide detailed ablations to dissect GRPO's design, such as in Appendix Table S3: Ablation of KL and Thinking Process in GRPO; Table S4: Comparison of DAPO and GRPO; and Ablation Study on Reward Design added during rebuttal below. These insights contribute to a deeper understanding of how RL paradigm impacts long-video understanding reasoning.

W2 & Q1: Missing ablation to validate the reward design.

Following your suggestions, we provide a more comprehensive ablation on reward design to demonstrate the effectiveness of our tIoU reward:

• original IoU reward: $r\_{\rm{IoU}}=\frac{[t\_s,t\_e]\cap[t\_s',t\_e']}{[t\_s,t\_e]\cup[t\_s',t\_e']}$
• only the second term of the tIoU-based reward: $r\_{\rm{tIoUsec}}=1-\frac{|t\_s-t\_s'|}{t}\cdot\frac{|t\_e-t\_e'|}{t}$
• Exact Matching: $r\_{\rm{em}}=1\ \rm{if\ t\_s=t\_s'\ and\ t\_e=t\_e'\ else\ }0$
• Absolute Error: $r\_{\rm{abs}}=1 - \frac{|t\_s-t\_s'| + |t\_e-t\_e'|}{2t}$
• RMSE: $r\_{\rm{rmse}}=1 - \sqrt{(\frac{t\_s-t\_s'}{t})^2 + (\frac{t\_e-t\_e'}{t})^2}$

All ablation studies here include the format reward, we omit it for simplicity.

Experimental results show that our tIoU outperforms all candidate methods.

Q2: Qualitative examples to illustrate advantage of tIoU over IoU.

Compared to IoU, tIoU offers a more reasonable definition by introducing a penalty for unreasonable boundary predictions. As shown in our above table in response to Q1, tIoU consistently outperforms IoU across all three benchmarks. Since adding videos or images is not allowed during the rebuttal phase, we will include qualitative examples demonstrating the advantages of tIoU over IoU in the appendix in our revised version.

We appreciate the time and effort you invested in providing these detailed observations, questions, and comments. We have carefully considered your comments and outlined our responses and proposed revisions below. We hope these clarifications and revisions address your concerns and further enhance the manuscript.

W1: Could the SOTA performance be attributed to a stronger backbone rather than the proposed training method and reward design?

In the following experiments, we aim to show that (1) open-sourced academic models, like TimeSuite, can benefit from our RL post-training; and (2) many well-pretrained model across model sizes and backbones can be substantially improved via our training procedure.

We conducted additional experiments with RL-based post-training on other base models, including TimeSuite-7B, Qwen-2.5-VL-3B, MIMO-VL-7B, InternVL-2B, and InternVL-8B. We post-train all base models using our 2.5K data filtered by Qwen-2.5-VL-7B. The results are presented as follows:

We observe several key findings, revealing that our approach is generally applied to many base models.

First, models of different sizes can also benefit from our approach. For instance, the Qwen-VL-3B model achieves a +13.4 mIoU improvement on TVGBench and a +34.9 mIoU improvement on Charades. This is attributed to the fact that our training data is distilled based on filtering from Qwen-VL-7B, demonstrating that our RL-based method can effectively learn from the data.

Second, models with different backbones also benefit. For example, InternVL-8B achieves a +12.4 mIoU on TVGBench and +23.5 on Charades, while MiMo-VL-8B improves by +11.7 and +22.2 on TVGBench and Charades, respectively.

Additionally, we notice that when using weaker model with a different backbone, the transferability of the training data becomes less effective, but still beneficial. For example, training InternVL-2B on data filtered by Qwen-2.5-VL-7B only yields marginal gains, improving only +3.1 on TVGBench and +0.3 on Charades. Similar phenomenon is observed on TimeSuite-7B backbone. This is mainly due to architectural differences, base model's intelligence, and the data filtered by Qwen-2.5-VL-7B are not well-suited for RL training on InternVL-2B. However, we expect that filtering the data using base model itself could yield improvements comparable to those observed when transferring from Qwen-2.5-VL-7B to Qwen-2.5-VL-3B.

W2: What is the role of tIoU and under what conditions is it most effective?

tIoU is particularly effective when combined with multi-epoch (ME) training. Further including Gaussian filtering and sample filtering consistently contribute to the performance gains.

Comparing Row 2 and Row 6 in Table 2, the combination of tIoU and ME achieves the best performance on R1@0.7, with a score of 16.4. In contrast, using IoU with ME (Row 4) yields lower performance on R1@0.7 of 14.2.

In Appendix Figure S2, tIoU consistently improves performance as the number of training epochs increases, and tIoU outperforms IoU in the multi-epoch training. We attribute this to the superior boundary sensitivity of tIoU.

In addition, Gaussian filtering (GF) consistently improves model performance, and its effectiveness is further amplified when combined with tIoU. Specifically, comparing Row 1 and Row 3 (IoU + GF), as well as Row 2 and Row 6 (tIoU + GF), we observe that tIoU leads to a performance gain of +3.5%, whereas IoU yields only a marginal improvement of +0.2%. Furthermore, sample filtering, as shown in the last row, provides additional stability to the model’s performance, demonstrating its complementary benefit in our training pipeline.

W3: The paper lacks ablation studies on the template reward function.

Below, we provide an ablation of the format reward. Comparing row 1 and row 2, the format reward consistently improves the performance. While training without reasoning format reward would lead to better performance on R1@0.3/0.5/0.7 on Charades, the actual mIoU of training with format reward outperforms training without format reward.

Moreover, if you have interest, we provide a more comprehensive ablation study on the choice of tIoU in response to Reviewer okJ6.

W4: The inference speed of this method seems slow. Is there any comparison available?

To address this, we conduct a comparison of inference speeds with and without CoT reasoning process. We adapt our model for implementing both vLLM and HuggingFace Transformers library, and evaluate the inference speed under both frameworks. The results below demonstrate that with vLLM library, our method achieves substantial improvements in inference efficiency, significantly reducing latency while maintaining strong performance.

Therefore, for a 1.5-minute input video sampled at 2 fps, our model—when deployed with vLLM—achieves an average inference time of 4.98 seconds with CoT and 4.14 seconds without CoT, which is actually not too slow. Given that most models adopt similar MLLM architectures, their inference speed is comparable with ours inference without CoT.

Q1: Comparison of training efficiency between RL and SFT.

We compared RL and SFT across various efficiency metrics, including training time, memory usage and experimental results below. To further demonstrate the data efficiency of the RL paradigm, we trained an SFT model with full size 339K TVG samples and found that it still underperforms compared to our RL model trained with only 2.5K samples. The SFT-LoRA-2.5K only underperforms SFT-LoRA-339K slightly, and even slightly surpasses SFT-LoRA-339K on ActivityNet.

These results highlight: (1) the strong potential of RL to achieve higher performance with significantly less data. (2) our data filtering strategy is both effective for RL and SFT. While RL incurs higher computational cost, its superior performance with minimal data suggests strong potential for future advancements.

Q2: Typo in Table 2 and in Supplementary Table S5

We apologize that "TimeZero-7B" is a typo. It should refer to the "Time-R1-7B". We will fix these typos in our revised version.

I appreciate the authors’ kind and detailed responses, which have addressed most of my concerns.

While considering other reviews as well, I would like to point out as a weakness that the performance improvement from the proposed tIoU is not marginal, and that too many techniques (tIoU, format, Gaussian filtering, multi-epoch, sample filtering, etc.) are combined empirically.

I have decided to increase my score to borderline accept. I hope that by reflecting on the rebuttal well, the paper will be finalized as a strong work.

We appreciate the time and effort you invested in providing these detailed observations, questions, and comments. We have carefully considered your comments and outlined our responses and proposed revisions below. We hope these clarifications and revisions address your concerns and further enhance the manuscript.

W1: The necessity of designing TVGBench

As stated in Figure 3 and Section 3.4 of our paper, existing benchmarks are often either excessively large or overly domain-specific, which poses challenges for effectively evaluating large multimodal models. In contrast, TVGBench is deliberately designed to be compact yet comprehensive. To show its advantage, we illustrate the comparison between benchmarks used in this work:

Noting that TVGBench encompasses a broader range of data sources, we built our implementation on top of the vLLM library to achieve faster inference speed. As our framework would be fully open-source, it provides a useful suite for the community.

Moreover, although TVGBench is constructed by curating samples from existing datasets, we re-annotated each sample with consistent semantics and re-balanced the distribution. This ensures the benchmark's reliability and representativeness, allowing it to serve as a useful evaluation suite.

W2: Solid work, novel but not much

Thank you for recognizing the solidity and novelty of our work. We address a crucial and underexplored challenge in long-video understanding, proposing a new RL paradigm that fundamentally shifts how this problem is approached. Our method not only introduces a fresh perspective but also achieves state-of-the-art performance, demonstrating both its practical effectiveness and scientific contribution. Furthermore, we will publicly release all our training data, training code, and inference code, which we believe will be valuable to the research community.

Q1: Will improvements be made on different base model and model sizes?

Yes, they would! Following your suggestions, we extend our evaluation to include Qwen-2.5-VL-3B, MIMO-VL-7B, InternVL-2B, InternVL-8B, and TimeSuite-7B. Due to the time limit, we post-train all base models using our 2.5K filtered by Qwen-2.5-VL-7B. The results are presented as follows:

We observe several key findings, revealing that our approach is generally applied to many base models.

First, models of different sizes can also benefit from our approach. For instance, the Qwen-VL-3B model achieves a +13.4 mIoU improvement on TVGBench and a +34.9 mIoU improvement on Charades. This is attributed to the fact that our training data is distilled based on filtering from Qwen-VL-7B, demonstrating that our RL-based method can effectively learn from the data.

Second, models with different backbones also benefit. For example, InternVL-8B achieves a +12.4 mIoU improvement on TVGBench and +23.5 on Charades, while MiMo-VL-8B improves by +11.7 and +22.2 on TVGBench and Charades, respectively.

Additionally, we notice that when using a smaller model with a different backbone, the transferability of the training data becomes less effective, but still beneficial. For example, training InternVL-2B on data filtered by Qwen-2.5-VL-7B only yields marginal gains, improving only +3.1 on TVGBench and +0.3 on Charades. Similar phenomenon is observed on the TimeSuite-7B. This is mainly due to architectural differences, and the data filtered by Qwen-2.5-VL-7B are not well-suited for RL training on InternVL-2B. However, we expect that re-filtering the data using InternVL itself could yield improvements comparable to those observed when transferring from Qwen-2.5-VL-7B to Qwen-2.5-VL-3B.

Q2: Does the improvement in VQA performance come from using VQA data during RL?

No, and we sincerely apologize for the confusion. As clarified in the Appendix, there was a typo in the main text—we did not use any VQA data during reinforcement learning. The improvement is solely attributed to our proposed method, without reliance on external VQA supervision.

Q3: Could you provide an ablation study on the reward design?

Yes, certainly. In response to your request, we provide an comprehensive ablation on the reward design as follows:

• the CoT template: $r\_{\rm{format}}$
• original IoU reward: $r\_{\rm{IoU}}=\frac{[t\_s,t\_e]\cap[t\_s',t\_e']}{[t\_s,t\_e]\cup[t\_s',t\_e']}$
• only the second term of the tIoU-based reward: $r\_{\rm{tIoUsec}}=1-\frac{|t\_s-t\_s'|}{t}\cdot\frac{|t\_e-t\_e'|}{t}$
• Exact Matching: $r\_{\rm{em}}=1 \rm{if\ t\_s=t\_s'\ and\ t\_e=t\_e'\ else\ 0}$
• Absolute Error: $r\_{\rm{abs}}=1 - \frac{|t\_s-t\_s'| + |t\_e-t\_e'|}{2t}$
• RMSE: $r\_{\rm{rmse}}=1 - \sqrt{(\frac{t\_s-t\_s'}{t})^2 + (\frac{t\_e-t\_e'}{t})^2}$

The results, shown below, demonstrate the effectiveness of the tIoU reward design.

Experimental results demonstrate that our combination of tIoU and format reward achieves the best overall performance. Without the format reward, the model can still perform reasonably well even without CoT. However, without the tIoU reward, it becomes difficult to effectively activate the model’s TVG capabilities. While training with $r\_{\rm{tIoU}}$ would lead to better performance on R1@0.3/0.5/0.7 on Charades, the actual mIoU of training with $r\_{\rm{tIoU}}+r\_{\rm{format}}$ outperforms it.

Dear Reviewer XTMy,

We sincerely appreciate your recognition of our efforts to address your concerns. We are grateful for your positive feedback and valuable suggestion.

As you recommended, we will incorporate the relevant experiments into the revised version of our paper.

We welcome any further discussion and would be happy to provide additional clarification if needed.

Best regards,

Authors

We appreciate the time and effort you invested in providing these detailed observations, questions, and comments. We have carefully considered your comments and outlined our responses and proposed revisions below. We hope these clarifications and revisions address your concerns and further enhance the manuscript.

W1: Why does the TVG task help improve VQA performance?

From the data perspective, the TVG task equips the model with the ability to localize relevant segments based on a textual query. This capability is crucial for long-video VQA, as it helps the model focus on the most pertinent moments and reduces the distraction from irrelevant or redundant information.

From the learning perspective, our RL-based training mitigates the risk of overfitting to noisy or false negative data by allowing the model to learn from its own generated samples, enhancing the generalization ability of the model. It is the RL loss design and training paradigm that incentivizes model's capability without sacrificing its original ability.

Furthermore, prior work such as TimeSuite[ICLR'25] has shown that jointly training on TVG + VQA data via supervised fine-tuning (SFT) significantly boosts VQA performance. This indirectly validates the utility of the TVG task. Our RL-based paradigm achieves similar benefits without relying on any VQA data, highlighting its ability to enhance general-purpose video understanding.

W2: Why does SFT underperforms base model?

We have provided some clarifications and additional results in Appendix Figure S1. Specifically, we implemented two versions of supervised fine-tuning: Full-parameter fine-tuning of the LLM (denoted as SFT) and LoRA-based fine-tuning of the LLM (denoted as SFT-LoRA). Our RL training full-finetunes the LLM. We present the detailed experiment results below:

We observed that full fine-tuning via SFT leads to severe overfitting. While SFT-LoRA partially mitigates this issue and improves performance on TVG tasks, it still leads to a slight degradation in VQA performance as shown in Figure S1.

In contrast, RL consistently preserves generalization and improves performance. For example, on ActivityNet, RL boosts mIoU from 16.3 to 29.2, whereas SFT drops it to 15.4, and SFT-LoRA only improves it to 25.9. On the VideoMME VQA benchmark, RL improves performance from 53.0 to 54.2, while SFT-LoRA decreases it to 51.7.

W3: Why use dense video captions as cold-start data?

We observed that, without any cold-start initialization, the model gradually learns to generate video descriptions through Chain-of-Thought (CoT) reasoning. Building on this empirical observation, we employ SFT-based cold-start using dense video captions to accelerate convergence toward the format reward and to reduce the length of the reasoning chains.

W4: The paper lacks baselines and benchmark comparisons.

Thanks for your suggestions, we will expand our related work section to include discussion and citation of the suggested methods. In addtion, we now provide comparisions against additional baseline methods with corresponding evaluation metrics on Charades and TVGBench. As shown below, our Time-R1 still achieves state-of-the-art results:

Regarding the benchmark comparison:

• CG-Bench[8] is a multiple-choice benchmark, which by nature is not a timestamp prediction task.
• E.T.Bench[2], as shown in Table 4 on Page 11 of its paper, uses TVG data limited to Charades, EgoNLQ, and QVHighlights.
• RexTime[7] only includes ActivityNet and QVHighlights.

In contrast, our benchmark covers Charades, ActivityNet, EgoNLQ, TaCoS, and HiREST, with balanced distribution on video length, video domain and query center, making it potentially a more comprehensive benchmark for the true TVG task. Besides, we re-annotated each sample with consistent semantics and re-balanced the distribution. This ensures the benchmark's reliability and representativeness, allowing it to serve as a useful evaluation suite.

Thank you for your responses. We address your additional comments as follows:

W2: Hyperparameters for SFT-LoRA and parameter tuning.

Our current hyperparameters: For SFT-LoRA, the LoRA rank is set to 64, and the LoRA alpha is set to 128. Both SFT and SFT-LoRA use the following settings: a learning rate of 2e-5, the AdamW optimizer with $\beta\_1$=0.9, $\beta\_2$ = 0.999, a weight decay of 0, the batch size of 8, and accumulation steps of 2. We fine-tune for 5 epochs on our 2.5K data, and use a linear scheduler to gradually decay the learning rate to 0.

Parameter tuning for SFT-LoRA: To further address your concerns, we provide an ablation study of SFT-LoRA below on the key LoRA hyperparameters, as well as the number of training epochs. We conclude that under all current configurations, RL training outperform all SFT trainnig settings by a large margin.

Due to time constraints, we are unable to evaluate on ActivityNet. which requires too much time for evaluation.

Regarding the ablation on SFT-LoRA: Our ablation study demonstrates that the SFT-LoRA configurations of 128/256 and 256/512 yield the best performance. Specifically, the 128/256 setting performs better on TVGBench, while 256/512 achieves superior results on Charades. Second, increasing the number of training epochs does not necessarily degrade performance on the TVG task. For instance, the 128/256 setting continues to improve with longer training, whereas the 64/128 setting tends to overfit as training progresses.

W3: The effect of using dense video caption data for SFT cold-start.

We summarize the role of SFT as a cold-start as follows:

• Caption data as Chain-of-Thought (CoT) provides video content priors as text: Intuitively, CoT-style video captions help the model better understand the visual scene and offer a prior over the visual content, which may be beneficial for generating timestamp predictions conditioned on the query. This textual prior makes it easier for the MLLM to infer the query-relevant temporal segments.
• Functional role in controlling CoT length: One key motivation for using SFT is to control the length of the CoT. We empirically observed that the model tends to generate CoTs resembling video captions, so we naturally opted to use caption data to regulate the CoT length.

We present the results of the 3B model here to illustrate the role of SFT as follows:

We leave video CoT and complex reasoning as Future work: In this work, our focus is introducing a new RL-based paradigm for the TVG task and demonstrating that CoT plays a supportive role. The CoT reasoning is an important direction for video reasoning, and cold-starting with SFT is a critical mechanism for enabling the model to effectively generate more complex CoTs. A promising future direction is to develop models capable of generating more complex CoTs to tackle harder reasoning tasks, such as multi-hop query understanding. We leave the exploration of more effective CoT designs and better cold-start strategies for future work.