Universal Video Temporal Grounding with Generative Multi-modal Large Language Models

Thanks for the constructive comments. We provide our responses as follows.

---

Q1: Fine-grained Prediction

We appreciate your insightful observations. Our method ensures fine-grained temporal prediction through the following design choices:

• We utilize 2 FPS for video processing, which offers relatively fine temporal resolution. This aligns with standard practices in video temporal grounding (VTG)—e.g., SnAG [1] (1 FPS) and UniVTG [2] (0.5 FPS).

• For long videos, we adopt a coarse-to-fine grounding strategy in terms of temporal resolution to effectively distinguish closely adjacent events.

• Our framework supports arbitrary temporal resolutions and can be adapted for finer precision. To validate this, we conducted experiments on Charades at 4 FPS, achieving slightly improved performance:

  FPS	R1@0.5	R1@0.7	mIoU
  2 (default)	74.33	53.71	63.15
  4	75.27	54.89	63.97

---

Q2: Query-guided Frame Importance Estimation

We appreciate your valuable suggestion. A lightweight module for dynamic frame importance estimation—enabling adaptive token allocation—is indeed a promising direction. We fully acknowledge the potential of this idea and plan to explore it in future research.

---

Q3: Timestamp Retrieval vs. Explicit Temporal Alignment

We appreciate your insightful suggestion regarding explicit temporal alignment supervision. After careful consideration, we opted not to include such supervision for the following reasons:

• Inherent Multimodal Retrieval Capabilities in MLLMs:

  • Modern MLLMs exhibit strong multimodal retrieval abilities without explicit supervision, as evidenced by tasks like Needle-in-a-Haystack (NIAH) [3,4], which require models to accurately retrieve target images ("needles") within long videos ("haystacks").
  • By reformulating temporal grounding as a timestamp retrieval task using timestamp-interleaved sequences, we leverage and extend these inherent capabilities to temporal grounding, eliminating the need for auxiliary alignment supervision.

• Theoretical Equivalence to Explicit Alignment Methods:

  • Existing methods (e.g., GeLM [5]) employ alignment supervision by optimizing a similarity matrix between special tokens and visual features, ultimately enabling the token to effectively retrieve query-relevant visual features.
  • Although our generative method differs from discriminative methods based on explicit alignment, it follows a similar principle: accurately retrieving timestamp tokens that correspond to query-relevant visual features, thereby achieving comparable grounding without requiring explicit contrastive supervision.

---

Q4: Unfreezing the Visual Backbone During Training

We appreciate your valuable suggestion. To investigate its impact, we conducted experiments where we unfroze the visual backbone of Qwen2-VL-2B (using LoRA) during training and evaluated performance on Charades-STA:

These findings validate your insight and suggest that our current approach could potentially benefit from this modification. Accordingly, we plan to incorporate this adjustment when retraining our UniTime-Full model.

---

Q5: Latency and Computational Cost

Thank you for your valuable feedback. While Table 4 in our original submission demonstrates the performance advantages of our multi-stage approach, we have now conducted a comprehensive computational analysis to quantify the associated overhead. Using the calflops library [6], we measured the latency, TFLOPs, and TMACs per query for videos from the Ego4D dataset, comparing multi-stage and single-stage inference:

• The multi-stage approach achieves 1.8× higher mIoU (17.25 vs. 9.83)
• This improvement comes with 5.2× higher latency and 3.6× greater computational cost (TFLOPs and TMACs)

This analysis reveals the fundamental trade-off between grounding accuracy and computational efficiency in our method. Future work could explore efficiency optimizations.

---

Q6: Impact of Frame Thresholds and Token Budgets

We appreciate your insightful questions regarding these key hyperparameters. To systematically evaluate their effects, we conducted comprehensive ablation studies:

• Threshold $N_f^{long}$

  • Role: Determines the maximum video length that can be processed directly

    • If video length ≤ $N_f^{long}$: Process directly.
    • If video length > $N_f^{long}$: Split into segments, process separately, and aggregate results.

  • Evaluation on Ego4D-NLQ:

    $N_f^{long}$	R1@0.3	R1@0.5	mIoU
    1024 (default)	24.79	16.83	17.25
    512	24.72	16.63	16.95
    256	23.94	16.10	16.68

  • Key Findings:

    • The choice of $N_f^{long}$ has minimal impact on model accuracy because our method segments over-length videos and aggregates results.
    • Adaptive Frame Scaling enforces a fixed token budget per segment. Consequently, a smaller $N_f^{long}$ increases inference time due to more video segments and more input tokens.
    • Default Choice: 1024 for optimal efficiency-performance balance.

• Threshold $N_f^{short}$

  • Role: Decides single- or multi-stage processing:

    • If video length ≤ $N_f^{short}$: Single-stage
    • If video length > $N_f^{short}$: Multi-stage (coarse-to-fine)

  • Evaluation on TaCoS:

    $N_f^{short}$	mIoU (64-128s)	mIoU (128-256s)	Avg. mIoU
    64	58.43 (multi)	54.57 (multi)	50.28
    128 (default)	56.59 (single)	54.57 (multi)	50.00
    256	56.59 (single)	42.90 (single)	46.82

  • Key Findings:

    • For 64-128s videos: Minimal performance difference between multi-stage ($N_f^{short}$=64) and single-stage ($N_f^{short}$>=128) processing (ΔmIoU=1.84). However, the lower threshold increases average inference time as it forces more videos into multi-stage processing.
    • For 128-256s videos: Multi-stage processing yields significantly better results (54.57 vs 42.90 mIoU)
    • Default Choice: 128 to balance performance and efficiency.

• Token Budget

  • Evaluation on Ego4D:

    Token Budget	R1@0.3	R1@0.5	mIoU
    16384 (default)	24.79	16.83	17.25
    8192	24.08	16.10	16.69
    4096	23.04	15.07	15.96

  • Key Findings: Performance improves consistently with higher token budgets (up to GPU limits).

---

Q7: Multiple Valid Grounding Regions

For queries associated with multiple valid temporal moments (e.g., in QVHighlights), we incorporate all ground truth moments in chronological order within the training sequence. By doing so, the model learns to accurately handle ambiguous queries that may correspond to multiple grounding regions or moments. During inference, UniTime can naturally predict multiple valid moments for such queries, maintaining consistency with the training paradigm.

---

We will incorporate the analysis, along with additional experiments, in our final draft.

---

[1]: Fangzhou Mu, Sicheng Mo, and Yin Li. "Snag: Scalable and accurate video grounding." In CVPR, 2024.

[2]: Lin, Kevin Qinghong, et al. "Univtg: Towards unified video-language temporal grounding." In ICCV, 2023.

[3]: Zhang, Peiyuan, et al. "Long context transfer from language to vision." arXiv:2406.16852, 2024.

[4]: Zhao, Zijia, et al. "Needle in a video haystack: A scalable synthetic framework for benchmarking video mllms." arXiv:2406.09367, 2024.

[5]: Chen, Qirui, Shangzhe Di, and Weidi Xie. "Grounded multi-hop videoqa in long-form egocentric videos." In AAAI, 2025.

[6]: Ye, Xiaoju. "calflops: a FLOPs and params calculate tool for neural networks in pytorch framework." 2023.

Thank you for your constructive comments. We have provided our responses below.

---

Q1: General Video Understanding Capabilities

• Our goal is to enhance temporal grounding by leveraging their strong multimodal understanding. While task-specific fine-tuning may reduce generality, our focus is on improving temporal reasoning, which is critical for many downstream applications.

• UniTime uses pluggable LoRA parameters, allowing the base MLLM’s original video understanding capabilities to be fully restored by detaching LoRA. This is exactly how we conduct VideoQA evaluation in Table 6 in our original submission: first perform temporal grounding with UniTime-Full, then detach LoRA and answer questions using the grounded video segment.

---

Q2: Baseline Comparison Experiments

Thank you for your feedback. Below, we address the raised points in detail.

• Baseline Comparison on VTG Task

  Method	Ego4D mIoU	TaCoS mIoU	Charades mIoU	ANet-Captions
  Qwen2-VL-7B	0.46	2.64	8.98	5.97
  Qwen2.5-VL-7B	0.82	5.29	52.42	21.85
  UniTime-SP (Qwen2)	17.25	45.02	63.15	52.38
  UniTime-SP (Qwen2.5)	18.80	50.00	64.55	51.41
  UniTime-Full (Qwen2)	18.80	50.00	64.55	51.41

  Here, we evaluate the Video Temporal Grounding (VTG) performance of Qwen2-VL-7B and Qwen2.5-VL-7B across multiple benchmarks. Both variants of our UniTime model, UniTime-SP (dataset-specific setting) and UniTime-Full (universal setting), demonstrate significantly superior performance compared to the Qwen-VL baselines, especially on long-form video benchmarks.

• Baseline Comparison on VideoQA Task: Comparisons on video question-answering (VideoQA) are presented in Table 6 in our original submission:

  • The first row, "Uniform Sample", denotes the Qwen2-VL-7B baseline that uniformly samples 32 frames.
  • The last row, "UniTime-Full", also stated in Q1, first grounds the relevant video segment with UniTime-Full, detaches LoRA, and then answers questions on the grounded clip.
  • UniTime significantly enhances the baseline MLLM's VideoQA accuracy (e.g., +6.43 QA accuracy on CG-Bench).

• Ablation on Main Modules: In the main draft, Table 4 presents our module ablation study with full implementation details in Appendix A.2.1. Here, we are happy to systematically analyze how each component enhances VGT performance upon the baseline:

  Configurations	R1@0.3	R1@0.5	mIoU
  Qwen2-VL-7B	0.48	0.17	0.46
  +Finetune	14.25	7.54	9.83
  +Iterative Grounding	18.42	12.13	12.78
  +Adaptive Scaling	17.91	11.99	12.39
  +Segment Retrieval	24.79	16.83	17.25

  The above results were evaluated on Ego4D-NLQ:

  • Qwen2-VL-7B: The baseline sparsely samples frames from long videos, yielding poor accuracy (0.46 mIoU) caused by significant visual information loss.
  • +Finetune: While task-specific finetune delivers +9.37 mIoU gain, it retains sparse sampling limitations, remaining suboptimal (-7.42 mIoU compared to the last row).
  • +Iterative Grounding: Multi-stage grounding improves accuracy (+2.95 mIoU vs. finetuned baseline). However, it still suffers from the omission of keyframes due to temporally sparse sampling, resulting in the inability to generate accurate coarse-grained predictions (-4.70 mIoU compared to the last row).
  • +Adaptive Scaling: Dense sampling with token reduction slightly decreases performance (-0.39 mIoU) but is essential for enabling Segment Retrieval, wtih sufficient frames sampled.
  • +Segment Retrieval: Our full approach implements a coarse-to-fine grounding strategy that first identifies relevant video segments at a coarse level and then performs precise grounding within these segments. This delivers substantial performance gains (+16.79 mIoU over baseline, +4.86 mIoU over Row 4).

---

Q3: Comparison with SOTA VideoQA Methods

Thank you for your question. We’d like to clarify a few key points:

• UniTime is specifically designed for video temporal grounding (VTG) and is not directly comparable to end-to-end VideoQA methods.
• In Table 6 (in our original submission), we use VideoQA benchmarks solely to evaluate how effectively different VTG models identify question-relevant video segments.
• UniTime can be seamlessly integrated with various Video-LLMs to enhance their performance by improving temporal grounding, reducing noise from irrelevant frames, and focusing computation on critical moments.

Below, we evaluate using VideoChat-Flash@224 as the base VideoQA model:

• To ensure fair comparison under identical environments, we first reproduce the baseline results.
• UniTime's temporal grounding provides consistent improvements over the reproduced baseline (+2.1 on LongVideoBench, +1.1 on MLVU).

---

Q4: Flexibility Verification

Thank you for your interest in the generalizability of our approach. In Appendix G, we provided preliminary experiments with different model sizes of Qwen2-VL and Qwen2.5-VL. To further validate flexibility, we conducted additional experiments by integrating UniTime with various scales of InternVL-2.5 models:

The results demonstrate that our approach serves as an effective plug-and-play module, adapting well to different architectures and scales. For the final draft, we plan to extend testing to include a wider variety of models.

---

Q5: Closed-source Model Evaluation

Thank you for the valuable suggestion. To evaluate closed-source models, we conducted temporal grounding experiments on a randomly sampled subset of 100 videos from both the Ego4D and Charades datasets. It's important to note that we do not have insight into the training data used by these closed-source models, which could potentially result in data leakage.

The results are summarized below:

• Doubao achieved the best performance on short videos (72.2 Charades mIoU).
• Gemini-2.5-pro showed superior grounding capability for long videos (20.5 Ego4D mIoU).
• These closed-source models struggle to achieve high performance concurrently on both long and short videos, indicating a lack of generalization.
• UniTime achieves competitive performance compared to these closed-source models while maintaining more balanced capabilities for both short and long-form video grounding benchmarks, proving our approach to be overall the best in terms of consistent performance across varying video types.

---

We sincerely appreciate these valuable comments. Based on your suggestion, we will incorporate these comprehensive evaluation results and analyses in our final manuscript revision.

Thank you for your constructive comments. Our responses are provided below.

---

Q1: Novelty of Timestamp-Interleaved Sequences

Thank you for the feedback. We would like to clarify the key novelty of UniTime’s timestamp-interleaved sequence representation as follows:

• Dynamic Timestamp Resolution: Unlike prior methods that rely on fixed per-frame insertion, UniTime introduces dynamic resolution control. Timestamp tokens can be interleaved at multiple scales—either per frame for fine-grained predictions or per segment for coarse-grained reasoning. This adaptability enhances flexibility in temporal modeling (see Section 2.2, Timestamp-Interleaved Sequence Construction).

• Multi-Scale Temporal Predictions: Coupled with adaptive frame scaling, UniTime supports coarse-to-fine inference over varying video lengths. This improves grounding accuracy on long videos, addressing limitations of prior methods that rely on sparse sampling or short clips.

---

Q2: Explanation for Frame Resizing and Token Compression Choices

Thank you for the insightful comments. To resolve the ambiguity, we conducted an ablation study on the Ego4D-NLQ dataset, evaluating the effects of different video processing strategies on grounding performance and computational efficiency:

Our findings indicate that:

• Long Video: Adaptive Frame Scaling reduces tokens per frame while keeping the frame rate fixed. Though some visual cues are lost, this feature-level token compression retains more semantic information than frame resizing, which discards input resolution. This leads to better segment retrieval accuracy and recall (Rows 1 vs. 2).

• Short Video: For short clips with sufficient tokens, both methods perform similarly (Rows 1 vs. 3). However, token compression requires high-resolution inputs before merging, incurring higher computational costs. In UniTime-Full, using token compression for short videos increased training time by 5 days compared to frame resizing.

Therefore, instead of a unified strategy, we adopt a hybrid approach to balance performance and efficiency: frame resizing for short videos (for computational efficiency) and token compression for long videos (to preserve visual details). This design achieves strong performance with practical training costs.

---

Q3: Ablation Study of Adaptive Frame Scaling Hyperparameters

Thank you for the thoughtful feedback. We have conducted a systematic ablation study on two key hyperparameters:

• Threshold $N_f^{long}$

  • Role: Determines the maximum video length that can be processed directly

    • If video length ≤ $N_f^{long}$: Process directly.
    • If video length > $N_f^{long}$: Split into segments, process separately, and aggregate results.

  • Evaluation on Ego4D-NLQ:

    $N_f^{long}$	R1@0.3	R1@0.5	mIoU
    1024 (default)	24.79	16.83	17.25
    512	24.72	16.63	16.95
    256	23.94	16.10	16.68

  • Key Findings:

    • The choice of $N_f^{long}$ has minimal impact on model accuracy because our method segments over-length videos and aggregates results.
    • Adaptive Frame Scaling enforces a fixed token budget per segment. Consequently, a smaller $N_f^{long}$ increases inference time due to more video segments and more input tokens.
    • Default Choice: 1024 for optimal efficiency-performance balance.

• Threshold $N_f^{short}$

  • Role: Decides single- or multi-stage processing:

    • If video length ≤ $N_f^{short}$: Single-stage
    • If video length > $N_f^{short}$: Multi-stage (coarse-to-fine)

  • Evaluation on TaCoS:

    $N_f^{short}$	mIoU (64-128s)	mIoU (128-256s)	Avg. mIoU
    64	58.43 (multi)	54.57 (multi)	50.28
    128 (default)	56.59 (single)	54.57 (multi)	50.00
    256	56.59 (single)	42.90 (single)	46.82

  • Key Findings:

    • For 64-128s videos: Minimal performance difference between multi-stage ($N_f^{short}$=64) and single-stage ($N_f^{short}$>=128) processing (ΔmIoU=1.84). However, the lower threshold increases average inference time as it forces more videos into multi-stage processing.
    • For 128-256s videos: Multi-stage processing yields significantly better results (54.57 vs 42.90 mIoU)
    • Default Choice: 128 to balance performance and efficiency.

---

We will incorporate the above clarifications, experiments, and analyses into our final draft.

Thanks for your positive comments! We provide the feedback as follows.

---

Q1: Pre-training Datasets of the Related Works

Thank you for your valuable suggestion. Here, we summarize the pre-training datasets used by the related works. Some of the methods (like TimeMarker) are trained on a dozen datasets. Due to space constraints, we will provide the complete list in the final draft.

---

Q2: Timestamp Token Insertion vs. Dense Position Encodings

Thank you for your insightful question. To address this, we conducted ablation experiments on Charades:

• Our base model, Qwen2-VL-7B, utilizes Multimodal Rotary Position Embedding (MRoPE), which decomposes position embeddings into temporal, height, and width components. After fine-tuning, temporal grounding remains suboptimal with dense position encodings.
• In contrast, timestamp token insertion yields significant gains. We attribute this improvement to the MLLM's strong multimodal retrieval ability and the simplification of the task into (1) identifying question-relevant video segments and (2) directly reading their corresponding timestamp tokens.

---

Q3: The “end” Timestamps of the Segments

In the segment retrieval (coarse-grained grounding) stage, the model is provided with the starting timestamps of each segment, denoted as {$t_1$, $t_2$, ..., $t_N$}. The model predicts a segment boundary $t_i$, and the corresponding segment interval is defined as [$t_i$, $t_{i+1}$]. For the final segment, the end timestamp t$_{i+1}$ is set to the total duration of the video.

---

Q4: The Color of Negative Numbers

Thank you for bringing this to our attention. We acknowledge the need for improved visual distinction and will use a distinct color for negative numbers in the revised manuscript.

---

Q5: Inclusion of Appendices in the Main Submission

We appreciate your question regarding appendix placement. In accordance with this year's NeurIPS submission guidelines, which state that “authors may optionally choose to include some or all of the technical appendices in the same PDF above.”, we have incorporated the most substantive appendices into our main submission.

---

We will incorporate the above clarifications, experiments, and analyses into our final draft.