TempSamp-R1: Effective Temporal Sampling with Reinforcement Fine-Tuning for Video LLMs

W1&Q2: One model only (Qwen2.5-VL-7B); missing SFT and GRPO baselines in Table 1.

Thank you for the valuable question. We have added comparisons on both Qwen2.5-VL-7B and Qwen2.5-VL-3B, including full baselines (SFT and GRPO). As shown in the updated table, TempSamp-R1 consistently outperforms GRPO and SFT under both model sizes, highlighting the robustness of our method beyond a single backbone.

Notably, GRPO struggles to provide stable gains on the weaker Qwen2.5-VL-3B model, particularly on QVHighlights, where its performance is comparable to SFT. We attribute this to the high variance and sparse reward signals in GRPO, which are more difficult to optimize for smaller models. TempSamp-R1 effectively mitigates this limitation through off-policy supervision and stabilized reward shaping.

We will include these experimental results and analyses in the final version.

Table: Results on Charades-STA Dataset

Table: Results on ActivityNet Captions Dataset

Table: Results on QVHighlights Dataset

W2&Q3: Clarifying the main contributions.

Thank you for the clarification request. Prior work, including GRPO for VTG, IoU reward, format reward, and think prompting, serves as the foundation of our baseline. Building upon these components, our method addresses a central limitation in video temporal grounding: unstable policy optimization under sparse and temporally misaligned reward supervision.

To address these limitations, our contribution lies in designing a more stable and generalizable reinforcement fine-tuning framework through two novel components:

(1) Off-policy guidance, which for the first time incorporates high-quality, instruction-aligned supervision from ground-truth annotations directly into policy updates. Unlike prior GRPO-based methods that treat ground-truth solely as a reference for reward evaluation of on-policy samples, our approach leverages these annotations as active supervision signals, providing temporally precise guidance that improves stability and convergence during training.

(2) Non-linear reward transformation, which dynamically reshapes sparse and skewed reward distributions to enable stable policy optimization. In contrast to GRPO, which treats reward values uniformly and is prone to instability under outlier-biased distributions, our transformation preserves informative gradients from partially correct predictions while mitigating the influence of extreme values.

(3) As shown in Table 4, our ablation and comparative studies demonstrate the complementary effects of both components. In addition, we extend the evaluation in W1 across multiple datasets (Charades-STA, ActivityNet Captions, QVHighlights) and model scales (Qwen2.5-VL-3B and 7B), confirming the effectiveness and generalizability of our method.

We will clarify this distinction more explicitly in the final version.

W3&Q1: Missing values and incomplete comparisons; Qwen2.5-VL-7B setup in Table 1.

We appreciate the reviewer’s observation. As shown in the table in W1, we have added several missing metrics, including GRPO, SFT and baseline results, to support a more complete and fair comparison. Additionally, QVHighlights is a key benchmark for temporal grounding, yet was omitted in prior RL-based methods. To address this gap, we have included GRPO baselines on this dataset and extended the evaluation to multiple model sizes, demonstrating the effectiveness and generalizability of TempSamp-R1.

The reported results for Qwen2.5-VL are from zero-shot evaluation without any fine-tuning. For fair comparison with TimeZero and VideoChat-R1, we standardized the total number of input pixels per video to approximately 2.8 million, which may lead to slight discrepancies from the original paper. We will clarify this setup in the final version.

Q4: Clarification on training data for the first stage.

Thank you for the question. The first-stage fine-tuning uses the same in-domain datasets as the second stage, matching the corresponding evaluation benchmarks (e.g., Charades-STA and QVHighlights). We will clarify this detail in the final version.

Q5: TimeZero discrepancy.

We thank the reviewer for noting this discrepancy. We had already noted the May 26th update and will revise the TimeZero results accordingly, clarifying the cited version in the final revision.

Formatting concerns.

Thank you for pointing this out. We will fix the placeholders and move all table captions above the tables.

Please feel free to leave additional comments if you have further questions or suggestions.

A2: Planned revisions for the final version.

We thank the reviewer for the constructive feedback and for highlighting the need to make our contributions and distinctions from prior GRPO-based methods more explicit. In the final version, we will incorporate the following revisions to ensure clarity and completeness:

1. Introduction: We will insert a concise paragraph explicitly articulating our core contribution: “Building upon GRPO, the core contribution of TempSamp-R1 lies in integrating off-policy ground-truth annotations as active supervision within policy optimization, and employing a non-linear reward transformation to stabilize learning under sparse and temporally misaligned rewards.”

2. Method: In Sec. 3.2 (Mixed-policy sampling), we will add a clarifying sentence before the current description of advantage computation: “In contrast to prior GRPO-based methods, which utilize ground-truth only for reward evaluation of on-policy samples, TempSamp-R1 incorporates these annotations directly into advantage estimation to actively guide policy updates.”

3. Method: In Sec. 3.2 (Non-linear Reward Transformation), we will add a clarifying sentence at L175: “The proposed non-linear transformation preserves informative gradients from partially correct predictions and mitigates the dominance of outlier rewards, thereby enhancing training stability under sparse and skewed reward distributions.”

4. Experiments: We will supplement Table 1 with complete baselines, including SFT, GRPO, and our TempSamp-R1 across different model scales (Qwen2.5-VL-3B and 7B). We will also add the following analysis: “The results demonstrate that TempSamp-R1 consistently outperforms SFT and GRPO across model scales, highlighting the robustness and generalizability of our framework.”

We welcome any further comments or suggestions that could help improve the clarity and completeness of the final version.

Thank you for your response. We appreciate the reviewer’s careful evaluation and constructive feedback.

A1: Regarding Qwen2.5-VL-7B zero-shot evaluation

Thank you for raising this important point. After rechecking our setup, we confirm that the discrepancy with the original paper arises from two distinct evaluation settings. The table below reports the scores under both settings, illustrating that the difference is primarily due to the total_pixels and prompt ordering.

• TimeZero / VideoChat-R1–aligned setting: For fair and computationally feasible comparison with TimeZero and VideoChat-R1, we standardized the total_pixels input to ≈2.8M per video and adopted the “Query+Video” prompt order. In practice, this corresponds to ≈86 frames at 168×336 resolution with FPS=2 (total visual input ≈2 × 2.8M pixels due to the initial 2×3×3 convolution used for visual embedding in Qwen2.5-VL). This lower input resolution and prompt design reduce the zero-shot score.

• lmms-eval–aligned setting (original model reference): The original Qwen2.5-VL-7B evaluation in lmms-eval uses ≈19M total_pixels per video at FPS=2 (≈86 frames at 476×868 resolution), and the “Video+Query” prompt order. For example, a typical prompt is structured as: <|vision_start|><|video_pad|><|vision_end|>To accurately pinpoint the event "a person is standing in their entryway undressing." in the video, determine the precise time period of the event. Provide the start and end times (in seconds, precise to two decimal places) in the format "start time to end time" within the <answer> </answer> tags. For example: "12.54 to 17.83".

Table: Results on Charades-STA Dataset

We also re-evaluated ActivityNet Captions and QVHighlights under the lmms-eval–aligned setting. We observed that the performance on QVHighlights is lower than under the TimeZero–aligned setting.

Table: Results on ActivityNet Captions Dataset

Table: Results on QVHighlights Dataset

Following the lmms-eval–aligned setting, we re-trained and evaluated both GRPO and our TempSamp-R1 method using a 2.8M total_pixel input per video and the “Video+Query” prompt format. As shown in the table, both prompt configurations (Query+Video vs. Video+Query) yield comparable performance, and TempSamp-R1 consistently outperforms GRPO under either setting. These results further confirm the effectiveness and robustness of our method across different prompt designs.

Table: Results on Charades-STA Dataset

In summary, we will make the following updates in the final version for clarity:

• Update Table 1 to report the Qwen2.5-VL results under their corresponding evaluation settings, with explicit annotation of the evaluation protocol and prompt order for each dataset. Charades-STA and ActivityNet Captions are evaluated under the lmms-eval–aligned setting using the Video+Query prompt. QVHighlights is evaluated under the TimeZero–aligned setting using the Query+Video prompt.
• Explicitly indicate that the results for TimeZero, VideoChat-R1, and TempSamp-R1 follow the TimeZero–aligned setting to ensure fair comparison.
• Provide additional results for GRPO and TempSamp-R1 trained and evaluated under the lmms-eval–aligned setting, so that the performance under the original evaluation protocol is also fully reported.

Q1: How is ground-truth used for off-policy supervision?

Thank you for the clarification request. The model is not forced to output the ground-truth prediction. Instead, ground-truth annotations are incorporated into the advantage estimation as external supervision signals to stabilize learning. The policy itself is still optimized based on gradients computed from solutions and their corresponding advantage values. We will clarify this distinction more explicitly in the final version.

Q2: Justification for the functional form of Eq. 4.

Thank you for the insightful question. In video temporal grounding, rewards are often sparse and temporally misaligned, making it difficult for partially correct predictions to receive useful feedback, especially when incorporating off-policy guidance that may diverge from the policy distribution. To address this, Eq. 4 is motivated by self-adaptive reward shaping techniques that have been widely applied to stabilize learning. We adopt an asymmetric transformation that compresses the advantages of near-optimal outputs while amplifying differences among suboptimal ones. This encourages the model to focus on high-IoU outputs while still benefiting from partially correct predictions, thereby improving the quality of gradient feedback. We will clarify the design rationale and task-specific motivations more explicitly in the final version.

Q3: How can the model learn reliable CoT outputs?

Thank you for the thoughtful question. Rather than explicitly enforcing CoT generation, our framework treats CoT as an adaptive strategy that the model may utilize when beneficial for task performance. First, the base Qwen2.5-VL model already demonstrates CoT-style reasoning, and this ability is preserved after the first training stage. During the second stage, the model learns through reward-guided exploration that producing intermediate reasoning steps can improve grounding rewards. Representative examples of such outputs can be found in Figures 5, 6, and 7. We will clarify this design intent more explicitly in the final version.

Q4: How is the “better output” selected in Mixed CoT mode?

In Table 1, TempSamp-R1 Mixed CoT simply selects the output (CoT or non-CoT) with the higher grounding score to illustrate their complementarity. We also experimented with simple heuristics such as using video length for selection, but found no consistent advantage over single-mode inference. We will clarify the selection strategy more explicitly in the final version.

Q5: Why is TempSamp-R1 more time-efficient than GRPO?

TempSamp-R1 achieves higher time efficiency primarily through two factors. First, our off-policy guidance allows the direct incorporation of ground-truth solutions into advantage estimation, which reduces the number of on-policy samples required per query from $G$ (in GRPO) to $G{-}1$, thereby lowering the computational cost of sampling. Second, we observe that our method produces shorter average completions during training (78 vs. 118 tokens), leading to reduced sequence generation overhead. We will clarify both aspects of this efficiency gain in the final version.

Q6: How is the skewness score computed?

We compute skewness using the standard third standardized moment:

$S = \frac{1}{G} \sum_{i=1}^G \left( \frac{r_i - \mu}{\sigma} \right)^3$,

where $r_i$ is the reward of the $i$-th sample, $\mu$ and $\sigma$ are the mean and standard deviation of the reward distribution, and $G$ is the total number of solutions. This computation is implemented using scipy.stats.skew, and it quantifies the asymmetry and sharpness of the reward distribution during training. We will clarify this in the final version.

Q7: The necessity of CoT

Thank you for the valuable question. In our experiments conducted on a single NVIDIA A100 GPU, the inference time for CoT outputs remains comparable to that of non-CoT outputs (2.12s vs. 2.10s per sample), due to the relatively short length of generated reasoning steps. Furthermore, our framework is designed to accommodate both CoT and non-CoT reasoning within a unified model. As shown in Table 1, we find that CoT can improve grounding quality with minimal computational overhead in some settings (e.g., Charades-STA), making it a viable option when quality gain justifies the cost. However, in tasks where CoT offers limited performance benefits (e.g., QVHighlights), it can be omitted to reduce latency. We will clarify this point in the final version.

Please feel free to leave additional comments if you have further questions or suggestions.

Q1&W1: Clarifying the contribution of CoT vs. reward design.

Thank you for the insightful question. We conducted additional comparative experiments between GRPO and TempSamp-R1 under both CoT and non-CoT prompting strategies. The results confirm that TempSamp-R1 consistently outperforms GRPO across all settings, with or without CoT. Notably, the performance gains observed in the CoT setting (+1.5 R1@0.5 and +4.8 R1@0.7) are comparable to or even larger than those in the non-CoT setting (+1.4 R1@0.5 and +3.7 R1@0.7). This suggests that CoT prompting benefits more from stabilized training, allowing the model to more effectively exploit intermediate reasoning steps. Additionally, these results indicate that CoT prompting and the stabilized reward optimization, achieved through the combination of off-policy guidance and non-linear reward transformation, contribute complementary benefits. We will include these comparative results and further discussion in the final revision.

Q2&W2: Heuristic parameter selection and sensitivity.

Thank you for the thoughtful question. In video temporal grounding, rewards are often sparse and temporally misaligned, making it difficult for partially correct predictions to receive useful feedback, especially when incorporating off-policy guidance that may diverge from the policy distribution. To address this, Eq. 4 is motivated by self-adaptive reward shaping techniques that have been widely applied to stabilize learning. We adopt an asymmetric formulation to better distinguish partially correct outputs from suboptimal ones.

We have added corresponding ablation experiments. Results show that the model is relatively insensitive to $\alpha_1$ and $\alpha_2$ within reasonable ranges (e.g., $\alpha_1 \in [0.01, 0.02]$, $\alpha_2 \in [0.5, 1]$). Regarding $\tau$, values near 0.8 generally led to improved training stability and performance in our experiments. Higher values (e.g., 0.9) tended to exacerbate reward imbalance by concentrating reward mass on very few samples, which limits the model's ability to benefit from informative but suboptimal solutions.

We will include these ablation results and analyses in the final revision.

Q3: Trade-off between exploration and stable convergence.

Thank you for the valuable question. As figures cannot be included in the rebuttal, we instead report policy entropy values at different training steps. Compared to GRPO, TempSamp-R1 demonstrates higher policy entropy on ActivityNet, suggesting stronger exploratory behavior during training. This effect arises in part from the use of ground-truth annotations as off-policy guidance, since these solutions typically fall outside the model's high-probability prediction region. Integrating such informative but underexplored outputs into the learning process enables the model to expand its exploration space without compromising convergence stability. We will include the relevant results and analysis in the final revision.

Table: Entropy per Step for Policy Models (GRPO & TempSamp-R1)

Please feel free to leave additional comments if you have further questions or suggestions.

W1: Lack of ablations for advantage anchoring and non-linear reward shaping.

Thank you for the valuable suggestion. We have added ablations for $\lambda_{\text{off}}$ (used in advantage anchoring), as well as $\tau$, $\alpha_1$, and $\alpha_2$ (hyperparameters within non-linear reward shaping).

• We observe that a low $\lambda_{\text{off}}$ (e.g., 1.0) weakens off-policy supervision, while overly large values (e.g., 1.4) introduce distributional shifts between on- and off-policy solutions, leading to biased advantage estimation and degraded policy learning stability.
• For non-linear reward shaping, the model shows robustness to $\alpha_1$ and $\alpha_2$ within reasonable ranges (e.g., $\alpha_1 \in [0.01, 0.02]$, $\alpha_2 \in [0.5, 1]$). Regarding $\tau$, values near 0.8 generally led to improved training stability and performance in our experiments. Higher values (e.g., 0.9) skew the reward distribution toward a few high-reward solutions, limiting the model’s ability to learn from suboptimal solutions.

We will include these ablation results and analyses in the final revision.

W2: Significant practical value, with concerns regarding limited novelty.

We appreciate the reviewer’s recognition of the practical value of our work. The effectiveness of our framework stems from addressing a key limitation in prior RL-based approaches (e.g.,TimeZero and VideoChat-R1) for video temporal grounding, specifically unstable policy learning under sparse and temporally misaligned reward supervision.

To address these limitations, we propose two novel mechanisms:

(1) Off-policy guidance. Our method introduces a novel use of ground-truth annotations by directly integrating them into the policy update process, rather than merely using them for reward evaluation of on-policy samples as in GRPO-based approaches. These instruction-aligned annotations serve as active supervision signals that enhance training stability by providing temporally accurate and semantically grounded feedback.

(2) Non-linear reward transformation. To address the distributional bias introduced by incorporating off-policy rewards, we adopt a soft advantage estimation mechanism inspired by adaptive reward shaping. Rather than treating all rewards uniformly, our method applies an asymmetric transformation that compresses the advantage values of near-optimal solutions while amplifying distinctions among suboptimal ones. This strategy produces more informative gradients and mitigates the suppression of promising on-policy samples, thereby improving training stability and encouraging effective policy exploration.

These components are seamlessly integrated within a CoT-compatible framework, resulting in consistent improvements over GRPO baselines across multiple dataset. We will make this contribution more explicit and emphasize its significance in the final version.

Q1: Generalizability and robustness across tasks.

To assess generalizability beyond temporal grounding, we conducted additional experiments on the NExT-GQA benchmark following the training and evaluation setting introduced by VideoChat-R1. This task involves spatio-temporal grounding for video-based question answering, where the model must identify both the correct answer (acc) and its supporting temporal segment (mIoU).

We observe that both advantage anchoring and non-linear reward shaping consistently outperform GRPO across multiple metrics. Notably, these results are achieved using the same hyperparameter configurations as in our temporal grounding experiments, without any task-specific tuning. This demonstrates that our proposed methods are robust and transferable to structurally different tasks that require temporal grounding with multimodal reasoning.

Please feel free to leave additional comments if you have further questions or suggestions.