DeepVideo-R1: Video Reinforcement Fine-Tuning via Difficulty-aware Regressive GRPO

We sincerely thank the reviewer for the positive and constructive comments. We are encouraged that the reviewer recognized multiple strengths in our paper, including:

(i) clear writing

(ii) The novel Reg-GRPO approach

(iii) Effective difficulty-aware data augmentation

(iv) Strong empirical results, with DeepVideo-R1 outperforming prior state-of-the-art models on complex video reasoning benchmarks.

We have tried our best to address the reviewers’ questions and concerns within the available time. We believe that incorporating this constructive feedback significantly enhances the quality of the paper. Below, we provide the response to the given comments:

---

Q1. Performance on other video baselines. [Weakness 1]

A1. We have already demonstrated the effectiveness of our DeepVideo-R1 compared to other GRPO-based VideoLLM such as TimeZero and VideoChat-R1 on both Charades-STA and ActivityNet-Captions in Table 10 of the supplement. The table shows that our DeepVideo-R1 achieves the best performance compared to these baselines across both benchmarks. These results highlight the effectiveness of our proposed Reg-GRPO and difficulty-aware data augmentation approaches.

Furthermore, we measured the performance of our DeepVideo-R1 on diverse video understanding reasoning benchmarks such as VideoMMMU, MMVU (mc), MVBench, TempCompass, and Video-MME as reported in Table 9 of the supplement. As suggested, we compare our DeepVideo-R1 with Video-R1 on these benchmarks. We use training data of Video-R1 and use Qwen2.5-VL-3B as a base model. The experimental results in the table below indicate that our DeepVideo-R1 shows better performance on 4 out of 5 benchmark datasets compared to Video-R1, validating the generalizability and robustness of DeepVideo-R1 across diverse video-language reasoning tasks.

---

Q2. Computational costs of difficulty-aware data augmentation. [Weakness 2, Question 2]

A2. Good question. As suggested, we compare the computational cost of GRPO, DeepVideo-R1 without difficulty-aware data augmentation, and DeepVideo-R1 by measuring the average training and inference time per video across three different benchmarks (SBR-L1, SBR-L2, SBR-L3) in the table below. We use Qwen2.5-VL-Instruct 3B as a base multimodal large language model and NVIDIA A100 GPUs, following the experimental setting in our supplement.

The table shows that the inference time remains constant across methods since both Reg-GRPO and D.A. Aug. are training approaches. Notably, DeepVideo-R1 significantly improves the performance of GRPO without increasing inference cost. Even though the difficulty-aware augmentation slightly increases training time, it has a superior performance gain of 6.92 on SBR-L2 compared to the model without difficulty-aware data augmentation, demonstrating the necessity of the augmentation.

---

Q3. Generalizability of the difficulty-aware data augmentation. [Weakness 3]

A3. To evaluate the generalizability of difficulty-aware augmentation on different types of video reasoning tasks, we additionally conduct the experiments by comparing the performance of DeepVideo-R1 with and without difficulty-aware augmentation on five video reasoning benchmarks. The experimental results are described in the table below. The table demonstrates that our difficulty-aware data augmentation strategy is broadly applicable and effective beyond SEED-BENCH-R1 datasets.

---

Q4. Training stability of the Reg-GRPO. [Question 1]

A4. To assess the training stability of Reg-GRPO, we analyze the divergence between $\pi_\theta\left(y|x\right)$ and $\pi_{\theta_{\text{old}}}\left(y|x \right)$. Specifically, we report the average absolute log-ratio $|\log \frac{\pi_\theta \left(y|x \right)}{\pi_{\theta_{\text{old}}}\left(y|x \right)}|$ and the ratio of tokens whose likelihood ratios $\frac{\pi_\theta(y_t | x, y_{<t})}{\pi_{\theta_{\text{old}}}(y_t | x, y_{<t})}$deviate significantly from 1 (e.g., less than 0.5 or greater than 1.5), which indicates instability in updates. The experimental results in the table below show that Reg-GRPO consistently yields smaller deviations than GRPO across all metrics. These findings suggest that Reg-GRPO leads to more stable policy updates, which is crucial for ensuring reliable training. We will include additional plots illustrating these stability metrics over training iterations in the final version to further support this analysis.

• Avg. token ratio of $\\lvert 1-\frac{\pi_\theta(y_t | x, y_{<t})}{\pi_{\theta_{\text{old}}}(y_t | x, y_{<t})}\\rvert>0.3$

  • GRPO: 0.000972
  • Reg-GRPO: 0.000435

• Avg. token ratio of $\\lvert 1-\frac{\pi_\theta(y_t | x, y_{<t})}{\pi_{\theta_{\text{old}}}(y_t | x, y_{<t})}\\rvert>0.5$

  • GRPO: 0.000231
  • Reg-GRPO: 0.000073

• Avg. token ratio of $\\lvert 1-\frac{\pi_\theta(y_t | x, y_{<t})}{\pi_{\theta_{\text{old}}}(y_t | x, y_{<t})}\\rvert>1.0$

  • GRPO: 0.000036
  • Reg-GRPO: 0.000009

---

Q5. Robustness of the Reg-GRPO on sparse or noisy reward signals. [Question 3]

A5. Thank you for an insightful and important question. We have already shown the robustness of the Reg-GRPO under sparse reward signals where a reward signal is assigned only when the generated output is correct. This setup reflects common conditions in video QA tasks with binary correctness-based supervision. Table 1 and 4 of the main paper present results under this sparse reward setting. Notably, DeepVideo-R1 equipped with Reg-GRPO significantly outperforms GRPO, highlighting its resilience to sparse reward signals.

To further evaluate the robustness of our proposed Reg-GRPO under noisy reward signals, we conduct additional experiments with a newly introduced noise-injected setting. To the best of our knowledge, there has not been an experimental setting using noisy reward signals. So, we newly introduce a noisy-injected reward experimental setup, in which the reward signal is flipped with 30% probability: correct outputs are incorrectly assigned a reward of 0, and incorrect outputs receive a reward of 1. Under this setting, we compare GRPO and Reg-GRPO on SEED-Bench-R1 in the table below. To ensure a fair comparison that isolates the effect of the reinforcement learning algorithm, we exclude difficulty-aware data augmentation from all models in this experiment.

The experimental results demonstrate that Reg-GRPO exhibits significantly lower performance degradation than GRPO under noisy rewards, maintaining relatively stable performance across all datasets. In particular, the performance degradation of Reg-GRPO is only 1.75 on SBR-L3 dataset while GRPO suffers a substantial performance drop of 9.38. This indicates that the regression-based formulation of Reg-GRPO is inherently more robust to noise in reward signals compared to GRPO. We will include this analysis and new experimental setting in the final version of the paper if the paper gets accepted.

We sincerely thank the reviewer for the valuable and constructive review. We are encouraged that the reviewer highlights multiple strengths of our work, including:

(i) A principled reformulation of GRPO

(ii) Effective advantage alignment via Reg-GRPO

(iii) Difficulty-aware data augmentation that addresses the key limitation of GRPO

(iv) Comprehensive empirical validation

We believe that incorporating this constructive feedback significantly enhances the quality of the paper. Below, we respond to the reviewer’s insightful questions below:

---

Q1. Typos and clarity of notations [Question 1,2(b)].

A1. We appreciate the feedback. We identified the mentioned typographical issues and some notations with a lack of explanation. We will reflect them in the final version if the paper gets accepted.

---

Q2. More detailed derivation of reg-grpo. [Question 2(a)]

A2. As requested, we specifically clarify why $\epsilon$ is needed in Eq. (8) of the supplement as below.

Our optimization objective is :

$$\max_{\pi} \ \mathbb{E}_{ x, y\sim\pi\_\theta (\cdot | x )} \mathcal{R} ( x, y ) - \epsilon \cdot \mathbb{D}\_{\text{KL}} (\pi\_\theta || \pi\_{\theta\_{\text{old}}} ),$$

where $\epsilon$ is a regularization strength.

From Section A.1 of the supplement in the DPO paper [1] and our supplement, we can obtain that the optimal policy for our objective is

${\pi}\_{\theta}^* ( y | x ) = \frac{1}{Z ( x )} \pi\_{\theta_{\text{old}}} ( y | x )\exp (\frac{1}{\epsilon} \mathcal{R} ( x, y ) ),\ Z(x) = \sum_y \pi\_{\theta\_{\text{old}}} (y|x)\exp (\frac{1}{\epsilon} \mathcal{R}(x,y) )$

where $\mathcal{R} (x,y )$ is the reward value given $x$ and $y$.

From this equation, the reward $\mathcal{R} ( {x}, {y} )$ can be formulated under $\pi_{\theta_{\text{old}}} ( {y} | {x} )$ and $\pi_\theta ( {y} | {x} )$.

By multiplying both sides of Eq. (5) by $Z ( {x} )$, we can get

$Z ( x ) \cdot {\pi}\_{\theta}^* ( y| x ) = \pi\_{\theta\_{\text{old}}} ( y | x )\exp (\frac{1}{\epsilon} \mathcal{R} ( x, y ) ),$

$Z ( x )\cdot \frac {{\pi}\_{\theta}^* ( {y} | {x} )}{\pi\_{\theta\_{\text{old}}} ( {y} | {x} )} = \exp (\frac{1}{\epsilon} \mathcal{R} ( x, y ) ),$

Next, we take the logarithm of both sides and then multiply both sides by $\epsilon$ to obtain the following equation (Eq. (6) of the supplement):

$\mathcal{R} ( {x}, {y} ) = \epsilon \cdot (\log Z ( {x} ) + \log (\frac{\pi_{\theta}^* ( {y} | {x} )}{\pi_{\theta_{\text{old}}} ( {y} | {x} )} ) ),$

The advantage of GRPO is defined as

$A^{(i)} = \frac{\mathcal{R} ( {x}, {y}^{(i)} )-\mu_r}{\sigma_r}$. (Eq. (7) of the supplement), where $\mu_r, \sigma_r$ are mean and standard deviation of a set of reward values $\\{\mathcal{R} ( {x}, {y}^{(i)} ) \\}_{i=1}^G$.

Using Eq.(6) of the supplement, we can reformulate the advantage by substituting $\mathcal{R} ( {x}, {y} )$ with $\epsilon \cdot (\log Z ( {x} ) + \log (\frac{\pi_{\theta}^* ( {y} | {x} )}{\pi_{\theta_{\text{old}}} ( {y} | {x} )} ) )$. It can be written as

$$\hat{A}^{(i)} = \frac{\epsilon \cdot \log Z ( {x} ) + \epsilon \cdot\log (\frac{\pi_{\theta}^* ( {y}^{(i)} | {x} )}{\pi_{\theta_{\text{old}}} ( {y}^{(i)} | {x} )} ) - \epsilon \cdot \log Z ( {x} ) - \epsilon \cdot \frac{1}{G} \sum_{i=1}^G\log (\frac{\pi_{\theta}^* ( {y}^{(i)} | {x} )}{\pi_{\theta_{\text{old}}} ( {y}^{(i)} | {x} )} )}{\sigma_\rho}$$ $$\hat{A}^{(i)} = \frac{ \epsilon \cdot \log (\frac{\pi\_{\theta}^* ( {y}^{(i)} | {x} )}{\pi\_{\theta_{\text{old}}} ( {y}^{(i)} | {x} )} ) - \epsilon \cdot \frac{1}{G} \sum\_{i=1}^G\log (\frac{\pi\_{\theta}^* ( {y}^{(i)} | {x} )}{\pi\_{\theta\_{\text{old}}} ( {y}^{(i)} | {x} )} )}{\sigma\_\rho}$$ $$= \epsilon \cdot \frac{ \log \rho^* ( x, y^{(i)} ) - \mu\_\rho^\*}{\sigma\_\rho^\*},\quad \rho^* ( {x}, {y} ) = \frac{\pi\_{\theta}^* ( {y} | {x} )}{\pi\_{\theta\_{\text{old}}} ( {y} | {x} )}, \quad \mu\_\rho^* = \frac{1}{G}\sum\_{i=1}^G\log (\frac{\pi\_{\theta}^* ( {y}^{(i)} | {x} )}{\pi\_{\theta\_{\text{old}}} ( {y}^{(i)} | {x} )} ), \quad \sigma\_\rho^* = \text{Std} (\\{\log \rho^* ( {x}, {y}^{(i)} ) \\}\_{i=1}^G )$$

Thus, we can obtain Eq. (8) and $\epsilon$ is required. It arises naturally from the KL-regularized RL objective and controls the strength of the regularization and scale of the reward. We have updated the supplement to make this derivation more explicit and easier to follow.

---

Q3. Is Eq (5) in the supplement a necessary condition or an equivalent condition for $\pi^*$ to be an optimal policy? [Questions 2(c)]

A3.  $\pi^*$ is an optimal policy and the condition is an equivalent (necessary and sufficient) condition as proven from Section A in the supplement of DPO paper [1].

Based on the result, our derivation is provided in Section A.1 of our supplement.

We provide a sketch of proof here for the convenience.

The policy optimization objective is given as:

$$\pi_\theta^* = \text{arg}\max\_\pi \ \mathbb{E}\_{ {x}, {y}\sim\pi (\cdot | {x} )} \mathcal{R} ( {x}, {y} ) - \epsilon \cdot \mathbb{D}\_{\text{KL}} (\pi\_\theta || \pi\_{\theta\_{\text{old}}} )$$

From our supplement and other works, the above problem can be reformulated as

$$\arg \min_\pi\ \mathbb{E}\_{ {x}}\left[\mathbb{D}\_{\text{KL}} (\pi_\theta ( {y} | {x} )||\bar{\pi}_\theta ( {y}| {x} ) )-\log Z ( {x} ) \right],$$

where $\bar{\pi}\_{\theta} ( {y} | {x} ) = \frac{1}{Z ( {x} )} \pi\_{\theta\_{\text{old}}} ( {y} | {x} )\exp (\frac{1}{\epsilon} \mathcal{R} ( {x}, {y} ) )$. Since the partition function does not depend on $\pi$, the optimal policy minimizes the KL regularization.

The minimum KL is achieved if and only if two distributions are identical, the optimal solution is Eq. (5) as follows

$${\pi}\_{\theta}^* ( {y} | {x} ) = \frac{1}{Z ( {x} )} \pi_{\theta_{\text{old}}} ( {y} | {x} )\exp (\frac{1}{\epsilon} \mathcal{R} ( {x}, {y} ) ), \forall {x}, {y}$$

Thus, Eq (5) is a necessary and sufficient (equivalent) condition for $\pi$ to be an optimal policy under our objective. So, the update direction of Eq. (9) in our supplement is correct.

---

Q4. What connection exists between Reg-GRPO and video tasks? [Question 3]

A4. Good question. Reg-GRPO (Regressive Group Relative Policy Optimization) is a generic technique designed to enhance the reasoning capability of LLM-based models, particularly for complex tasks. It is not limited to video LLMs. To show the generalizability of Reg-GRPO, we evaluate it by comparing it with GRPO on multiple complex “mathematical reasoning” benchmarks in the table below.

The performance gain across these diverse reasoning datasets shows that Reg-GRPO is an effective post-training strategy for enhancing LLM reasoning. This generalizability extends naturally to challenging tasks such as video question answering, where complex temporal and causal reasoning is often required. Thus, Reg-GRPO strengthens the core reasoning ability of LLMs, which is essential for solving video tasks that demand high-level understanding and long-term dependencies.

---

Q5. Additional performance comparisons on video benchmarks. [Question 4]

A5. To further validate the generalizability of our method, we compare DeepVideo-R1 against Video-R1 using identical pretraining data and Qwen2.5-VL-3B as the base model. Results on five video-language reasoning datasets are shown below:

DeepVideo-R1 outperforms Video-R1 on 4 out of 5 datasets, highlighting the robustness and generalizability of our approach across diverse video-language reasoning tasks.

---

Q6. Are the SEED-Bench-R1 and NextGQA datasets combined or used separately? [Question 5]

A6. As noted in the supplement, SEED-Bench-R1 and NextGQA datasets are used separately for reinforcement fine-tuning. In response to the suggestion, we conducted an additional experiment using joint fine-tuning on the combined dataset (SEED-Bench-R1 + NextGQA). The results are summarized below:

The results indicate that joint fine-tuning on both datasets slightly improves performance on SEED-Bench-R1 while maintaining comparable performance on NextGQA.

---

Q7. Comparison of Reg-GRPO with GRPO regarding the gap between $\pi_\theta$ and $\pi_\text{old}$. [Question 6]

A7. Good point. As suggested, we analyze the divergence between $\pi_\theta (y|x )$ and $\pi_{\theta_{\text{old}}} (y|x )$. Specifically, we report the average absolute log-ratio $|\log \frac{\pi_\theta (y|x )}{\pi_{\theta_{\text{old}}} (y|x )}|$, which indicates instability in updates. The experimental results in the table below show that Reg-GRPO consistently yields smaller deviations than GRPO. It suggests that Reg-GRPO leads to more stable policy updates. We will include additional plots illustrating these stability metrics over training iterations in the final version to further support this analysis.

[1] Rafailov, Rafael, et al. "Direct preference optimization: Your language model is secretly a reward model." NeurIPS 2023.

We sincerely thank the reviewer for their continued engagement and are pleased that most concerns have been addressed. Below are our responses to the follow-up questions:

---

FQ1. Why do you replace $\sigma_r$ with $\sigma_\rho$ in Eq(8)?

We appreciate the reviewer’s feedback. This is a typo and $\epsilon$ was omitted. Sorry for the confusion. It should be $\epsilon \cdot\sigma_\rho = \sigma_r$.

In our experiments, we used the above form as L811-L816 of “grpo_trainer.py” in the supplementary materials. Furthermore, as noted in L139 of the supplement PDF, we used $\epsilon=1$ in the experiments. Thus, our experiments have no discrepancy with the equation above. We will update the manuscript in the final version.

---

FQ2. What data are used to finetune for the evaluation on the “mathematical reasoning” benchmarks? Why are the enhancements mainly reflected in the 'AMC23'?

Good question. For reinforcement fine-tuning the model, we follow the work [1] and use Math3to5, which consists of math problems at 3-5 difficulty levels of problems in MATH dataset [2]. We believe that the significant performance improvement on AMC23 arises from the strong generalization capability of our proposed Reg-GRPO method. Notably, AMC23 and OlympiadBench are out-of-distribution benchmarks that contain high-level, competition-style math problems, whereas MATH-500 and Minerva are more in-distribution, aligned with the training distribution of Math3to5.

As shown in Table A4, Reg-GRPO yields the largest gains on the more challenging, out-of-distribution datasets (AMC23 and OlympiadBench). This suggests that Reg-GRPO improves the model's robustness and ability to generalize beyond the training distribution.

[1] Liu, Ziru, et al. "GHPO: Adaptive Guidance for Stable and Efficient LLM Reinforcement Learning." arXiv:2507.10628 (2025).

[2] Hendrycks, Dan, et al. "Measuring mathematical problem solving with the math dataset." NeurIPS (2021).

We sincerely thank the reviewer for the positive and constructive review of our paper. We are encouraged that the reviewer recognizes multiple strengths in our paper, including:

(i) novel and fresh approaches

(ii) well-structured and thorough explanation

(iii) strong experimental results across multiple challenging benchmarks

We believe that incorporating this constructive feedback significantly enhances the quality of the paper. Below, we provide the response to the given comments in detail:

---

Q1. Discussion of computational complexity. [Weakness 1]

A1. We appreciate the reviewer’s suggestion to study the computational cost of our proposed method. To explore the computational cost, we report the average training and inference time per video across three different benchmarks (SBR-L1, SBR-L2, SBR-L3) in the table below. We use Qwen2.5-VL-Instruct 3B as a base multimodal large language model and NVIDIA A100 GPUs following the experimental setting in our supplement.

The table shows that the inference time remains constant across methods since our approach focuses on training. Notably, DeepVideo-R1 significantly improves the performance of GRPO without increasing inference cost. Even though the difficulty-aware augmentation slightly increases training time, it has a superior performance gain of 6.92 on SBR-L2 compared to the model without difficulty-aware data augmentation. This demonstrates the effectiveness of the augmentation by addressing the vanishing advantage problem.

---

Q2. Generalizability of difficulty-aware augmentation. [Weakness 2]

A2. To evaluate the generalizability of difficulty-aware augmentation on different types of video reasoning tasks, we additionally conduct the experiments by comparing the performance of DeepVideo-R1 with and without difficulty-aware augmentation on five video reasoning benchmarks. The experimental results are described in the table below. The table demonstrates that our difficulty-aware data augmentation strategy is broadly applicable and effective beyond SEED-BENCH-R1 datasets.

---

Q3. Ablation of Reg-GRPO with/without clipping function. [Question 1]

A3. Following the reviewer’s suggestion, we conduct additional experiments comparing DeepVideo-r1 trained with and without applying the clipping mechanism to investigate the impact of the clipping function on training stability. As NeurIPS policy does not allow figure uploads during the rebuttal phase, we provide only final performance instead of providing training curves. The experimental results are presented in the table below. The results show that the performance is comparable with and without clipping functions across all datasets, indicating that Reg-GRPO remains stable and effective without the clipping function. We will include the full training curve and analysis in our final version if the paper gets accepted.

---

Q4. Relation to reweighted imitation learning algorithm. [Question 2]

A4. Thank the reviewer for the insightful comment regarding the reweighted imitation learning and our Regressive GRPO (Reg-GRPO). As the reviewer pointed out, our Reg-GRPO is related to prior imitation learning methods [1,2,3], which are intuited by a KL-regularized optimization problem. However, while they focus on learning a reward function relying on ”expert demonstrations” to replicate expert actions, our Reg-GRPO does NOT require any expert trajectories or access to an expert policy. Instead, Reg-GRPO directly optimizes a policy by regressing onto the estimated advantages, which are derived from the model’s own rollouts.

We agree that discussing Reg-GRPO with reweighted imitation learning would improve the paper, and we will include a more comprehensive discussion and comparison in the final version. We are also happy to incorporate any additional related works the reviewer may suggest.

[1] Jacq, Alexis, et al. "Learning from a learner." ICML, 2019.

[2] Watson, Joe, Sandy Huang, and Nicolas Heess. "Coherent soft imitation learning." NeurIPS, 2023.

[3] Xu, Haoran, et al. "An Optimal Discriminator Weighted Imitation Perspective for Reinforcement Learning." ICLR, 2025.