GVPO: Group Variance Policy Optimization for Large Language Model Post-Training

We appreciate your recognition of the novelty in our loss formulation and the clarity of our analysis.

Building on this foundation, we respectfully address the following points of concern to further clarify our contributions.

1. Limited Experimental Evaluation (experiments run only on Qwen2.5-Math-7B model)

We have included experiments on an additional base model as presented below.

Furthermore, we have expanded our evaluation in the rebuttals by incorporating more benchmarks, conducting ablations on the regularization terms, performing robustness check across seeds, and evaluating an alternative task of summarization.

2. Limited theoretical novelty.

It straightforwardly follows from the definition of implicit rewards that the optimal solution to such a loss is also the optimal solution to the KL-constrained reward maximization problem.

While the necessity of the theorem ($R_\theta = R$ is an optimal solution) follows directly, establishing its sufficiency (the only optimal solution is $R_\theta = R$) is significantly more challenging and is addressed in detail in Appendix B.1.

The sufficiency result is novel—for instance, DPO establishes only necessity, not sufficiency, and admits multiple optimal solutions, some of which does not correspond to the optimal solution of the KL-constrained reward maximization objective (see Lines 231–238).

$Var(log(\pi_\theta))$ is not related to rewards at all. Why is this expression better than entropy regularization in terms of ensuring undesirable responses receive zero probability, while favorable responses retain comparable probabilities? Are there any experimental validation to this claim?

We justify this expression both theoretically and empirically.

First, we clarify that $Var(\log(\pi_\theta))$ is intended to ensure some responses receive zero probability, while other responses retain comparable probabilities. As just introduced in the paper, the term $R - \overline{R}$ promotes advantage maximization; therefore, $Var(\log(\pi_\theta))$ serves to ensure that undesirable responses receive zero probability, while favorable responses maintain similar probabilities. We will revise the manuscript to make this point clearer.

To illustrate the benefit of $Var(log(\pi_\theta))$, consider a toy example involving generation over the tokens $a, b, c, d, e$, where $\pi_\theta(a) = \pi_\theta(b) = 0$ and $\pi_\theta(c) = \pi_\theta(d) = \pi_\theta(e) = 1/3$. In this case, $Var(\log(\pi_\theta)) = 0$, indicating minimum variance and thus no penalty on this distribution. In contrast, entropy regularization only reaches its minimum either when the distribution is uniform or when it is one-hot, depending on whether one encourages higher or lower entropy.

We further validate this claim through ablation studies.

In the first set of experiments, we (a) remove the $Var(\log(\pi_\theta))$ term, and (b) replace it with entropy regularization. In both cases, the models fail to converge and produce incoherent outputs, demonstrating that $Var(\log(\pi_\theta))$ plays a crucial role in stabilizing training by discouraging extreme probability assignments.

In the second set, we lower the learning rate to $1\text{e}{-6}$ and repeat the experiments. We observe that removing the term still leads to divergence, whereas replacing it with entropy regularization shows slightly improved but still unsatisfactory performance. These results suggest that while entropy regularization serves a similar purpose, it does not fully substitute $Var(\log(\pi_\theta))$. Moreover, the poor performance may stem from entropy regularization being either too weak—failing to suppress extreme values—or too strong—hindering convergence. This highlights a key drawback of entropy regularization: it requires careful coefficient tuning. In contrast, the coefficient for $Var(\log(\pi_\theta))$ is determined analytically via the optimal solution theorem, thereby avoiding this sensitivity and improving robustness.

Were more experiments conducted comparing GRPO to GVPO apart from Qwen series of models?

Yes. We conducted additional experiments using Llama-3.1-8B-Instruct as a new base model. The results consistently show that GVPO outperforms GRPO under this new setting, demonstrating its effectiveness again.

We appreciate your recognition of GVPO’s advancements over GRPO. We also thank you for acknowledging the rigorous theoretical guarantees of our paper.

Below, we address your concerns and provide clarifications to further strengthen our contribution.

1. Separation of Reward Modeling and Policy Optimization

Does GVPO require a separate training phase for the reward model before applying the algorithm?

No, GVPO does not require a separate training phase. GVPO supports flexible sampling distributions, including those derived from human ratings or preferences, similar to DPO, and thus does not necessitate training a separate reward model (Lines 278–283).

In our paper, the reward model can take various forms: it may explicitly consist of human evaluation ratings, a trainable function that implicitly captures human preferences, or a predefined rule—such as correctness or accuracy (Lines 94–95).

How does the performance of GVPO depend on the quality of the separately trained reward model?

LLM post-training scenarios generally fall into two categories: reinforcement learning with verifiable rewards (RLVR) and reinforcement learning from human feedback (RLHF).

In RLVR settings, there is no need for a trained reward model. In RLHF, if GVPO uses human-labeled data directly—similar to DPO—it likewise avoids reliance on a reward model's quality. However, when GVPO is trained using a learned reward model as a proxy, its performance, like that of other RL algorithms, will depend on how accurately the proxy reflects true rewards. This proxy reward issue [1] is a well-known challenge in reinforcement learning. Consequently, it is common practice to evaluate post-training algorithms in the RLVR setting.

[1] Skalse, Joar, et al. "Defining and characterizing reward gaming." Advances in Neural Information Processing Systems 35 (2022): 9460–9471.

2. Redundant Presentation of Equations

We appreciate your detailed suggestion. We will revise the manuscript to first introduce $R_\theta$, followed by the MSE equations. We will then explain how these equations naturally arise from the condition $\sum w_i=0$.

3. Redundant Theoretical Proofs in Section 3.3

Thank you for this helpful suggestion. We will move less essential proofs (such as Lemmas 3.4 and 3.5) to the appendix to allocate more space for experimental results.

4. Misleading Name ("Variance")

We apologize for any confusion caused by the use of “Variance” in GVPO’s name. Our intention was to create a counterpart to GRPO’s "Relative" terminology. Specifically, while GRPO’s weight includes a term $R - \overline{R}$, GVPO’s MSE loss $[(R_\theta - \overline{R_\theta}) - (R - \overline{R})]^2$ can be rewritten in a variance-like form: $[(R_\theta - R) - \overline{(R_\theta - R)}]^2$. We will clarify this interpretation in the revised manuscript.

5. Insufficient Experimentation. How can the experimental section be expanded to provide more thorough empirical validation of the proposed method ?

We have strengthened our empirical evaluation in the rebuttals by

• adding experiments on an additional base model (in the Response to Reviewer KXfz)
• incorporating a broader range of benchmarks (in the Response to Reviewer X8wt)
• conducting ablation studies on the regularization terms (in the Response to Reviewer u3Wn)
• performing robustness check across seeds (in the Response to Reviewer u3Wn)
• including results on an alternative summarization task (in the Response to Reviewer X8wt).

Thank you for engaging with our work and highlighting the novel aspects we take pride in. We appreciate your recognition of the clarity with which we position our method in relation to existing approaches, as well as your appreciation of the three complementary interpretations of the gradient. We're also pleased that you found the experimental results compelling.

It would be nice to understand whether there is a more general principle at work here. ... In particular, the objective function studied here is of the form of mirror descent, which may provide a path towards a deeper understanding and even obtaining theoretical convergence rates.

Thank you for highlighting this promising direction for future research. Based on our current understanding, we are not aware of a unifying principle in OR or optimization literature that directly explains this method. Nonetheless, we share your interest in theoretical convergence rates and find questions—"Which sampling distributions exhibit faster convergence? What characteristics do they share?"—both insightful and significant. We will keep your suggestion regarding "mirror descent" in mind and plan to explore this perspective further in future work.

Having read the other reviews and the rebuttal I continue to view this as a valuable contribution to the literature on LLM post-traiing and I retain my favorable score.

We appreciate your recognition of the importance and clarity of our work, as well as the relevance of our positioning within the literature. We are particularly encouraged by the acknowledgment of our theoretical derivations and empirical results.

Below, we address your concerns and provide clarifications to further strengthen our contribution.

Insufficient empirical benchmarking

We have added three more recently popular benchmarks, including DAPO [1], Remax [2], and Reinforce++ [3].

The results demonstrate that GVPO continues to retain its effectiveness.

[1] Yu Q, Zhang Z, Zhu R, et al. Dapo: An open-source llm reinforcement learning system at scale[J]. arXiv preprint arXiv:2503.14476, 2025.

[2] Li, Ziniu, et al. "Remax: A simple, effective, and efficient reinforcement learning method for aligning large language models." arXiv preprint arXiv:2310.10505 (2023).

[3] Hu, Jian, Jason Klein Liu, and Wei Shen. "Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models." arXiv preprint arXiv:2501.03262 (2025).

I think the paper benefits a discussion surrounding these issues, and additional experimentation to highlight the nuanced nature of the discussion surrounding development of better policy search methods. Towards this, I'd like to see (a) additional results on natural language generation tasks such as summarization, question answering etc., (b) whether a better policy search method naturally translates to better evals on automated/human eval on alignment quality.

Thank you for your valuable suggestions, which help broaden the impact of this work within the alignment research community.

From a theoretical perspective, when an RL algorithm is trained using a proxy reward model instead of the true reward (e.g., actual human preferences), its performance on the true objective depends critically on the fidelity of the proxy [4]. This challenge of proxy rewards is well recognized in reinforcement learning literature [5].

[4] Wang, Binghai, et al. WorldPM: Scaling Human Preference Modeling. arXiv preprint arXiv:2505.10527, 2025.

[5] Skalse, Joar, et al. Defining and characterizing reward gaming. Advances in Neural Information Processing Systems 35 (2022): 9460–9471.

Following your advice, we conducted additional experiments on a summarization task. Specifically, we used the Reddit dataset from [6] and the reward model OpenAssistant/reward-model-deberta-v3-large-v2. First, we fine-tuned Qwen2.5 1.5B using the dataset to obtain an SFT reference model. We then applied both DPO and GVPO for post-training. The training data was sampled from the reference model and scored using the reward model.

For evaluation, we considered:

• The average reward of generated summaries on the test split.

• Win rate against human-written demonstration answers, as scored by the reward model.

• Preference accuracy using human-labeled comparison data: both DPO and GVPO are trained via $R_\theta(x, y) = \beta \frac{\log \pi_\theta(y|x)}{\log \pi_{\text{ref}}(y|x)} + \beta \log Z(x)$. Given a preference pair $(y_w, y_l)$ where $y_w$ is preferred, we compute whether $\frac{\log \pi_\theta(y_w|x)}{\log \pi_{\text{ref}}(y_w|x)} > \frac{\log \pi_\theta(y_l|x)}{\log \pi_{\text{ref}}(y_l|x)}$.

The results indicate that GVPO outperforms DPO across these metrics, supporting the view that improvements in policy search methods are positively correlated with alignment quality.

[6] Stiennon, Nisan, et al. Learning to summarize with human feedback. Advances in Neural Information Processing Systems 33 (2020): 3008–3021.

We appreciate your feedback and the recognition of the potential interest in our method. However, we believe there may be a misunderstanding regarding the motivation and contributions of the paper.

We clarify this further below and address your questions and concerns.

Overall, I think that the proposed method is interesting, but requires more in-depth empirical investigation. ... I would have really liked to see ablations targeting the purported variance-reduction benefits of GVPO.

We fully understand your concerns. However, we believe there may have been a misunderstanding. This paper does not aim to reduce variance, nor have we claimed that it can. Our primary objective is to propose a better post-training algorithm compared to GRPO. To that end, we have thoroughly compared GVPO and GRPO to highlight the benefits of our approach, and other reviewers have acknowledged this aspect positively.

Upon reflection, we recognize that the term "Variance" in GVPO’s name may have led to confusion. Our goal was to create a counterpart to GRPO's "Relative" term. Specifically, while GRPO's weight contains $R-\overline{R}$ (a relative term), GVPO’s MSE loss $[(R_\theta-\overline{R_\theta})-(R-\overline{R})]^2$ can also be expressed as a variance-like term, $[(R_\theta-R)-\overline{(R_\theta-R)}]^2$. We will clarify this distinction in the revised draft.

Motivation

The motivation for the method is that GRPO is unstable ... but there is no evidence within the paper that this is true, nor citations for this claim.

Still, we believe there may have been a misunderstanding. The motivation of this paper is not: because GRPO is unstable, so we improve its stability. But instead is: GRPO are reported not effective in practice (and many people attributes to its instability), so we propose a more effective one. In other words, this work focuses on better performance rather than low variance. Framing our motivation as “variance reduction” risks underestimating the true contribution of our work.

To clarify, we will add relevant references and provide a more clear explanation in Lines 106-111:

$

\mathcal{L}(\theta) = -\sum_{(x,{y_i}) \in \mathcal{D}} \sum_{i=1}^k \frac{R_i-\overline{R}}{\sigma(R)}\log \pi_\theta(y_i|x)

$

Rewards below average lead to negative weights, penalizing their likelihoods. However, minimizing likelihood can result in unstable training [1]. Additionally, off-policy training of GRPO, which involves importance sampling with the weight $\frac{\pi_\theta}{\pi_{\theta_{old}}}$, becomes unstable when $\pi_\theta$ significantly deviates from $\pi_{\theta_{\text{ref}}}$, potentially causing gradient explosions. To mitigate these issues, GRPO uses gradient clipping and a KL-divergence constraint between the updated and initial policies:

$

\mathcal{L}(\theta) = -\sum_{(x,{y_i}) \in \mathcal{D}} \sum_{i=1}^k Clip(\frac{\pi_\theta(y_i|x)}{\pi_\text{old}(y_i|x)})\frac{R_i-\overline{R}}{\sigma(R)}\log \pi_\theta(y_i|x)- \beta KL(\pi_\theta||\pi_\text{ref})

$

Nevertheless, empirical results [2,3] show that GRPO still exhibits training instability, which undermines its performance.

[1] Abdolmaleki, Abbas, et al. "Learning from negative feedback, or positive feedback or both." ICLR. 2025.

[2] Yu Q, Zhang Z, Zhu R, et al. Dapo: An open-source llm reinforcement learning system at scale[J]. arXiv preprint arXiv:2503.14476, 2025.

[3] Hu, Jian, Jason Klein Liu, and Wei Shen. "Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models." arXiv preprint arXiv:2501.03262 (2025).

Why is this (reduce policy learning to reward learning) better than optimizing the policy? And what is the relationship with GVPO (Eq. 9) itself?

Reducing policy learning to "reward learning" is also a key advantage of DPO: it does not replace policy optimization but rather provides a more smart pathway to it. This is because this method simplifies the joint optimization over policies and rewards to a simpler problem focused solely on rewards. This makes the problem more tractable, as it only requires aligning implicit rewards $R_\theta(x, y)$ with the true reward function $R(x, y)$ (Line 215-217). For instance, in joint optimization, the decision of whether to add standard deviation normalization is ambiguous. In contrast, "reward learning" only necessitates designing a loss function that trains $R_\theta$ to align with $R$, on which the question of whether adding standard deviation will depend. GVPO reduces policy learning to "reward learning" through an MSE loss.

Method and theoretical results

This seems possibly straightforward given the form of Eq. 9 and the definition of $R_\theta=\frac{\log \pi_\theta}{\log \pi_{\theta'}}$, and the theoretical results are somewhat simple

While the necessity of the theorem ($R_\theta = R$ is an optimal solution) follows directly, establishing its sufficiency (the only optimal solution is $R_\theta = R$) is significantly more challenging and is addressed in detail in Appendix B.1.

The sufficiency result is novel—for instance, DPO establishes only necessity, not sufficiency, and admits multiple optimal solutions, some of which does not correspond to the optimal solution of the KL-constrained reward maximization objective (see Lines 231–238).

I wonder if the paper might be made stronger if Sections 3.1 and 3.3 were shortened in favor of more insightful experiments and ablations

We agree with your suggestion and will prioritize experiments over less important theoretical discussions.

Experiments

It appears that no standard deviations or averages over seeds are reported, ..., I think that the experimental results would be more interesting if they explicitly highlighted the intended benefits of GVPO.

While the objective of this work is not to reduce variance, we have nonetheless conducted additional experiments using 10 random seeds for both methods on the 1.5B model with $k=4$ as a robustness check:

The results indicate that GVPO consistently outperforms GRPO in terms of performance, while maintaining comparable standard deviations.

The objective resembles the GRPO objective, except it doesn't divide by the standard deviation, and has an additional pair of log policy-ratio terms that can be viewed as implicit rewards.

We would like to respectfully point out that your comparison may overlook a critical component of GRPO—the clipped importance sampling weight, $Clip(\frac{\pi_\theta(y_i|x)}{\pi_\text{old}(y_i|x)})$. This clipping mechanism is essential in GRPO to prevent gradient explosion due to large importance weights. While it effectively stabilizes training, it introduces bias into the gradient estimates. In contrast, GVPO does not rely on importance sampling from the outset, thereby avoiding these issues entirely (see Lines 270–285). This represents a significant advantage over GRPO that may not have been fully considered in your comparison.

But it has been observed empirically in DPO that such policy-parameterized rewards can be suboptimal or less stable compared to reward models (Swamy et al. 2025; Lin et al. 2024).

Thank you for providing the relevant literature. Our interpretation is as follows: Although DPO employs log-ratios as implicit rewards—thereby functioning similarly to reward models—it is, at its core, a policy that generates responses. It is therefore reasonable to expect that it may be suboptimal for the specific task of reward estimation, especially when compared to reward models explicitly trained to output scalar values.

As such I am curious about the effect of the policy ratio terms, I would think that there could be interesting ablations to conduct on their role in stability, as well as in probing the predictions in Section 3.5.

We conduct comprehensive ablation studies on the regularization terms.

First, we remove $Cov(\log\pi_\theta, \log\pi_{\theta^\prime})$ and $Var(\log\pi_\theta)$ individually. In both cases, the models fail to converge and produce incoherent outputs, indicating that these terms are essential for stabilizing training. Next, we remove both regularization terms simultaneously, i.e., removing the ratio terms and reducing the objective to $R - \overline{R}$. While this model initially converges, it begins to diverge after approximately 10% of the training steps, further confirming that the regularization terms are critical components of GVPO.

Questions

Why does incorporating the log-policy ratio terms into GVPO increase stability?

From the perspective of regularization, the log-policy ratio terms are equivalent to the sum of $Var$ and $Cov$ after taking gradients. These two components help discourage extreme probability assignments and prevent the current policy from deviating excessively from a reference policy. Both effects contribute to improved stability.

Thank you again for your comments. We have carefully considered them and addressed the concerns in our responses above.