VinePPO: Unlocking RL Potential For LLM Reasoning Through Refined Credit Assignment

We thank the reviewer for their time and feedback.

We should note that we don’t completely agree with the provided summary. In our work we don’t show “PPO barely outperforms a random baseline”. Instead, we believe the reviewer meant that “The value network in PPO performs just slightly better than a random choice when trying to rank the values of actions.”

We now address the concerns raised in the review.

Variance and efficiency of MC estimation

While variance and computation efficiency are valid concerns for any MC-based estimation, we studied these extensively in Sec. 6.4 of the initial submission (noted by review 1twu): specifically, in Figs. 4, 7, D.7 of the original paper, and now in Fig. F.22 and F.23 of the updated draft. Across all experiments, VinePPO with its MC value estimation consistently outperformed all baselines (including new ones suggested during the Rebuttal) in both efficiency and accuracy, as noted by reviewer wh2b.

Also, while MC has been studied for a long time, we note that, as reviewer S7we highlighted, “The idea of Monte Carlo estimates, although it has been used in traditional RL tasks, is novel for PPO in the context of LLMs.” Moreover, MC value estimation in policy-gradient method is limited to environments that allow intermediate state resets, rare in classic RL environments [1]

Importance of K

The impact of K is studied in Fig. 4 of the original draft. Additionally, we study the effect of K in efficiency in Fig. F.23 of the updated paper. As noted by reviewer s7we, even small values of K (e.g. 1 or 3) perform well and outperform PPO with its value network (Fig. 4). Additionally, as shown in Fig. 23, higher values of K actually improve the wall-clock time efficiency in terms of reaching a target accuracy.

Generalizability

VinePPO is built on standard PPO from RLHF and introduces no additional assumptions to the RL formulation. Our empirical results demonstrate that VinePPO is more efficient than PPO, making it broadly applicable to RL problems in language environments where PPO is typically used, as noted by reviewer S7WE.

We highlight that the MATH dataset, our primary evaluation suite, includes competition-level problems with long trajectories (up to 2500 toks). Common RLHF methods like DPO often struggle in such tasks [2], demonstrating the challenging nature of our setup.

Questions

Q1. influence of K on performance and efficiency

As mentioned earlier, the impact of K on performance is analyzed in Fig. 4 of the original paper, showing that increasing K improves test accuracy. The effect of K on wall-clock efficiency is detailed in Fig. F.23, where VinePPO with higher K values shows greater efficiency. Specifically, VinePPO with K=9 is slightly more efficient than K=3, and both significantly outperform K=1 in reaching a target accuracy.

Q2. does VinePPO outperform PPO when MC estimation is not inaccurate?

If very few MC estimates are used, high variance could theoretically hinder training. However, we did not observe this empirically. As shown in Fig. 4 and F.23, even with one MC sample, VinePPO outperforms PPO in both final performance and wall-clock efficiency (as noted by reviewer S7We)

Q3. In Fig 9, the ground truth is chosen as results via 256 MC samples. Is this reasonable?

Yes. In our tasks, the value of a state follows a Bernoulli distribution representing the probability of successfully completing a solution. Assuming the maximum possible variance of this distribution (0.25), the variance of the ground truth estimator is 0.25 / 256 = 0.00097656, which is notably small.

Q4. are critical steps detected by VinePPO?

Thank you for the suggestion. Beyond Fig. 1.a in the original paper, additional examples are in Figures H.27–H.29 of the updated draft. Fig. H.27 shows an insightful reasoning step with positive advantages recovered by VinePPO, while Fig. H.28 illustrates an erroneous step with negative advantages. However, high or low values reflect the policy’s likelihood of solving the problem, which may not always align with human judgments of correctness (see Fig. H.29).

Q5. Some equations are not clear, for example: St+1=st;[at]

As noted in the background (lines 198–200), $s_t;[a_t]$ refers to “appending action $a_t$ to state $s_t$.” We will clarify this in the camera-ready version. If there are other equations requiring clarification, we are happy to address them too.

Thank you for your effort again. Efficiency was a key focus of our work, and we conducted a thorough analysis of VinePPO’s efficiency in the paper. Across all experiments VinePPO consistently demonstrated superior efficiency and final performance compared to baselines. We hope this clarification and the additional empirical evidence address your concerns and encourage a fresh evaluation of our work.

• [1] Trust Region Policy Optimization by Schulman et al. 2015
• [2] SimPO: Simple Preference Optimization with a Reference-Free Reward by Meng et al. 2024

Dear Reviewer,

Thank you for your time and feedback. With the end of the discussion phase approaching, we’re open to addressing any additional concerns or suggestions that could help us obtain a stronger evaluation. Especially, we updated the paper with additional efficiency plots on effect of K. Thank you once again for your time and effort in reviewing our work.

Dear Reviewer,

You raised efficiency as a concern regarding VinePPO. This was a primary focus of our work, and we extensively addressed it in the initial draft (see Section 6.4). During the rebuttal, we provided additional efficiency analysis as per your request. All empirical results consistently demonstrate VinePPO’s superior efficiency, achieving higher test accuracy in less wall-clock time in every experiment (see Figures 7, F.22, and F.23). As today is the final day for updates, we would appreciate knowing if any concerns remain that influenced your evaluation.

Dear Reviewer,

Thank you for your attention to our rebuttal and for considering increasing your score. Could we kindly ask you to update your rating in OpenReview using the "Edit" button on your first comment (your official review), as verbal mentions don't reflect in the review system?

Dear Reviewer,

Thank you for your consideration of our rebuttal. As the reviewer response deadline is tomorrow, we'd like to kindly remind you that your increased score is still not reflected in OpenReview. We kindly request that you update your score using the "Edit" option in your official review since verbal mentions are not reflected in the system.

We appreciate your time and effort!

We would like to thank the reviewer for their positive and insightful feedback. We will address mentioned points in detail as follows:

RLOO and GRPO Baselines

Thank you for the suggestion. As it was a common recommendation from reviewers, we updated the paper to include the result along with an in-depth analysis. We post this as a general comment. Please see “Summary Response + RLOO and GRPO Baselines” on OpenReview. A brief summary is provided below:

RLOO and GRPO show a clear disadvantage compared to VinePPO. On GSM8K, they score 44.5% and 44.6%, respectively, while PPO achieves 50.1%, and VinePPO achieves 53.4%. A similar pattern is observed on MATH, where RLOO and GRPO score 17.3% and 17.8%, compared to PPO’s 18.1% and VinePPO’s 23.0%. Additionally, training RLOO and GRPO proved to be less stable; for instance, we found it necessary to use higher KL coefficients during hyperparameter tuning to prevent instability. Notably, when controlling for wall-clock time efficiency, we found VinePPO achieves their peak performance up to 2.7x faster.

We also invite the reviewer to see the additional analysis included in the general comment, as we believe they offer interesting insights into the inner workings of RLOO and GRPO.

details on how the inference engines were utilized

We appreciate the reviewer’s interest in the technical details. We used the vLLM library [1] for fast inference and found that fortunately no special techniques were needed to achieve high throughput. Even 7B models can be deployed on each rank without any issues. That is, at every iteration current policy is loaded to the vLLM server and then sampling is done through vLLM serving API. We will update Section C.9 (“Software Stack”) in the camera-ready version to include more details, Additionally, we plan to release our code with the camera-ready version, allowing others to use our generation pipeline.

Questions:

Q1. Missing citations

We thank the reviewer for their suggestion and we make sure to cite these works in the camera ready paper.

Q2. How does VinePPO compare to the RLOO baseline as the value of K increases in RLOO?

Great question! Currently, we have focused our computational resources on running the RLOO and GRPO experiments with the 7B model. If time permits, we will conduct experiments with varying K values and share the results.

Q3. Did you do large-batch PPO updates?

We thank the reviewer for their detailed suggestion. We are using a relatively large rollout batch size of 512 and mini-batch size of 64. Note that we do not have a reward model running on the GPU. In our tasks, the reward function is a Python program that compares the model's output with the ground truth.

Q4. Why is PPO more deterministic in early steps, while VinePPO is more deterministic in later steps, as mentioned in the "Error per reasoning step" section?

We hypothesize that the value network relies on memorization. In the early steps, responses align closely with the training data, enabling accurate estimates. However, as responses progress, the number of possible sequences grows combinatorially, quickly moving out of the training data distribution. In contrast, MC estimation is not tied to the training data. At later steps, the LLM, conditioned on a long solution, becomes deterministic, requiring fewer MC samples for accurate estimation.

Q5. Could you share a plot showing the "explained variance" of the value function you learn with normal PPO?

Thank you for suggesting the "explained variance" metric—we found it very insightful and computed this for other methods too. As shown in Figure G.26 of the updated paper, PPO’s value network predictions exhibit non-negative explained variance values close to one, reflecting healthy and effective training. Notably, VinePPO achieves higher explained variance in value predictions compared to PPO, RLOO, and GRPO.

Thank you again for your valuable feedback; your suggestions have improved our paper. We hope we have adequately addressed all of your concerns and would be happy to clarify further if needed. With this in mind, we hope the reviewer increases their rating of our paper.

• [1] Kwon, Woosuk, et al. "Efficient memory management for large language model serving with pagedattention." Proceedings of the 29th Symposium on Operating Systems Principles. 2023.

Dear Reviewer,

Thank you for your constructive comments. As the discussion phase is ending, we would love to address any remaining questions or incorporate further suggestions to obtain a better evaluation. Especially, we had already added RLOO and GRPO results for the 1B model and have now included a 7B result as it became available. We deeply appreciate your time and commitment to the review process.

Dear Reviewer,

Thank you for your valuable insights and suggestions. As tomorrow is the final day for reviewer response, we wonder if you've had the chance to go over our previous response, and if you have any feedback on the integration of highlighted papers, and any remaining concerns about the RLOO baseline, or other aspects that prevents a stronger evaluation of our work.

We would like to thank the reviewer for their thorough feedback. We’re thrilled that you liked our work. We now address the key questions below:

RLOO and GRPO Baselines

This is a great recommendation! As this was a common suggestion among reviewers, we posted a general comment describing RLOO and GRPO results with additional in-detail analysis. Please see “Summary Response + RLOO and GRPO Baselines”. Here, we provide a short summary for convenience:

RLOO and GRPO perform worse than VinePPO. On GSM8K, their respective scores of 44.5% and 44.6% fall short of PPO’s 50.1% and VinePPO’s 53.4%. A similar trend is observed on the MATH benchmark, where RLOO and GRPO achieve 17.3% and 17.8%, compared to PPO at 18.1% and VinePPO at 23.0%. Moreover, we found their training process to be less stable, requiring a higher KL coefficient during hyperparameter tuning to stabilize their training. This likely stems from their uniform credit assignment method, which contrasts sharply with the fine-grained credit assignment strategies employed by PPO and VinePPO (see “Value Prediction Analysis” in general comment).

VinePPO is significantly more efficient than RLOO and GRPO, reaching their peak performance 2.7 times and 2.2 times faster on MATH, respectively. In GSM8K, which is an easier task for a 1B value network, even PPO achieves peak performance of RLOO and GRPO about 1.75x faster.

Also, additional analysis included in the general comment offers deeper insights into the inner workings of RLOO and GRPO, which we highly recommend the reviewer to visit.

DPO variants with fine-grained credit assignment

There are DPO variants that aim to provide finer credit assignments, such as Self-Explore [1], as noted in our first draft (lines 132–126). Self-Explore reports an improvement from SFT (34.14%) to 37.68% on MATH using DeepSeekMath 7B [1]. In our work, PPO achieves a larger improvement, from SFT (32.8%) to 42.8% in the same setup. Given this and the engineering complexity of these methods, we decided to focus on more established baselines in the literature, such as RestEM, DPO+, PPO, and now RLOO, and GRPO.

Tuning K in VinePPO

Excellent question! Tuning VinePPO is generally simpler because it involves only one key parameter, K. In contrast, tuning the value network comes with numerous hyperparameters associated with neural network training, such as the optimizer, making it more complex. Additionally, Fig. 4 of the original paper and Fig. F.23 of the updated draft show performance and efficiency improvements as K increases, suggesting a straightforward heuristic: start with the highest K that fits within the available compute budget.

Questions

Q1. How does the method's dependency on K differ by model? I am also curious about the K ablation.

We provided an ablation study on the effect of K in Fig. 4. However, this ablation was conducted only on the 1B model due to computational constraints. Running ablation on the 7B model is prohibitively expensive. We expect to see the same pattern as increasing K always reduces the value estimation variance.

Q2. a graph to show the trade-off between larger K values and efficiency

Thanks for the suggestion! Refer to Fig. 23 of the updated draft for this study on MATH. The results are quite interesting. VinePPO with higher K values achieves greater efficiency. Specifically, VinePPO with K=9 is slightly more efficient than K=3, while both significantly outperform K=1. This demonstrates the strong impact of low-variance value estimates on training, shifting the trade-off towards improved efficiency with more samples.

Q3. missing period on line 264

Thank you! We’ll fix it in the final draft.

Thank you again for your valuable feedback. We hope that our response has resolved any remaining questions and concerns. Would you consider increasing your ratings given the main clarifying points outlined?

• [1] Hwang, Hyeonbin, et al. "Self-Explore to Avoid the Pit: Improving the Reasoning Capabilities of Language Models with Fine-grained Rewards." arXiv preprint arXiv:2404.10346 (2024).

Dear Reviewer,

Thank you for your valuable feedback. With the discussion phase nearing its end, we’d be glad to address any further questions or suggestions to obtain a better evaluation. Importantly, we previously updated the paper with RLOO and GRPO results for the 1B model, and we’ve now added a new result for the 7B model, which completed today. Thank you again for your time and dedication to the review process.

Thank you for reading our rebuttal in depth and we’re thrilled that you liked our experiments and analysis. We now answer the remaining concerns:

According to the GRPO paper, after applying GRPO, the performance on Math exceeds 50%. Was there any difference in the settings?

We appreciate the reviewer's attention to details. In the GRPO paper, they use a different SFT model trained on a large (unpublished) mathematical instruction dataset containing 776K examples. This SFT model achieves 46.8% on MATH, improving to 51.7% after GRPO. In comparison, our SFT model, trained on a public MATH dataset (around 11.5K examples), scores 32.8%, which improves to 42.8% after PPO (and 46.0% after VinePPO).

Additionally, it seems that the benefit of GRPO diminishes for larger, well-performing models (being much more effective for the 1.1B model). Is there any specific reason for this?

We hypothesize that larger models result in more capable value networks, which may lead to better credit assignment. As a result, PPO with value network might perform closer to or even better than methods like GRPO, which lack fine-grained credit assignment mechanisms. That said, even larger value networks are still brittle, struggling with diverse trajectories (as shown in Figure 8), questioning their true scalability.

We hope that our response has addressed your remaining concerns and we are happy to engage in additional discussion if anything remains unclear.

Thank you so much for the thorough response. I highly appreciate it, and I apologize for the late reply.

Few Remaining Concerns:

• According to the GRPO paper, after applying GRPO, the performance on Math exceeds 50%. Was there any difference in the settings?
• Additionally, it seems that the benefit of GRPO diminishes for larger, well-performing models (being much more effective for the 1.1B model). Is there any specific reason for this?

We thank the reviewer for their time reviewing our paper and providing useful feedback and questions which we now address.

RLOO and GRPO Baselines

As RLOO and GRPO were asked by three reviewers, we described the results with additional analysis in a general message. Please see “Summary Response + RLOO and GRPO Baselines” on OpenReview.

As a short summary here: RLOO and GRPO perform worse than VinePPO. RLOO and GRPO achieve 44.5% and 44.6% on GSM8K respectively (compared to PPO’s 50.1% and VinePPO’s 53.4%) and 17.3% and 17.8% on MATH respectively (compared to PPO’s 18.1% and VinePPO’s 23.0%). Also, we found RLOO and GRPO to be less stable. For example, we found during our hyperparameter tuning that we need a higher KL coefficient to stabilize their training. When controlling for compute budget, VinePPO (and even PPO in some cases) surpasses the peak performance of RLOO and GRPO up to 2.7x faster.

We also encourage the reviewer to refer to the additional analysis we performed in the general comment, which offers deeper insights into the inner workings of RLOO and GRPO.

Misuse of Terminology

So it is better not to use "bias" in Line 467 and 475 but to use "inaccuracy".

Thank you for your attention to details. We believe our use of terminology is accurate. While the policy gradient is unbiased when λ=1 (as the value estimates act only as a baseline), the value network’s estimates themselves can still be biased (see Sec. 3 of [1]) as they are approximated by a neural network (as shown in Figure 9). The term “bias” in line 467 “PPO’s value network shows high bias.” and in line 475 “PPO’s value network, despite its bias,...” specifically refers to this inherent bias in the value network’s estimation of the value, not the policy gradient. We will revise the text to clarify this distinction between bias in the policy gradient and the value estimates within the same section.

Questions:

Q1. can we extend it to the more general alignment task?

Yes. VinePPO is built on standard PPO from RLHF and does not introduce any additional assumptions to the RL formulation. So, it is broadly applicable to RL problems in language environments (as noted by reviewer S7WE), including alignment tasks where PPO is typically used.

Q2. Is there any intuitive or theoretical explanation for why value networks fail to provide accurate estimates?

Yes. Empirically, the evidence in Section 7 (see “Error Per Reasoning Step”) suggests that the value network primarily relies on memorization, as learning a generalizable algorithm is likely less favorable given the training data and the challenging nature of the task [2]. Intuitively, the task of the value network in reasoning tasks is quite demanding: the value network must 1) implicitly understand the correct answer, 2) evaluate how the LLM’s generated solutions align with it, and 3) achieve all this in a single forward pass. This can be especially demanding given the value network is initialized from the same LLM and has similar size and capacity.

We thank the reviewer again for their effort and feedback. We hope the clarifications have addressed your concerns. Given the key points outlined, would you consider increasing your ratings?

• [1] Schulman, John, et al. "High-dimensional continuous control using generalized advantage estimation." arXiv preprint arXiv:1506.02438 (2015).
• [2] Nagarajan, Vaishnavh et al. “Understanding the Failure Modes of Out-of-Distribution Generalization.” ArXiv abs/2010.15775 (2020): n. pag.

Dear Reviewer,

Thank you for your constructive feedback. As the discussion phase nears its end, we’d be happy to address any remaining questions or suggestions to obtain a better evaluation. Previously, we updated the paper with RLOO and GRPO results for the 1B model, and we now added a 7B result that became available. Thank you again for your time and dedication to the review process.

Dear Reviewer,

Thank you for suggesting GRPO as a baseline. We were wondering if you've had a chance to review our previous response and results regarding GRPO. Since the lack of a GRPO baseline was the primary weakness you mentioned, and our experiments demonstrate that VinePPO consistently outperforms GRPO, we wanted to ask if there are any remaining concerns that, if addressed, could help us get a stronger evaluation.

Dear Reviewer,

Thank you again for suggesting GRPO as a baseline. With only one day left until the reviewer response deadline, we wonder if you’ve had a chance to review the posted results and additional analysis we performed following your suggestion regarding GRPO. Since this was the primary weakness noted, and our experiments show VinePPO consistently outperforms GRPO and is even more efficient, we wanted to know if there is any remaining concern that prevents a stronger evaluation of our work?

Dear Reviewer,

As today is the last day for responses, we kindly ask if you’ve had a chance to go over our GRPO results and whether there are any remaining concerns preventing a stronger evaluation.

...continuation of the above comment:

Implementation and Training Details of RLOO and GRPO For RLOO, we closely follow the implementation in the HuggingFace TRL’s library [3]. GRPO implementation is also a straightforward modification to PPO. To ensure fair comparison, we maintain equal training dynamic across runs. Specifically, we train RLOO, GRPO, PPO, and VinePPO, for 1000 iterations on MATH and 650 iterations on GSM8K (about 8 epochs for both datasets). All methods share the same rollout batch size of 512 and mini batch-size of 64. In all methods, we sample 8 responses per each question in each rollout batch. Given the 8 training epochs all methods see 64 responses per example throughout training. For all method, we initialize the policy from the SFT checkpoint. For RLOO and GRPO, we further tune the KL coefficient (search space: {1e-2, 3e-3, 1e-3, 3e-4, 1e-4}). We found that RLOO and GRPO are quite unstable and need a higher KL coefficient to stabilize their training (in our experiments 3e-3 is the smallest value they can tolerate and achieve best validation accuracy). As a final note, the results of the RLOO and GRPO experiments look very similar. This is not a mistake. The baseline computation in RLOO and GRPO is indeed very close. Assume R1, R2, .., and Rk are task returns of K responses (Y1, Y2, .., Yk) for a prompt X. GRPO computes the baseline for policy gradient on Y1 by averaging all the returns. However, RLOO leaves the R1 out and takes the average over the remaining K-1 returns.

• [3] https://github.com/huggingface/trl