CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models

We sincerely appreciate your encouraging feedback on our work, including comments like "CPPO is a novel method that tackles a significant problem of training efficiency of GRPO", "clear motivation", and "The train time speedup, while retaining accuracy, is impressive, positioning CPPO as a viable alternative to GRPO." Please see our point-by-point responses to your comments below.

Q1: How do you compare in speedup and accuracy against other mentioned RL algorithms (REINFORCE++, PPO without KL divergence)?

A1: As shown in Table 1, we compare our CPPO with other reinforcement learning algorithms using the GSM8K test subset on the verl[1] framework, which supports various RL algorithms. CPPO achieves the best accuracy of 78.92% with a training time of 5192s, outperforming REINFORCE++ and PPO without KL divergence. We will include these results in the camera-ready version.

Table 1: Comparison between different reinforcement learning algorithms on the GSM8K test subset. We train Qwen2.5-1.5B-Instruct on the GSM8K training subset, and the number of retained completions after pruning is $k = \left\lfloor G \times (1 - P) \right\rfloor$. All experiments are conducted on the verl framework.

[1] Sheng et al. HybridFlow: A Flexible and Efficient RLHF Framework

Q2: Ideally, authors would evaluate the algorithm across the same models and capture the variance of results.

A2: As requested, for more rigid evaluation, we choose to conduct multiple independent runs for the Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct models and report the results in Tables 2 and 3. The results demonstrate the stability and robustness of our CPPO.

Table 2: Comparison between GRPO and CPPO on the GSM8K test subset. We independently train Qwen2.5-1.5B-Instruct on the GSM8K training subset three times to calculate the mean and standard deviation, and the number of retained completions after pruning is $k = \left\lfloor G \times (1 - P) \right\rfloor$.

Table 3: Comparison of GRPO and CPPO on the MATH test subset. We independently train Qwen2.5-7B-Instruct on the MATH training dataset three times to calculate the mean and standard deviation, and the number of retained completions after pruning is denoted by $k = \left\lfloor G \times (1 - P) \right\rfloor$.

Q3: The authors claim "CPPO reduces computational overhead without compromising model performance."; However, high pruning rate configurations of the CPPO-trained 7B Qwen model have lower accuracies on MATH and AMC compared to GRPO. The claim should be amended.

A3: Thank you for pointing this out. We will revise the claim to "CPPO reduces computational overhead while maintaining comparable model performance at an appropriate pruning rate. An excessively high pruning rate may lower the model performance."

Q4: The stopping criterion for training is not discussed in the 4.1 section.

A4: In our experimental settings, we use the same stopping criterion for all methods for fair comparison. Specifically, we set the training epoch to 1, and the training will stop after the model has been trained on the entire training set once. We will add this to Section 4.1 of the main paper.

Q5: In Figure 4, it is not clear how this demonstrates that accuracy stems from higher quality completions.

A5: In Figure 4, we compare the accuracy of GRPO and CPPO with different question numbers $b$ and retained completion numbers $k$ per training step. The key difference between CPPO and GRPO is that CPPO first generates a group of $G$ ($G > k$) completions and then retains only the top $k$ completions per training step, whereas GRPO directly generates $k$ completions per step. As a result, the completions retained by CPPO are of higher quality, since they are selected from a larger pool based on their absolute advantages. In contrast, the completions retained by GRPO are generated without selection and are more likely to include low-quality completions. The results in Figure 4 show that CPPO achieves higher accuracy than GRPO with all different settings of $b$ and $k$, demonstrating that the improved accuracy is due to the higher quality of completions retained by CPPO.

Q6: Why did you choose to use different models for different datasets? Can you provide some data showing it wouldn't be reasonable to do so?

A6: We choose different models for different datasets for two main reasons: (1) Demonstrating generalizability: Using various models across datasets shows that CPPO works for both small and large models, and on different datasets, highlighting its robustness. (2) Reasonable experimental settings: GSM8K contains 8.5K grade-school math problems, which are relatively simple for larger models like Qwen2.5-7B-Instruct (as shown in Table 3, its GSM8K accuracy is already 83%). MATH, on the other hand, includes 7.5K competition-level problems that require a stronger reasoning ability. Therefore, we use Qwen2.5-1.5B-Instruct for GSM8K and Qwen2.5-7B-Instruct for MATH to ensure the experiments are meaningful and challenging for each model.

Table 3: The initial reasoning capabilities of different models.

Q7: High pruning rates with the Qwen 7B model don't result in an increase in accuracy across MATH and AMC2023 datasets, yet you claim that "CPPO achieves better performance at high pruning rates". Can you explain this, as these two seem contradictory?

A7: Thank you for highlighting this issue. We will revise our claim to: "CPPO sometimes achieves better performance at higher pruning rates on the GSM8K and MATH datasets. For example, CPPO with a 75% pruning rate achieves 78.81% accuracy on GSM8K and 77.20% accuracy on MATH, compared to 77.67% and 75.20% accuracy with a 50% pruning rate, respectively." We will also clarify that CPPO with excessively high pruning rates, such as 93.75%, may influence the model performance negatively on some datasets, such as MATH and AMC2023.

Thank you again for spending time reviewing our work and providing valuable feedback. We hope our responses address your concerns. If you have any further questions or suggestions, please feel free to let us know.

We extend our gratitude to your valuable and encouraging feedback on our work, such as "genuinely novel", "well-motivated", and "well-executed". Please kindly see our responses to your comments below.

Q1: The experiments are conducted only on the Qwen2.5 series, making it difficult to assess whether the proposed method generalizes to different LLM backbones.

A1: We conduct experiments on the Llama series models and present the results in Table 1. The results show that CPPO can also accelerate the training of Llama models without compromising accuracy and achieving a significant speedup of up to 3.13×. This demonstrates the generalizability of CPPO across different LLM backbones. We will include these results in the camera-ready version.

Table 1: Comparison between GRPO and CPPO on the GSM8K test subset. We train Llama-3.2-1B-Instruct on the GSM8K training subset, and the number of retained completions after pruning is $k = \left\lfloor G \times (1 - P) \right\rfloor$. All experiments are conducted on the verl framework, which supports various RL algorithms.

Q2: As a highly engineering-oriented work, I strongly recommend that the authors release their code, as it would be greatly beneficial to the community.

A2: Thank you for your suggestion. The code is included in the supplementary material. We will also release our code publicly upon acceptance of the paper.

Q3: Can this algorithm generalize to other RL variants?

A3: CPPO reduces training cost by pruning low-quality completions. Therefore, CPPO can be generalized to other group relative policy optimization based RL algorithms such as DAPO[1] and Dr.GRPO[2]. As shown in Table 2, CPPO can be combined with DAPO and Dr.GRPO to further improve training speed and accuracy, demonstrating the strong generalizability of CPPO.

Table 2: Comparison between different reinforcement learning algorithms on the GSM8K test subset. We train Qwen2.5-1.5B-Instruct on the GSM8K training subset, and the number of retained completions after pruning is $k = \left\lfloor G \times (1 - P) \right\rfloor$. All experiments are conducted on the verl framework, which supports various RL algorithms.

[1] Yu, Qiying, et al. "Dapo: An open-source llm reinforcement learning system at scale." arXiv preprint arXiv:2503.14476 (2025).
[2] Liu, Zichen, et al. "Understanding r1-zero-like training: A critical perspective." arXiv preprint arXiv:2503.20783 (2025).

We hope our responses address your concerns. If you have any further questions, please let us know. Thank you again for your time and valuable feedback.

Thank you for spending time reviewing our work. Please kindly see our responses to your comments below.

Q1: Threshold Sensitivity: How sensitive is CPPO’s performance to the choice of $\gamma$? Have you evaluated multiple threshold settings to identify an optimal or adaptive strategy?

A1: Eq. (7) is only an intermediate step in deriving our final policy objective function. Final policy objective function is formally defined in Eq. (9) and Eq. (10) of the main paper. We do not use a fixed threshold $\gamma$ in our final policy objective function. Since the range of advantage values varies across tasks and questions, using a fixed threshold $\gamma$ for completion pruning is unsuitable. During multi-GPU training, we observed that the number of completions exceeding a fixed threshold differs across devices, resulting in overall efficiency being limited by the device with the most completions (the "bucket effect"). Therefore, for each GPU, CPPO retains the $k$ completions with the largest absolute advantages for each question, rather than using a fixed threshold. More details can be found in Section 3.3 of the main paper.

Q2: Integration with Inference Acceleration: In scenarios where completion generation dominates, have you explored combining CPPO with methods like TokenSkip or speculative sampling to further reduce wall-clock time?

A2: We would like to clarify that inference acceleration methods are orthogonal to our work. Even in scenarios where completion generation dominates, our CPPO can achieve a considerable speedup, such as 2.83x (Table 2 with the 14B model). Inference acceleration is a popular research topic, and we believe that combining CPPO with more advanced inference acceleration methods can lead to further improvements in training speed and efficiency, which will also be our future focus.

Q3: Generality to Larger Models and Tasks: Could you extend CPPO’s evaluation to models > 7B parameters and non-math reasoning benchmarks to validate scalability and robustness?

A3: As requested, we conduct experiments on larger models, specifically Qwen2.5-14B-Instruct, and present the results in Table 2. The results show that CPPO can accelerate the training of larger models by up to 2.83x without compromising accuracy, demonstrating the scalability and generalizability of our method. While we aim to evaluate our method on even larger models, limited academic computational resources currently restrict us to Qwen2.5-14B-Instruct. Additionally, we evaluate our method on more benchmark datasets in Table 3. Specifically, we train Qwen2.5-7B-Instruct on the MATH training dataset and evaluate the model on non-math QA datasets (GPQA:Diamond) and code generation tasks (LiveCodeBench). The results show that both CPPO and GRPO, when trained on math datasets, achieve improved performance on GPQA. However, due to the large domain gap between math and code generation, performance on code generation tasks is slightly reduced for both methods, with CPPO exhibiting less degradation. This demonstrates that our method can accelerate GRPO training while maintaining the model’s generalization ability. Validating rule-based reinforcement learning algorithms on math tasks is common practice in this field [1][2], so we also choose math tasks to evaluate our method in this paper. Evaluating CPPO on more tasks, such as code generation and multimodal tasks, will be our future work, as these require substantial computational resources and engineering effort.

Table 2: Comparison of GRPO and CPPO on the MATH test subset. We train Qwen2.5-14B-Instruct on the MATH training dataset, and the number of retained completions after pruning is denoted by $k = \left\lfloor G \times (1 - P) \right\rfloor$.

Table 3: Comparison between GRPO and CPPO on additional benchmark datasets. We train Qwen2.5-7B-Instruct on the MATH training dataset and evaluate the model on various out-of-distribution benchmarks using lighteval. The number of retained completions after pruning is given by $k = \left\lfloor G \times (1 - P) \right\rfloor$.

[1] Yu, Qiying, et al. "Dapo: An open-source llm reinforcement learning system at scale." arXiv preprint arXiv:2503.14476 (2025).
[2] Liu, Zichen, et al. "Understanding r1-zero-like training: A critical perspective." arXiv preprint arXiv:2503.20783 (2025).

Q4: Robustness of Results: Have you conducted multiple independent runs to compute error bars or statistical significance measures (e.g., t-tests) for the reported accuracy and speedup?

A4: GRPO experiments are computationally expensive and time-consuming, so we do not include error bars in our main paper, following common practice in the community [1][2]. As requested, for more rigorous evaluation, we conducted multiple independent runs for the Qwen2.5-1.5B-Instruct and Qwen2.5-7B-Instruct models and report the results in Tables 4 and 5. The results demonstrate the stability and robustness of CPPO.

Table 4: Comparison between GRPO and CPPO on the GSM8K test subset. We independently train Qwen2.5-1.5B-Instruct on the GSM8K training subset three times to calculate the mean and standard deviation, and the number of retained completions after pruning is $k = \left\lfloor G \times (1 - P) \right\rfloor$.

Table 5: Comparison of GRPO and CPPO on the MATH test subset. We independently train Qwen2.5-7B-Instruct on the MATH training dataset three times to calculate the mean and standard deviation, and the number of retained completions after pruning is denoted by $k = \left\lfloor G \times (1 - P) \right\rfloor$.

[1] Shao, Zhihong, et al. "Deepseekmath: Pushing the limits of mathematical reasoning in open language models." arXiv preprint arXiv:2402.03300 (2024).
[2] Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning." arXiv preprint arXiv:2501.12948 (2025).

We hope that our responses can well address your concerns. If you have any further questions, please feel free to contact us. Thank you again for spending time reviewing our work.

Thank you for your valuable feedback, including remarks such as "The problem formulation is clear and well-motivated" and "a good intuitive solution." We give our point-by-point responses to your comments below:

Q1: Algorithmic innovation.

A1: With all due respect, we believe our work presents significant algorithmic innovation. GRPO is a widely used and influential reinforcement learning algorithm, but its substantial training overhead limits efficiency and scalability. Improving training efficiency is an important and practical problem. To our knowledge, this is the first study to accelerate GRPO by pruning completions, supported by rigorous theoretical analysis. In Appendix B, we provide detailed derivations, and in Section 3.2, we analyze the gradient of the GRPO policy objective to clarify the role of each term. Our analysis demonstrates that not all completions contribute equally to the policy gradient, and the contribution of each completion depends on its absolute advantage value. Based on this insight, we propose CPPO and dynamic completion allocation to speed up GRPO training by a considerable margin. All the theoretical analyses, insights, and methods are introduced for the first time in this paper. The novelty and theoretical soundness of our work are also recognized by reviewers ZKXt and kqfe.

Q2: More benchmark datasets.

A2: We evaluate models trained with GRPO and CPPO on the MATH training dataset and test them on several out-of-distribution benchmarks, including GPQA:Diamond, Olympiad_bench:zh, Agieval:gaokao-mathqa, MMLU:college_mathematics, and MathQA. As shown in Table 1, CPPO achieves comparable average performance to GRPO across these datasets, demonstrating that CPPO can accelerate GRPO training without compromising the model’s generalization ability. While we agree that evaluating our method on more benchmark datasets would be beneficial, limited academic computational resources restrict us to experiments on representative and high-quality datasets such as GSM8K, MATH, AMC, and AIME to validate the effectiveness of our method. We will continue to evaluate our method on additional benchmarks and tasks in future work.

Table 1: Comparison between GRPO and CPPO on additional benchmark datasets. We train Qwen2.5-7B-Instruct on the MATH training dataset and evaluate the model on various out-of-distribution benchmarks using lighteval. The number of retained completions after pruning is given by $k = \left\lfloor G \times (1 - P) \right\rfloor$.

Q3: Comparison with other acceleration methods for RL training.

A3: Our work aims to accelerate the GRPO algorithm, which is widely used and influential in reinforcement learning. Due to significant differences between GRPO and other reinforcement learning algorithms, such as PPO, it is not appropriate to compare our method with acceleration techniques designed for other algorithms. To our knowledge, this is the first work to accelerate GRPO by pruning completions. Therefore, we only compare our method with CPPO variants in Table 3 of the main paper. Recently, a concurrent work[1] has also sought to accelerate GRPO, but it focus on different aspects and is orthogonal to our work. As shown in Table 2, CPPO can be combined with it to further improve training time and accuracy. We will include these results in the camera-ready version.

Table 2: Comparison between different reinforcement learning algorithms on the GSM8K test subset. We train Qwen2.5-1.5B-Instruct on the GSM8K training subset, and the number of retained completions after pruning is $k = \left\lfloor G \times (1 - P) \right\rfloor$. All experiments are conducted on the verl framework, which supports various RL algorithms.

[1] Yu, Qiying, et al. "Dapo: An open-source llm reinforcement learning system at scale." arXiv preprint arXiv:2503.14476 (2025).

Q4: Comparisons with other completion selection strategies.

A4: As shown in Table 3 of the main paper, we compare our CPPO with other completion selection strategies. "Random" refers to randomly pruning completions. "Largest"/"Smallest" prune completions with the highest/lowest absolute advantages, respectively. "Largest*"/"Smallest*" prune completions with the highest/lowest advantages, which is equivalent to pruning completions with the highest/lowest rewards, since higher rewards correspond to higher advantages according to Eq.(3) of the main paper. The results in Table 3 of the main paper demonstrate that our CPPO, which prunes completions with the smallest absolute advantages, outperforms all other completion selection strategies.

Q5: Experimental results on larger models beyond 7B.

A5: As requested, we conduct experiments on larger models, specifically Qwen2.5-14B-Instruct, and present the results in Table 3. The results show that our CPPO can also accelerate the training of larger models by up to 2.83x without compromising accuracy, demonstrating the scalability and generalizability of our method. We would like to evaluate our method on even larger models, but due to limited academic computational resources, we can only conduct experiments on Qwen2.5-14B-Instruct at this time. We will include these results in the main paper.

Table 3: Comparison of GRPO and CPPO on the MATH test subset. We train Qwen2.5-14B-Instruct on the MATH training dataset, and the number of retained completions after pruning is denoted by $k = \left\lfloor G \times (1 - P) \right\rfloor$.

Q6: How to set hyperparameters for different settings? If different tasks want to adopt this technique, how do they choose $k$, $b$, and the pruning rate $P$?

A6: Assuming the group size is $G$ and the pruning rate is $P$, the hyperparameters are set as $k = \left\lfloor G \times (1 - P) \right\rfloor$ and $b = G / k$. Therefore, it is only necessary to choose appropriate values for $G$ and $P$ for different tasks. A larger group size $G$ generally improves performance but increases training cost, while a higher pruning rate $P$ provides greater speedup but may degrade performance if set too high. We recommend selecting $G$ and $P$ based on available computational resources and task performance requirements. In most cases, setting $G$ to 16 and $P$ to 50% achieves a good balance between performance and speedup.

We hope our responses address your concerns. If you have any further questions, please feel free to contact us. Thank you again for spending time reviewing our work and providing valuable feedback.