Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts

Response to Reviewer 4 (sAP6)

Thank you very much for your thoughtful review and constructive suggestions. We are glad that you found our idea both interesting and novel, and the presentation clear. We appreciate the opportunity to address the points you raised and hope you will consider raising your score in light of our response.

---

Q1: GRPO can be treated as a baseline.

A1: Thanks for your suggestion to compare GRESO with GRPO. We would like to address your concerns from the following two perspectives:

1. We have already included a GRPO as a baseline in Figure 1 in the main paper: GRPO without DS can achieve a worse final accuracy in some settings.

2. GRESO is complementary to GRPO and can be applied to accelerate vanilla GRPO.

3. Discussion of GRPO performance on DeepSeek-R1-Distill-Qwen-1.5B.

1. GRPO without DS can achieve a worse final accuracy in some settings. We included GRPO as a baseline for training the Qwen2.5-Math-7B model in Figure 1. We show that, although GRPO training is faster, but achieves a worse final accuracy. This phenomenon appears in several settings, like the 32B model results reported in the DAPO paper [1]. Similarly, in our additional RLOO (Q2) and Code (Q5) evaluations in the rebuttal, we also notice that vanilla GRPO achieves an inferior performance. For this reason, many recent methods adopt DS as a default component to stabilize training [1,2,3]. Therefore, in our main comparison table, we focus our comparison on DS rather than GRPO.

2.GRESO is complementary Vanilla GRPO: More importantly, GRESO is not incompatible with GRPO—in fact, the GRESO and GRPO are complementary. While our paper focuses on applying GRESO to accelerate DS, the same idea can be used to improve the efficiency of GRPO as well. Given a batch of prompts in GRPO, GRESO can be used to predict which prompts are likely to result in zero-variance rollouts. These prompts can then be skipped during rollout generation, reducing redundant computation. To verify this claim, we conduct an evaluation by applying GRESO to GRPO for training DeepSeek-R1-Distill-Qwen-1.5B on the DAPO+MATH dataset. The results are as follows:

Best Accuray for AIME24 (This result exceeds the paper-reported number because Table 1 reports checkpoints selected for the best average accuracy across benchmarks, whereas here we report the best accuracy on this individual benchmark.):

Best Accuray for AIME24:

Best Accuray for AIME25:

Total #rollouts for 1000 iterations:

For both AIME24 and AIME25, GRESO achieves comparable performance while significantly reducing the number of rollouts. For all 1000 training iterations, GRPO has accumulated 1.02 million rollouts, whereas GRPO + GRESO only requires 650k—achieving over 35% reduction in rollout cost for the same number of updates. These results confirm that GRESO not only accelerates DS training but also boosts the efficiency of vanilla GRPO by reducing redundant rollouts.

3. Discussion of GRPO performance on DeepSeek-R1-Distill-Qwen-1.5B for Math. Thank you for sharing the evaluation results on DeepSeek-R1-Distill-Qwen-1.5B. We also note that the effectiveness of DS varies depending on the training dataset difficulty for the base model. For instance, when training on DeepSeek-R1-Distill-Qwen-1.5B, the observed zero-variance rate is relatively low (around 50%), allowing GRPO to maintain training quality. In contrast, models such as Qwen2.5-Math-7B exhibit much higher zero-variance rates (up to 80%) under the same setup, which significantly harms performance. Furthermore, as we will present in Q5, when training DeepSeek-R1-Distill-Qwen-1.5B on the code_contests dataset, vanilla GRPO yields inferior performance.

We thank the reviewer again for the insightful suggestion. We will include this discussion and the additional experimental results in the revised version of the paper to better illustrate the relationship between GRESO and GRPO

Q2: Selective rollout can also be applied with other baselines like VC-PPO and VAPO.

A2: We agree that GRESO can be applied to other RL methods. However, VC-PPO and VAPO are value-based approaches that do not suffer from the zero-variance issue in RLVR, which GRESO is specifically designed to address. To demonstrate the generality of GRESO, we applied it to RLOO [4] on the MATH + DAPO benchmark using Qwen2.5-Math-7B and report average accuracy across 6 benchmarks used in the paper. The results are as follows (we report the best accuracy and the corresponding number of rollouts at that iteration):

These results show that GRESO improves accuracy over vanilla RLOO while requiring less than half the number of rollouts compared to RLOO+DS. This demonstrates that GRESO is both effective and efficient in other RLVR methods. While GRESO was not applied to value-based methods in this work, we agree that selective rollout for value-based RL is an important direction. We will include a discussion of this extension in the main paper, and we consider it a promising direction for our future research.

Q3: Can the authors provide some insights on the exploration mechanism in GRESO?

A3: As shown by the green curve in Figure 4(a) of the main paper, approximately 20% of prompts previously labeled as zero-variance become effective again in subsequent training. This highlights the importance of maintaining a certain level of exploration—even for prompts that were previously considered uninformative. GRESO addresses this by computing an exploration probability for each prompt based on its recent zero-variance history. Specifically, the more frequently a prompt has been identified as zero-variance in consecutive rollouts, the less likely it is to be informative, and thus it is assigned a lower exploration probability. This adaptive mechanism allows GRESO to selectively revisit potentially useful prompts while minimizing unnecessary rollouts.

Q4: Evaluation on 32B models.

A4: Thank you for the suggestion. We have already initiated experiments on 14B and 32B-scale models to evaluate the scalability of GRESO. Due to the high computational cost, these runs are still in progress, but we report interim results below (average accuracy across the 6 benchmarks used in the paper):

These preliminary results indicate that GRESO achieves comparable accuracy to Dynamic Sampling while using fewer rollouts. We are continuing these experiments and will update the results during the discussion phase. We will also ensure that the complete results, especially for the 32B model, are included in the final version of the paper.

Q5: Can GRESO be extended to other domains beyond math reasoning, such as coding?

A5: Yes, GRESO can be extended beyond math reasoning. To further evaluate the generalizability of GRESO, we applied it to the code generation domain. Specifically, we trained DeepSeek-R1-Distill-Qwen-1.5B on the code_contests dataset and evaluated the models on the LiveCodeBench benchmark. The results are as follows (we report the best accuracy and the corresponding number of rollouts at that iteration):

We trained all the models until convergence. GRESO achieves comparable accuracy to DS with fewer rollouts, and GRPO fails to match the final accuracy of GRESO and DS. These results suggest that GRESO is also effective in other reasoning-intensive domains such as code generation. Thank you for the suggestion, we will include the results in the final version of our paper.

Q6: How are prompts designed for the paper?

A6: We empirically selected our prompt by adding a reasoning instruction to the default Verl prompt used for Math tasks (our final prompt is provided in Appendix E). We also evaluated the DAPO prompt during our experiments, but found that it led to inferior performance. We hypothesize that, one possible reason for the performance drop is the mismatch in answer formatting — the DeepSeek distilled model is trained to place the final answer in a box, whereas the DAPO template does not follow it.

---

Thanks for the attentive reading of the manuscript and constructive feedback. We will incorporate these changes into our final version. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.

---

[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale

[2] ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models.

[3] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning

[4] BUY 4 REINFORCE SAMPLES, GET A BASELINE FOR FREE!

Response to Reviewer 3 (vbkh)

Thank you very much for your thoughtful review and constructive suggestions. We are glad that you found our evaluation extensively shows GRESO achieves performance gain and the analysis is thorough. We appreciate the opportunity to address the points you raised and hope you will consider raising your score in light of our response.

---

Q1: The proposed method highly relies on the premise that the training queries are used multiple times in RLVR, which might not holds when the base model is strong, which usually trains less than one epoch to avoid overfitting or reward hacking.

A1: Multi-epoch RL training is a common practice in existing RLHF and RLVR pipelines [1,2,3,4], even for strong base models such as 32B-scale LLMs [1]. In our experiments, we also observe that training over multiple epochs consistently leads to performance improvements. Therefore, we believe this is a valid and practical setting that can benefit from selective rollout strategies like GRESO.

However, we acknowledge this as a valid limitation. One potential way to address it is to use the base model to profile the dataset before training, enabling effective data filtering even within the first epoch. We thank the reviewer for this insightful suggestion and will work on it as a direction for future work.

Q2: The proposed method seems to have many hyper parameters to tune, such as $\beta$, $\alpha$, $p_{easy}$ and $p_{hard}$, is there any principles or guidelines to tune these parameters in broader scenarios, and how is the performance so as the training efficiency affected by different hyper parameters?

A2: While our method introduces several hyperparameters, we argue that these are not manually tuned for each task and do not pose a significant burden in practice:

1. The exploration bounds $p_{\text{easy}}$ and $p_{\text{hard}}$ are automatically adjusted during training and do not require manual specification.

2. The target zero-variance percentage $\alpha$ is discussed in Appendix F.1. We show that GRESO is robust across a wide range of values, and as long as $\alpha$ is sufficiently high, GRESO shows pretty good performance.

3. The exploration adjustment rate $\beta$ is a parameter inherited from GRPO; we follow prior work and simply set $\beta = 0$, which works well in our setting.

More importantly, we use the same default configuration across all experiments and consistently observe efficiency gains, indicating that GRESO works robustly without the need for extensive hyperparameter tuning.

---

Thanks again for the attentive reading of the manuscript and constructive feedback. We will incorporate these changes into our final version. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.

---

[1] Yu, Qiying, et al. "Dapo: An open-source llm reinforcement learning system at scale." arXiv preprint arXiv:2503.14476 (2025).

[2] Yue, Yu, et al. "Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks." arXiv preprint arXiv:2504.05118 (2025).

[3] Fu, Wei, et al. "AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning." arXiv preprint arXiv:2505.24298 (2025).

[4] Wang, Shenzhi, et al. "Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning." arXiv preprint arXiv:2506.01939 (2025).

Response to Reviewer 2 (KuoT)

Thank you very much for your thoughtful review and constructive suggestions. We are glad that you found our presentation clear and our solution practical, with detailed implementation. We appreciate the opportunity to address the points you raised and hope you will consider raising your score in light of our response.

---

Q1: The observation that zero-variance samples are not very helpful for the LLM finetuning is not novel. For example, [1] also utilized similar mechanism to filter the dataset.

A1: Thanks for bringing this for discussion. As acknowledged in the paper, this observation has been discussed in prior work[1,4], and we do not claim it as a contribution in our paper. Our key contribution lies in how to identify zero-variance samples prior to performing rollouts—a nontrivial challenge that, to the best of our knowledge, has not been addressed in prior work, and our work is the first method to provide a practical and efficient solution to this problem.

Thank you for highlighting this point—we will further make this distinction clearer in the revised version.

Q2: Temporal correlation of sample utility is well-known in curriculum learning and NLP tasks [2,3]. GRESO applies this concept to RL rollouts but without significant algorithmic innovation.

A2: While we agree with the reviewer that leveraging temporal correlation in sample utility is a well-established principle in curriculum learning and active learning, we respectfully argue that our paper tackles a fundamentally different and underexplored problem for RLVR for LLMs, where zero-variance prompts emerge as a unique challenge, which was discussed by recent work [1,4].

While similar temporal correlation principles have been explored in areas such as curriculum learning, how to model and leverage temporal signals to detect and filter zero-variance prompts prior to rollout remains underexplored and non-trivial. For instance, unlike prior works [2,3] that operate on static datasets, RLVR involves online data generation, where the training data can vary significantly across iterations, making it non-trivial to apply existing approaches directly.

To the best of our knowledge, our paper is the first to propose a practical and effective solution to filter out zero-variance prompts in RLVR, by tackling the following key challenges:

1. How to detect and filter uninformative prompts before performing rollouts (i.e., without access to the full training data)?

2. How to perform such detection and filtering with minimal overhead?

3. How to balance exploration and efficiency, ensuring that potentially useful prompts can be revisited in the future training stage?

Each of these challenges is non-trivial in the RLVR setting. Our proposed method, GRESO, offers a lightweight mechanism that effectively addresses all three and demonstrates strong empirical performance across multiple benchmarks and RL algorithms.

However, we agree with the reviewer that a discussion of these works would strengthen the paper. We will conduct a comprehensive discussion of the related prior works, including [2,3], on curriculum learning and active learning in the revised version to better clarify the connections and differences.

Q3: This paper has a limited scope to math reasoning and small models (≤7B). Modern LLM RL (e.g., 70B+ models) that may face distinct challenges (e.g., prompt complexity, reward sparsity) are not addressed here. Speedup gains may not generalize. There is no formal mathematical analysis of how skipping prompts affects convergence, reward shaping, or generalization beyond test accuracy.

A3:

Training on Code Tasks: To further evaluate the generalizability of GRESO, we applied it to the code generation domain. Specifically, we trained DeepSeek-R1-Distill-Qwen-1.5B on the code_contests dataset and evaluated the models on the LiveCodeBench benchmark. The results are as follows (we report the best accuracy and the corresponding number of rollouts at that iteration):

We trained all the models until convergence. GRESO achieves comparable accuracy to DS with fewer rollouts, and GRPO fails to match the final accuracy of GRESO and DS.

These results suggest that GRESO is also effective in other reasoning domains such as code generation.

Scale to 32B model. Thank you for the suggestion. We have already initiated experiments on 14B and 32B-scale models to evaluate the scalability of GRESO. Due to the high computational cost, these runs are still in progress, but we report interim results below (average accuracy across the 6 benchmarks used in the paper):

These preliminary results indicate that GRESO achieves comparable accuracy to Dynamic Sampling while using fewer rollouts. We are continuing these experiments and will update the results during the discussion phase. We will also ensure that the complete results, especially for the 32B model, are included in the final version of the paper.

Besides, we agree that a formal theoretical understanding of the mechanism of our method is an important and open-ended research question. The primary goal of this work is to develop a practical and effective method for improving rollout efficiency. GRESO demonstrates consistent speed-up across multiple models and domains, which we believe provides strong evidence of its effectiveness in practice. We hope that our method and findings can serve as a foundation for future theoretical investigations into the dynamics of selective rollout in RL-based LLM training.

---

Thanks for the attentive reading of the manuscript and constructive feedback. We will incorporate these changes into our final version. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.

---

[1] Muennighoff, Niklas, et al. "s1: Simple test-time scaling." arXiv preprint arXiv:2501.19393 (2025).

[2] Zhou, Tianyi, Shengjie Wang, and Jeff Bilmes. "Curriculum learning by optimizing learning dynamics." International Conference on Artificial Intelligence and Statistics. PMLR, 2021.

[3] Bejan, Irina, Artem Sokolov, and Katja Filippova. "Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.

[4] Yu, Qiying, et al. "Dapo: An open-source llm reinforcement learning system at scale." arXiv preprint arXiv:2503.14476 (2025).

Thank you for your response. I still have the following concerns:

• In the leaderboard of the LiveCodeBench benchmark, it utlizes pass@1 as the metric. Could you please also report the performance comparison with this metric?

• From your first table, it is difficult to infer that your proposed method is better than GRPO since we cannot tell if the GRPO algorithm has converged or not.

• Additionally, if collecting more rollouts (i.e., with the same number of rollouts as DS), will your algorithm outperform DS when the model is 14B/32B?

• It is better to include mean and std for the reported results.

Response to Reviewer 1 (3M88)

Thank you very much for your thoughtful review and constructive suggestions. We are glad that the reviewer found our work addresses an important challenge in RL for LLMs, and the dynamic control of exploration probability is insightful. We appreciate the opportunity to address the points you have raised.

Q1: The context of Deepseek model is 8k, which can introduce a high truncation rate.

A1: Thank you for pointing this out. We are currently running the 32k training and will make sure to put the results in the final version. We also note that, despite using truncation, we use the same setting for baselines and our method to guarantee the fairness of the comparison in the evaluation.

In the meantime, we would like to respectfully argue that such a truncation setting is also commonly used in the previous works [1,2,3,4] to improve training efficiency, as longer context lengths significantly increase memory usage and wall-clock time.

Importantly, we view the context length constraints can be treated as a special reward constraint to encourage models to reason in a limited context length (e.g., 8k). During training, we observed that the initial truncation rate was around 50%, but the model quickly adapted—dropping to below 3% after approximately 200 iterations—while continuing to improve in accuracy.

We appreciate the reviewer’s concern and will include a more detailed discussion in the final version. We will also include the 32k results in the final version.

Q2: An ablation study on zero-variance target can strengthen the paper.

A2: Thank you for the suggestion. We have conducted an ablation study on the targeted zero-variance percentage, as presented in Appendix F.1, showing consistent improvement across a wide range of zero-variance targets. Our results also show that when the percentage is sufficiently high (e.g., 25% used in our experiments), GRESO matches the final accuracy of dynamic sampling. Importantly, all our RL training experiments adopt the same target value, indicating that this parameter is non-sensitive and easy to select in practice.

Q3: Ablation study of adaptive batch size should also consider actual rollout time savings.

A3: We appreciate the reviewer’s insightful observation. It is indeed true that using a batch size of 1 would minimize the number of rollouts per step, but not the overall rollout time. This is because inference latency for LLMs scales nonlinearly with batch size—small batch sizes are not time-efficient due to under-utilized resources, while large batch sizes can significantly reduce per-sample cost.

To illustrate this, we conducted an experiment using Qwen2.5-Math-1.5B on 4 H100 to measure the actual rollout time at different batch sizes. The results are shown below:

We observe that when the batch size is small (e.g., ≤64), increasing the batch size yields per-sample rollout little time saving. However, beyond a certain point, the rollout time increases almost linearly with batch size. Therefore, by using ABS to reduce the batch size in this regime, we can significantly reduce overall rollout time without sacrificing efficiency.

In our training setup, we set the default rollout batch size to 384. When ABS reduces the batch size adaptively (e.g., from 384 to 128), it leads to a substantial reduction in actual rollout time. This is the core motivation behind ABS, and we will clarify this point in the final version of the paper.

Q4.1: Figure 6: the zero-variance ratio of easy problems fails to converge to the target, and the corresponding base exploration probability is bounded excessively high. This may result from too large a Δp; if so, the method should be revised to handle this case.

A4.1: Thank you for pointing this out. The reason the base exploration probability appears excessively high is that we enforce a minimum exploration rate (5%) to ensure a non-zero level of exploration. This lower bound helps prevent the model from entirely ignoring potentially informative examples. While this constraint may prevent the zero-variance ratio from fully converging to the target, it guarantees a certain level of exploration throughout training. Importantly, even without fully reaching the target ratio, our method still eliminates a significant amount of unnecessary rollouts and achieves substantial speedup. We will revise the manuscript to clarify this design choice and avoid potential confusion.

Q4.2: In Figure 5(d), the rollout per step for GRESO shows large spikes after step 600, which seems abnormal given the continuous adaptation of batch size.

A4.2: In the rollout stage, we enforce a minimum rollout batch size of 64, as small-batch LLM inference does not yield significant differences in wall-clock time (as discussed in Q3). The spikes observed after step 600 in Figure 5(d) are caused by GRESO triggering a second round of rollouts, during which an additional 64 × 8 responses are sampled. We will clarify this behavior in the revised version to avoid confusion.

Q4.3: The DS/GRESO rollout ratio of DM-Qwen2.5-1.5B is smaller than in Figure 5 (a). Does this indicate a performance drop when training steps are fixed? Including a step-to-accuracy curve would help address this concern.

A4.3: The rollout ratio in Figure 5(a) reflects the per-step ratio, while the cumulative ratio is much closer to the values reported in Table 1. In Table 5, we report the best-performing checkpoints during training, evaluated every 50 iterations as described in Appendix D. We observed that the final iteration is not always the best one. However, after convergence, the accuracy tends to fluctuate slightly rather than showing a clear performance drop. We will include a step-to-accuracy curve in the revised version of our paper.

---

Thanks again for the attentive reading of the manuscript and constructive feedback. We will incorporate these changes into our final version. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions!

---

[1] Wang, Yiping, et al. "Reinforcement learning for reasoning in large language models with one training example." arXiv preprint arXiv:2504.20571 (2025).

[2] Liu, Mingjie, et al. "Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models." arXiv preprint arXiv:2505.24864 (2025).

[3] Cui, Ganqu, et al. "The entropy mechanism of reinforcement learning for reasoning language models." arXiv preprint arXiv:2505.22617 (2025).

[4] Cheng, Daixuan, et al. "Reasoning with exploration: An entropy perspective." arXiv preprint arXiv:2506.14758 (2025).