Thank you for the feedback and acknowledgement of our work. We hope the following responses could clarify your concerns.

1. However, the ablation for each component is not studied.

We did conduct careful small scale ablation studies on our alternations of the GRPO pipeline, including decoupling hyperparameters, adding reference model (and optimizer reset) hard reset. We apologize for not including such experiment details.

We will be providing an additional detailed technical report alongside the final version of the paper. Unfortunately the conference review process prohibits sharing additional documents, thus we are only able to provide brief text descriptions on the ablation studies and key observations as follows:

1.1 Rollout Sampling Temperature

Ablations with temperatures {0.6, 1.2} show that low temperature (0.6) causes early instability, reduced diversity, and overexploitation, hurting reward and validation. High temperature (1.2) yields steadier improvement, better generalization, and avoids mode collapse even in late-stage training, so we use 1.2 throughout.

1.2 Decoupled Clipping

We vary ϵlow {0.1, 0.2} and ϵhigh​ {0.2, 0.3, 0.4}; ϵlow=0.2 better handles negative advantage, and ϵhigh=0.4 prevents entropy collapse, giving best validation. Smaller ϵlow increases entropy but not learning, so we adopt (0.2, 0.4).

1.3 Dynamic Sampling

We further verified that dynamic sampling led to faster improvements than static sampling by filtering out prompts with zero advantage. This increases reward signal density in each batch, thereby improving the overall sample efficiency of training.

1.4 Resetting Reference Policy

We observe that extending training with the same setting and training hyperparameters does not always lead to consistent improvements. We observed that by simply extending the training or Run1 (this is the DAPO baseline) without reference policy reset, it encounters a sharp drop in validation scores on the Codeforces benchmark. We were not able to scale RL training beyond 600 steps with DAPO, and in contrast resetting the reference policy allowed us to scale training to beyond 2k steps.

1.5 Mitigating Entropy Collapse

We conducted a series of ablation studies evaluating the training dynamics under different strategies to mitigate entropy collapse. We investigated the following settings:

• GRPO-KL=0: We employ standard GRPO training without KL divergence penalty or entropy bonus.

• GRPO-KL=1e-4: We apply a small KL regularization term 1e-4 to control the divergence between the online policy and a reference policy. This helps prevent the model from drifting too far and encourages continued exploration.

• DAPO: This is the DAPO baseline with decoupled clipping and dynamic sampling. No KL divergence penalty or entropy bonus.

We observe that GRPO-KL=0 suffers from entropy collapse and the validation score stops improving after the entropy degrades to ~0 in ~500 steps. In contrast, DAPO and KL penalty both can mitigate entropy collapse as the validation scores continue to improve. Based on these findings, we adopt a combination of DAPO and a KL penalty in our final training setup.

2. It is important to include another family of base models for conducting ProRL and with a larger model larger than 1.5B.

We conduct further experiments by applying our training recipe to a larger model and different base model. We use Llama-3.1-Nemotron-Nano-8B-v1, which is a derivative based on llama-3.1-8B-Instruct, further finetuned on diverse reasoning tasks. Due to time constraints and compute limitations, we only scale training to ~1.4k steps. We provide detailed comparison with the base model across tasks as follows

Note: The above reported results on Code only consists of {apps, condecontests, codeforces, taco} and report the obtained continuous reward. All other benchmarks are the same as in the paper.

3. The main results did not compare other training baselines, such as GRPO and DAPO, on the curated training datasets.

We apologize again for not including detailed ablation studies in the original manuscript. We hope our response to the 1st question could provide confidence in our results when comparing against other training baselines such as GRPO and DAPO. In our ablation studies, we found GRPO performance to saturate within ~600 steps, largely due to entropy collapse as the model loses the capability to explore. We also found DAPO without reference policy reset to have training stability issues, in having some validation scores degrading ~500 steps.

As requested from Reviewer TRB6, we further conducted ablation studies comparing with other finetuning techniques such as SFT and DPO. We decided to focus on tasks within the Reasoning Gym suite and used a stronger base model DeepSeek-R1-Distilled-Qwen-32B model to generate reasoning traces and used verifiers to filter negative samples. We constructed a finetuning dataset of ~200k verified responses. We further collected additional negative responses across the same dataset, constructing a preference dataset of ~200k entries for DPO finetuning. We conduct SFT and DPO for 2 epochs using DeepSeek-R1-Distilled-Qwen-1.5B as the base model. We provide comparisons with these additional baselines with our ProRL-1.5B model on the Reasoning Gym tasks as follows:

4. It can also be beneficial to provide an ablation on the training data mixture.

In regards to training data mixture, we conduct a small scale experiment with the following two data mixtures: Math-Only, consisting of only the math dataset; and Math-Code-STEM-Reasoning, consisting of the math, code, stem reasoning and logic puzzle (Reasoning Gym) data. We only train for ~600 steps with the comparison as follows:

Note: The above reported results on Code only consists of pass@1 for {apps, condecontests, codeforces, taco}.

As shown in the above, training only on math data does bring some level of transfer to other tasks but not as significant as directly including similar tasks in training. This highlights the importance of training on a diverse mixture of data.

Thank you for the feedback and acknowledgement of our work. We hope the following responses could clarify your concerns.

1. I wonder whether the authors have explored what happens when starting from a base instruction-tuned model like Llama or Qwen.

Thank you for the question. We primarily wanted to experiment whether RL can bring improvements to already strong reasoning models. We thus leave research on scaling RL from non-reasoning models as an open question for future research.

We conduct further experiments by applying our training recipe to a larger model and different base model. We use Llama-3.1-Nemotron-Nano-8B-v1, which is a derivative based on llama-3.1-8B-Instruct, further finetuned on diverse reasoning tasks. Due to time constraints and compute limitations, we only scale training to ~1.4k steps. We provide detailed comparison with the base model across tasks as follows

Note: The above reported results on Code only consists of {apps, condecontests, codeforces, taco} and report the obtained continuous reward. All other benchmarks are the same as in the paper.

2. It is not clear to me how the authors determine when the model stagnates. Based on Section 3.3, this seems to be a manual process and therefore might be very model-specific. I think it would be useful to report this point in the limitations or, in general, it would be useful to be explicit about the limitations of this training regime.

We determine stagnation through manual inspection guided by quantitative signals. Specifically, we reset the reference model when (i) the KL loss exceeds a threshold (e.g., 0.2), indicating excessive policy drift, or (ii) validation performance shows a consistent downward trend. In such cases, we resume from the most recent checkpoint with the best validation score. We will explicitly note in the limitations section that this process involves manual intervention and may be model-specific.

3. I was wondering how the solution proposed by this paper compares with current real-world implementations of GRPO such as TRL (from Huggingface).

I believe both the work mentioned (Open-Reasoner-Zero and DAPO) mainly experiment with using pretraining checkpoints (no instructional finetuning) as the base model. In our case, we select already strong reasoning models as the base model. In the case of starting from a weak model, removing KL penalty could facilitate the model learning as this additional penalty term constrains the policy term potentially limiting exploration. In our case, our ablation studies still found small KL divergence to be beneficial when starting from a strong base model, since it mitigates the entropy collapse issue and enables scaling training to thousands of steps. The reference policy resets ensures stability, and also allows the policy model also to gradually diverge from the base model.

Most popular RL frameworks (such as TRL and verl) still keep KL divergence penalty coefficient as a tunable parameter, allowing easy application of our proposed method.

4. Did you notice any specific difference in performance between tasks that have binary or continuous rewards?

Thank you for the great question! We do notice differences on tasks such as coding with binary and continuous rewards. Using continuous reward during initial training stages improves the model learning since these partial rewards help guide weaker models to achieve some level of competence. Furthermore, since dynamic sampling filters instances with pass rate of 0/1, continuous reward would improve efficiency as instances with intermediate rewards still have non-zero advantage. However, we also observe that during the late stages, switching to binary rewards increases the pass@1 score (but lowers the continuous reward scores) of the model performance. Nonetheless we keep tasks to continuous reward as to keep consistency during training.

Thank you for the feedback and acknowledgement of our work. We hope the following responses could clarify your concerns.

1. No ablation studies.

We did conduct careful small scale ablation studies on our alternations of the GRPO pipeline, including decoupling hyperparameters, adding reference model (and optimizer reset) hard reset. We apologize for not including such experiment details.

We will be providing an additional detailed technical report alongside the final version of the paper. Unfortunately the conference review process prohibits sharing additional documents, thus we are only able to provide brief text descriptions on the ablation studies and key observations as follows:

1.1 Rollout Sampling Temperature

Ablations with temperatures {0.6, 1.2} show that low temperature (0.6) causes early instability, reduced diversity, and overexploitation, hurting reward and validation. High temperature (1.2) yields steadier improvement, better generalization, and avoids mode collapse even in late-stage training, so we use 1.2 throughout.

1.2 Decoupled Clipping

We vary ϵlow {0.1, 0.2} and ϵhigh​ {0.2, 0.3, 0.4}; ϵlow=0.2 better handles negative advantage, and ϵhigh=0.4 prevents entropy collapse, giving best validation. Smaller ϵlow increases entropy but not learning, so we adopt (0.2, 0.4).

1.3 Dynamic Sampling

We further verified that dynamic sampling led to faster improvements than static sampling by filtering out prompts with zero advantage. This increases reward signal density in each batch, thereby improving the overall sample efficiency of training.

1.4 Resetting Reference Policy

We observe that extending training with the same setting and training hyperparameters does not always lead to consistent improvements. We observed that by simply extending the training or Run1 (this is the DAPO baseline) without reference policy reset, it encounters a sharp drop in validation scores on the Codeforces benchmark. We were not able to scale RL training beyond 600 steps with DAPO, and in contrast resetting the reference policy allowed us to scale training to beyond 2k steps.

1.5 Mitigating Entropy Collapse

We conducted a series of ablation studies evaluating the training dynamics under different strategies to mitigate entropy collapse. We investigated the following settings:

• GRPO-KL=0: We employ standard GRPO training without KL divergence penalty or entropy bonus.

• GRPO-KL=1e-4: We apply a small KL regularization term 1e-4 to control the divergence between the online policy and a reference policy. This helps prevent the model from drifting too far and encourages continued exploration.

• DAPO: This is the DAPO baseline with decoupled clipping and dynamic sampling. No KL divergence penalty or entropy bonus.

We observe that GRPO-KL=0 suffers from entropy collapse and the validation score stops improving after the entropy degrades to ~0 in ~500 steps. In contrast, DAPO and KL penalty both can mitigate entropy collapse as the validation scores continue to improve. Based on these findings, we adopt a combination of DAPO and a KL penalty in our final training setup.

2. Why stop at 2000 steps.

We stopped at 2k steps largely due to resource limitations and time constraints. We were able to further scale training for an additional 1k steps to a total 3k steps. We compare the new model with the original 2k model in our paper as follows:

As shown above, prolonged RL continues to bring improved performance across most tasks. Since we largely observe log-linear scaling, linear growth in the performance translates to exponential training computation. Furthermore data might be another limitation, as dynamic sampling will filter easy tasks (with pass rate of 1). We also provided a Limitations section in the Appendix.

3. How do you know if your task is going to benefit from extended RL? Would you need task specific stopping criteria?

In our experimental observations, all five tasks are generally suitable for ProRL. However, some tasks—such as those related to Code—show more noticeable and consistent improvement, while tasks in the MATH category improve more slowly. We do not impose a stopping criteria, as dynamic sampling removes too easy and too difficult problems from training. We do agree that having a stopping criteria (such as if the validation scores reach beyond a certain threshold) would greatly improve the efficiency of dynamic sampling.

4. DeepScaler, DeepCoder in Tables 1 and 2 are not explained.

We included text descriptions under "Comparison with Domain-Specialized Mdoels”: “We compare the performance of NemotronResearch-Reasoning-Qwen-1.5B with two domain-specialized baselines: DeepScaleR-1.5B [3], tailored for mathematical reasoning, and DeepCoder-1.5B [7], focused on competitive programming tasks.” We will also include the related citations in the Table for clarity.

5. I suggest you use the same colours for the models in both Figure 5 and Figure 6 for easing understanding

Thank you for pointing this out. We will keep colours of models consistent for Figure 5 and 6 in the final version.

Thank you for your review and suggestions. We hope the following responses could clarify your concerns.

1. Additional baseline request such as SFT or DPO

Thank you for the suggestion. Our main contribution is to provide empirical analysis in demonstrating the benefits of prolonged reinforcement learning in expanding the reasoning boundary and uncovering novel reasoning strategies inaccessible to the base model. Such results are in fact not as straightforward, especially works targeting RLVR, since numerous concurrent works have argued otherwise. We hope our paper can bring forth more discussion within the research community on the benefits (or lack thereof) of prolonged RL on LLMs.

We do agree that providing additional baselines would greatly strengthen our argument on the increased benefit of scaling Reinforcement Learning. Since the concern seems to be particularly centered on novel tasks outside the capabilities of the base model, we decide to focus on tasks within the Reasoning Gym suite. As per your recommendations, we used a stronger base model DeepSeek-R1-Distilled-Qwen-32B model to generate reasoning traces and used verifiers to filter negative samples. We constructed a finetuning dataset of ~200k verified responses. We further collected additional negative responses across the same dataset, constructing a preference dataset of ~200k entries for DPO finetuning. We conduct SFT and DPO for ~2 epochs using DeepSeek-R1-Distilled-Qwen-1.5B as the base model. We will make sure to include these additional baselines in our paper and details on the training setups etc. We provide only the following results for brevity:

2. No ablation study is presented for the proposed approach.

We did conduct careful small scale ablation studies on our alternations of the GRPO pipeline, including decoupling hyperparameters, adding reference model (and optimizer reset) hard reset. We apologize for not including such experiment details.

We will be providing an additional detailed technical report alongside the final version of the paper. Unfortunately the conference review process prohibits sharing additional documents or figures, thus we are only able to provide brief text descriptions on the ablation studies and key observations as follows:

2.1 Rollout Sampling Temperature

Ablations with temperatures {0.6, 1.2} show that low temperature (0.6) causes early instability, reduced diversity, and overexploitation, hurting reward and validation. High temperature (1.2) yields steadier improvement, better generalization, and avoids mode collapse even in late-stage training, so we use 1.2 throughout.

2.2 Decoupled Clipping

We vary ϵlow {0.1, 0.2} and ϵhigh​ {0.2, 0.3, 0.4}; ϵlow=0.2 better handles negative advantage, and ϵhigh=0.4 prevents entropy collapse, giving best validation. Smaller ϵlow increases entropy but not learning, so we adopt (0.2, 0.4).

2.3 Dynamic Sampling

We further verified that dynamic sampling led to faster improvements than static sampling by filtering out prompts with zero advantage. This increases reward signal density in each batch, thereby improving the overall sample efficiency of training.

2.4 Resetting Reference Policy

We observe that extending training with the same setting and training hyperparameters does not always lead to consistent improvements. We observed that by simply extending the training or Run1 (this is the DAPO baseline) without reference policy reset, it encounters a sharp drop in validation scores on the Codeforces benchmark. We were not able to scale RL training beyond 600 steps with DAPO, and in contrast resetting the reference policy allowed us to scale training to beyond 2k steps.

2.5 Mitigating Entropy Collapse

We conducted a series of ablation studies evaluating the training dynamics under different strategies to mitigate entropy collapse. We investigated the following settings:

• GRPO-KL=0: We employ standard GRPO training without KL divergence penalty or entropy bonus.

• GRPO-KL=1e-4: We apply a small KL regularization term 1e-4 to control the divergence between the online policy and a reference policy. This helps prevent the model from drifting too far and encourages continued exploration.

• DAPO: This is the DAPO baseline with decoupled clipping and dynamic sampling. No KL divergence penalty or entropy bonus.

We observe that GRPO-KL=0 suffers from entropy collapse and the validation score stops improving after the entropy degrades to ~0 in ~500 steps. In contrast, DAPO and KL penalty both can mitigate entropy collapse as the validation scores continue to improve. Based on these findings, we adopt a combination of DAPO and a KL penalty in our final training setup.

3. No analysis on the various model sizes.

We conduct further experiments by applying our training recipe to a larger model and different base model. We use Llama-3.1-Nemotron-Nano-8B-v1, which is a derivative based on llama-3.1-8B-Instruct, further finetuned on diverse reasoning tasks. Due to time constraints and compute limitations, we only scale training to ~1.4k steps. We provide detailed comparison with the base model across tasks as follows

Note: The above reported results on Code only consists of {apps, condecontests, codeforces, taco} and report the obtained continuous reward. All other benchmarks are the same as in the paper.

4. Further scaled training results

We were able to further scale training for an additional 1k steps to a total 3k steps. We compare the new model with the original 2k model in our paper as follows:

As shown above, prolonged RL continues to bring improved performance across most tasks, with only GPQA as an exception.

5. Font sizes should be increased for the most of the figures

We apologize for the inconvenience and will take this into consideration when revising the final paper.

6. Figure 1 (middle) demonstrates that base model produces some amount of outliers which score much higher in terms of creativity compared to samples after RL finetuning. What are those outliers? Are those just random garbage? And do you have any hypothesis why we don't see those for RL model?

Thanks for the great observation! The base model sometimes produces outliers with much higher creativity indices, likely because these tasks are highly out-of-distribution—unseen during the base model’s training and unfamiliar to it, such as the dice task in Reasoning Gym, which has near-zero pass@1 accuracy. After RL training, the model learns to handle these tasks (e.g., the dice task's pass@1 accuracy improves to 39.9%), making them more in-domain for the model.

7. Is any weight decay used? AdamW is used as an optimizer, however no weight decay parameter is mentioned. I would also recommend to report all the hyperparameters in the form of a table in Appendix and not as parts of the text.

We use a weight decay of 0.01. We will include detailed hyperparameter configurations in the Appendix for our final version.