Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay

We sincerely thank Reviewer NvgW for the thoughtful and constructive feedback. In response to the suggestions, we have conducted additional experiments during the rebuttal period on (1) adding standard errors for 3 LLM-dataset combinations to support statistical significance, and (2) evaluating our data-efficient RL training pipeline on a new non-math domain.

Below, we address the questions and concerns point by point.

---

W1: No statistical significance tests due to high computational costs

Re.: Thank you for the suggestion. We have added error bars for 3 LLM-dataset combinations by independently repeating both DOTS+RR and original GRPO three times. The results are shown in the table below and confirm that the improvements of our method over original GRPO are statistically significant, with nearly all p-values below 0.05.

Specifically, we report the average accuracy (mean ± standard error) based on three independent runs. We also perform statistical testing using one-sided Welch’s t-tests. Due to the rebuttal policy, we cannot include plots, so we report each metric at 0.5-hour intervals in the tables below. Since DOTS+RR completes training earlier than original GRPO, we report results only up to the DOTS+RR completion point for comparison purposes.

These additional runs represent our best effort within the limited rebuttal period and available compute resources. In the final version, we will extend this analysis to the remaining 3 combinations and increase the number of independent runs.

Table A: Average accuracy (mean ± standard error) of our method and original GRPO on Qwen2.5-Math-1.5B with DeepScaleR dataset.

Table B: Average accuracy (mean ± standard error) of our method and original GRPO on Qwen2.5-Math-1.5B with ORZ dataset.

Table C: Average accuracy (mean ± standard error) of our method and original GRPO on Qwen2.5-3B with DeepMath dataset.

Q2 & L1: Choice of mathematical reasoning dataset and generalization to non-math domains

Re.: We focused on mathematical reasoning datasets because math is a representative and important use case for LLM reasoning, and prior work on both algorithmic [1,2] and data-centric [3] improvements in LLM RL fine-tuning commonly adopts math datasets for training and evaluation.

That said, inspired by your suggestion, we conduct an additional experiment in a non-math domain. Specifically, we apply our full pipeline, including both RL training and evaluation, to the science domain using the curated SCP-25K dataset [4], which contains advanced physics, chemistry, and biology questions. We use the Qwen2.5-3B model and train a new adaptive difficulty predictor. All other RL settings are kept unchanged.

We evaluate performance on the science subsets of MMLU, and the results (reported in the table below) show that our method continues to significantly improve RL data efficiency in this non-math domain, supporting its broader applicability.

Table D: Average accuracy of DOTS+RR (ours) and original GRPO on Qwen2.5-3B with science subsets of MMLU.

W2: Generalization ability of the MLP adapter

Re.: We do not require the MLP adapter to transfer across domains. Instead, for each domain, we can train a separate adapter, which is both justified and practical.

• Justification: This adapter is trained specifically for the data domain used in the subsequent LLM RL fine-tuning, which is known in advance. If data from a particular domain will be used during RL fine-tuning, it is naturally included in the adapter training data. Therefore, there is no need to generalize to out-of-domain data. This setting is fundamentally different from standard supervised learning, where models are expected to perform well on unseen distributions.

• Practicality: The training cost is low, taking about 11.5 minutes on a single NVIDIA A100 GPU. This makes per-domain retraining entirely manageable in practice.

Q1: Generalization of Theorem 1 to complex rewards

Re.: Theorem 1 uses binary rewards for simplicity, a standard choice in recent literature on RLVR (Reinforcement Learning with Verifiable Rewards). Its key step involves computing the second central moment of the group-normalized rewards $\mathbb{E}[\hat{A}_i^2]$, which can be easily recomputed under more complex reward formulations.

For example, if the reward consists of two independent components such as a correctness reward $c_i \sim \text{Bern}(\alpha)$ and a format reward $f_i \sim \text{Bern}(\beta)$, with $r_i = c_i + f_i$, then we have $\mathbb{E}[\hat{A}_i^2] = (\alpha(1-\alpha) + \beta(1-\beta))(1 - 1/G)$, which is maximized when both $\alpha = 0.5$ and $\beta = 0.5$. This shows that our insight generalizes naturally to multi-component reward settings.

L2: Potential bias due to difficulty-based selection

Re.: If the concern refers to the possibility that our difficulty prediction framework consistently favors a particular type of question (e.g., from certain domains or with certain formats), we believe this is not a significant issue in our setup.

Since our method selects questions whose predicted adaptive difficulty under the current policy is close to 0.5, those that have been repeatedly selected and trained on will become easier and thus gradually fall outside the 0.5 region. As a result, they are less likely to be selected again, allowing other question types to enter the selection pool over time. This dynamic naturally promotes broader coverage during training.

We will include a relevant discussion in the revised version.

---

[1] Yu, Qiying, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai et al. "Dapo: An open-source llm reinforcement learning system at scale." arXiv preprint arXiv:2503.14476 (2025).

[2] Lin, Zhihang, Mingbao Lin, Yuan Xie, and Rongrong Ji. "Cppo: Accelerating the training of group relative policy optimization-based reasoning models." arXiv preprint arXiv:2503.22342 (2025).

[3] Li, Xuefeng, Haoyang Zou, and Pengfei Liu. "Limr: Less is more for rl scaling." arXiv preprint arXiv:2502.11886 (2025).

[4] Liu, Mingjie, et al. "Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models." arXiv preprint arXiv:2505.24864 (2025).

[5] Zeng, Thomas, Shuibai Zhang, Shutong Wu, Christian Classen, Daewon Chae, Ethan Ewer, Minjae Lee et al. "VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data." In Forty-second International Conference on Machine Learning.

[6] Namgoong, Hyuk, Jeesu Jung, Sangkeun Jung, and Yoonhyung Roh. "Exploring domain robust lightweight reward models based on router mechanism." arXiv preprint arXiv:2407.17546 (2024).

We sincerely thank Reviewer puy2 for the thoughtful and constructive feedback. In response to the suggestions, we have conducted additional experiments during the rebuttal period on (1) adding standard errors for 3 LLM-dataset combinations to support statistical significance, (2) extending the training horizon to observe convergence behavior, and (3) evaluating on the AIME benchmark.

Below, we address the questions and concerns point by point.

W1 & Q4 Part 1: No error bars

Re.: Thank you for the suggestion. We have added error bars for 3 LLM-dataset combinations by independently repeating both DOTS+RR and original GRPO three times. The results confirm that the improvements of our method over original GRPO are statistically significant, with nearly all p-values below 0.05.

Due to space limitations, detailed results are provided in our response to Reviewer NvgW, W1.

Q4 Part 2: Extended training to observe convergence behavior

Re.: Due to resource constraints, we initially limited all runs to 60 steps. Following your suggestion, we extended training to 100 steps for two settings: Qwen2.5-Math-1.5B+DeepScaleR and Qwen2.5-3B+DeepMath. The updated results are shown in the table below and remain consistent with our original conclusions.

Table A: Average accuracy (%) of DOTS+RR (Ours) and original GRPO over 100 training steps on Qwen2.5-Math-1.5B with DeepScaleR and Qwen2.5-3B with DeepMath datasets.

Q1 & Q2: Reuse of reference set rollouts

Re.: This is an insightful point. In our current experiments, rollouts from the reference set are not reused for training.

Inspired by your suggestion, we conducted additional experiments to assess the benefit of reusing reference rollouts for training when their predicted difficulty is near 0.5. Approximately 10% of the reference questions per step fall into this moderate-difficulty range (see Table B below). Reusing the rollouts associated with these questions in RL training can reduce overall rollout generation time by 4–12% per step, while maintaining the same final policy performance. We find this strategy effective and will adopt it in the revised version.

Relatedly, to clarify the overhead, in our experiments, data selection is conducted every 2 steps (stated in L185), so after amortization, only 128 (instead of 256) reference questions are needed to do rollouts per step.

Table B: Proportion of questions with adaptive difficulty=0.5 in reference sets at different training steps.

Q3 Part 1: Detailed time breakdown of the overall process

Re.: Thank you for your question. We provide a detailed breakdown of the time cost of DOTS+RR (Ours) on the Qwen2.5-Math-1.5B and Qwen2.5-3B models trained on DeepScaleR, averaged over 60 steps. The experiments are conducted on 8x NVIDIA L40S GPUs. The overall process consists of four components: (1) Reference rollout time, (2) Adaptive difficulty prediction time, (3) Selected question rollout time, and (4) Actor update time.

Below is the breakdown of the average time (in seconds) amortized per step. We highlight three key observations:

(1) The cost of adaptive difficulty prediction is negligible.

(2) The selected question rollout time is relatively short due to the use of replay in DOTS+RR.

(3) As mentioned in our response to Q1&Q2, reference rollouts whose predicted difficulty is close to 0.5 can be reused for training, thereby further reducing the cost of selected question rollouts.

Table C: Breakdown of the average time (in seconds) amortized per step of DOTS+RR. We note that data selection is performed once every two steps (as stated in L185), so reference rollouts are only generated at those steps, and the reported cost is amortized accordingly.

Q3 Part 2: Wall-clock efficiency of DOTS (without RR) vs. original GRPO

Re.: Yes, even when using DOTS without RR, the efficiency advantage in wall-clock time is still present.

Although DOTS (without RR) introduces a small additional overhead due to reference rollouts and difficulty prediction, it requires much fewer steps to reach the same final accuracy as the original GRPO. As a result, the overall training time is reduced.

For example, on the Qwen2.5-Math-1.5B + DeepScaleR setting, DOTS reaches the final accuracy of original GRPO 31.23% faster in terms of wall-clock time. More results can be found in the response to Q5 below.

Q5: Add training steps as x-axis in Figure 3 and GPU time in Figure 5

Re.: Thank you for the suggestion. Since figures are not permitted during the rebuttal, we provide the corresponding results in tabular form to address both of your concerns:

(1) Figure 3 → Now with training steps as x-axis (Table D):

(2) Figure 5 → Now with GPU time as x-axis (Table E).

Across both views (training steps and wall-clock time), DOTS & RR and DOTS consistently demonstrate strong performance, confirming the robustness of our advantage regardless of the presentation format. We will include the updated figures in the revised appendix.

Table D: Average accuracy of DOTS+RR (Ours) and original GRPO across various LLM-dataset combinations.

Table E: Accuracy comparison of DOTS+RR, DOTS (without RR), and original GRPO on Qwen2.5-Math-1.5B in the ablation study. The “Final” row refers to the last evaluation point for each run and the “Time to Achieve original GRPO Final Performance” row refers to the required time to match original GRPO final performance. DOTS can still reduce training time compared to original GRPO. DOTS+RR consistently achieves comparable performance with DOTS while saving the most training time.

Q6: AIME results exclusion

Re.: We initially excluded AIME results because AIME is known to cause instability in evaluation for smaller models due to high variance and limited data size (only 30 questions) [1].

To support this point empirically, we provide additional experimental results using various LLM-dataset combinations with the original GRPO. Specifically, we evaluated each of the AIME 24 questions 8 times and computed the average accuracy (avg@8) across training steps.

As shown in the following table, accuracy fluctuates heavily across steps, with no clear trend (results shown every 10 steps). This high variance across checkpoints makes it difficult to obtain reliable evaluation signals on AIME.

Table F: Accuracy of original GRPO on AIME 24.

As suggested, we will include AIME evaluation results for all models in the appendix for reference, and provide a detailed discussion on why we exclude AIME from the main analysis.

---

[1] Hochlehnert, Andreas, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. "A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility." arXiv preprint arXiv:2504.07086 (2025).

We sincerely thank Reviewer 1Wxg for the thoughtful and constructive feedback. In response to the suggestions, we have conducted additional experiments during the rebuttal period on (1) providing empirical evidence supporting our theoretical analysis assumptions, (2) including reference solutions when training the adaptive difficulty prediction framework, and (3) adding an oracle training baseline using full rollouts for every question.

Below, we address the questions and concerns point by point.

---

W1: Effectiveness of the adaptive prediction framework & correlation between estimated difficulty and ground-truth

Re.: Thank you for the thoughtful feedback. We agree that in mathematics, surface-level similarity does not always imply similar difficulty. This is precisely why we do not rely on off-the-shelf embeddings. Instead, we train an adapter to refine the embedding space to capture difficulty signals rather than surface form. As shown in Table 5 of the ablation study (L148–154, Appendix F.1), this leads to a substantial improvement in prediction alignment with ground-truth adaptive difficulty compared to off-the-shelf alternatives.

While the overall Pearson correlation ranges from 0.7 to 0.8, we emphasize that our goal is not to precisely predict the difficulty of every question. Rather, at each training step, we only need to identify B questions that are highly likely to be informative (those near 0.5 adaptive difficulty). Our method achieves this effectively: it significantly increases the ratio of effective questions (as shown in Figure 4 on Page 8), and selects about 30–40% more moderate-difficulty questions compared to random sampling, resulting in substantial data efficiency gains.

Q1: Incorporate reference solutions in the embeddings

Re.: Thank you for the suggestion! We conduct an additional experiment on the MATH dataset by including reference solutions when constructing the input embeddings. This results in a slight improvement in Pearson correlation, indicating that solutions can provide helpful additional signals beyond the problem statement. However, since reference solutions are not available for all datasets (e.g., DeepScaleR and ORZ), this approach may have limited applicability. We will include this discussion in the revised version.

Table: Comparison of average Pearson correlation ($\rho$) between predicted and ground-truth adaptive difficulties. Reported as mean ± standard deviation over 60 training steps.

W2: Independence assumptions in theoretical analysis

Re.: Thank you for your question. We agree that assuming full independence between policy gradients and rewards may be overly strong in practice. In the original proof, we adopted this assumption for simplicity. However, we now provide a more general and relaxed formulation without affecting our main conclusion.

We now expand the original derivation (L65, Appendix C) to include the covariance correction term $T_2$:

$\mathbb{E}[||g||^2] =\underbrace{\sum\_{i,j=1}^G\mathbb{E}[\hat{A}_i \hat{A}_j] \cdot \mathbb{E}[\nabla\_{\theta}\log \pi\_\theta(o_i \mid q)^\top\nabla\_\theta \log \pi\_\theta(o_j \mid q)]}\_{T_1}$ + $\underbrace{\sum\_{i,j=1}^G \mathrm{Cov} (\hat{A}_i\hat{A}_j,\nabla\_\theta\log \pi\_\theta(o_i \mid q)^\top \nabla\_\theta \log \pi\_\theta(o_j \mid q))}\_{T_2}$.

We denote the first term as the independence term and the second as the covariance correction term. To relax the independence assumption, we introduce a weak-dependence assumption:

$\frac{\left|T_2\right|}{T_1} \ll 1$.

Under this assumption, our original subsequent derivation and final conclusion remains unchanged, as the independence term continues to dominate. This confirms that our previous theoretical conclusion was not incorrect, but rather based on a simplified case that can now be justified under a broader condition.

• Empirical evidence: To support this relaxed assumption empirically, we compute the covariance-to-main ratio described above. We randomly sample 512 questions for each of two LLM-dataset combinations, generate 8 rollouts per question, and evaluate the ratio ($\frac{\left|T_2\right|}{T_1}$). The results are:

– Qwen2.5-Math-1.5B + MATH: 0.081 ± 0.0065

– Qwen2.5-3B + DeepMath: 0.081 ± 0.0051

These low ratios empirically validate the weak-dependence assumption.

We thank the reviewer again for raising this point. We will revise the theoretical section in the final version to explicitly incorporate the relaxed assumption and include the empirical results that support it.

Q2: Oracle training baseline with full-rollout difficulty estimates

Re.: Thank you for the insightful suggestion. We implemented such an oracle baseline by performing full rollouts on all questions every $\mu$ steps and selecting $\mu \times B$ questions, prioritizing those with adaptive difficulty near 0.5, for the next $\mu$ training steps. We set $\mu = 6$ to allow the oracle baseline to use approximately twice the computational budget of DOTS. To make the comparison fair, we also removed rollout replay from both methods to isolate the effect of difficulty-based selection.

As shown in the table below, this oracle baseline underperforms our DOTS method. This is likely because these difficulty scores become increasingly off-policy as training progresses. Increasing the frequency of full rollouts could mitigate this issue, but would incur even greater computational cost—recall that running full rollouts every 6 steps already doubles the training cost compared to DOTS.

In contrast, our adaptive difficulty prediction framework strikes a strong balance between on-policy difficulty estimation and computational efficiency: it generates on-policy difficulty predictions with minimal overhead, enabling it to better capture evolving training dynamics and deliver stronger performance.

Table: Average accuracy of different methods on Qwen2.5-Math-1.5B+DeepScaleR.

We sincerely thank Reviewer 9x3o for the thoughtful and constructive feedback. In response to the suggestions, we have conducted additional experiments during the rebuttal period on (1) providing empirical evidence supporting our theoretical analysis assumption and (2) increasing the reference set size from 256 to 512 to demonstrate that our current choice is sufficient.

Below, we address the questions and concerns point by point.

---

W1 & Q1:Clarification on innovation

Re.: We would like to clarify the novelty of our contributions. Prior work on difficulty-based curriculum learning either relies on external difficulty labels [1,2], which we show in §5.4 to be less effective than our method, or computes adaptive difficulty but requires full rollouts for every question [3], making them computationally impractical. In contrast, we propose a novel attention-based adaptive difficulty prediction framework that efficiently and accurately estimates adaptive difficulty without full rollouts. We also provide Proposition 1 to theoretically justify the benefit of selecting questions with difficulty near 0.5. Together, these innovations enable scalable and principled online data selection for LLM RL fine-tuning, which prior methods cannot achieve due to either limited effectiveness or prohibitive cost.

Regarding RR, while experience replay is common in off-policy RL, it has not, to our knowledge, been applied to LLM RL fine-tuning; we bridge this gap by adapting it with a modified GRPO loss to enable stable and efficient rollout reuse.

Finally, our method improves data efficiency of LLM RL fine-tuning from two complementary angles: DOTS reduces the number of steps needed, while RR lowers the cost per step. This unified approach to optimizing both axes has not been explored in prior work.

W2 & Q2: Independence assumptions in theoretical analysis

Re.: Thank you for your question. It involves two parts: (1) the i.i.d. assumption on Bernoulli rewards and (2) the assumption that the policy gradient is independent of the reward. We address both below.

(1) i.i.d. reward assumption is reasonable and standard

This assumption holds in our setting because given the same prompt, each response is randomly sampled from the same policy, and the reward is then computed by a deterministic rule-based verifier. As a result, the only source of randomness lies in the policy’s stochastic generation process, and the rewards are i.i.d. from the same Bernoulli (p) distribution. This assumption is commonly used in prior work [4,5].

(2) Independence between gradient and reward can be relaxed without affecting the main conclusion & empirical evidence to support this relaxed assumption

We agree that assuming full independence between policy gradients and rewards may be overly strong in practice. In the original proof, we adopted this assumption for simplicity. However, we now provide a more general and relaxed formulation without affecting our main conclusion.

We now expand the original derivation (L65, Appendix C) to include the covariance correction term:

$\mathbb{E}[||g||^2] =\underbrace{\sum\_{i,j=1}^G\mathbb{E}[\hat{A}_i \hat{A}_j] \cdot \mathbb{E}[\nabla\_{\theta}\log \pi\_\theta(o_i \mid q)^\top\nabla\_\theta \log \pi\_\theta(o_j \mid q)]}\_{T_1}$ + $\underbrace{\sum\_{i,j=1}^G \mathrm{Cov} (\hat{A}_i\hat{A}_j,\nabla\_\theta\log \pi\_\theta(o_i \mid q)^\top \nabla\_\theta \log \pi\_\theta(o_j \mid q))}\_{T_2}$.

We denote the first term as the independence term and the second as the covariance correction term. To relax the independence assumption, we introduce a weak-dependence assumption:

$\frac{\left|T_2\right|}{T_1} \ll 1$.

Under this assumption, our original subsequent derivation and final conclusion remains unchanged, as the independence term continues to dominate. This confirms that our previous theoretical conclusion was not incorrect, but rather based on a simplified case that can now be justified under a broader condition.

• Empirical evidence: To support this relaxed assumption empirically, we compute the covariance-to-main ratio described above. We randomly sample 512 questions for each of two LLM-dataset combinations, generate 8 rollouts per question, and evaluate the ratio. The results are:

– Qwen2.5-Math-1.5B + MATH: 0.081 ± 0.0065

– Qwen2.5-3B + DeepMath: 0.081 ± 0.0051

These low ratios empirically validate the weak-dependence assumption.

We thank the reviewer again for raising this point. We will revise the theoretical section in the final version to explicitly incorporate the relaxed assumption and include the empirical results that support it.

W3 Part 1 & Q3: Implementation details of adaptive difficulty framework

Re.: The implementation details of the adaptive difficulty framework are provided in L81-95 in Appendix D.1. Specifically, the adapter is a GELU-activated MLP with three hidden layers, each containing 896 units and a dropout rate of 0.1. A LayerNorm is applied to the projection output to stabilize training.

W3 Part 2: The construction of training data for the adapter

The construction process is detailed in Appendix D.1 (L87–93). Each training instance consists of a query question, a reference set with known difficulty scores, and a ground-truth adaptive difficulty label. To build this dataset, we sample questions from the math datasets and use a fixed set of external LLMs (disjoint from the policy model) to generate responses and compute adaptive difficulty scores, which serve as supervision signals for training the adapter.

The adapter training is independent of the RL process, as the policy model trained during RL is excluded at this stage, and the adapter is trained before RL fine-tuning begins. This ensures that no information from the policy model is leaked.

W4 & Q4 Part 1: Potential failure case of adaptive prediction framework & Coverage of Reference Set

Re.: We agree that in mathematics, surface-level similarity does not always imply similar difficulty. This is precisely why we do not rely on off-the-shelf embeddings. Instead, we train an adapter to refine the embedding space to capture difficulty signals rather than surface form. As shown in Table 5 of the ablation study (L148–154, Appendix F.1), this leads to a substantial improvement in prediction alignment with ground-truth adaptive difficulty compared to off-the-shelf alternatives.

To demonstrate that a reference set size of 256 is sufficient, we conduct additional experiments increasing the size to 512. As shown in the table below, the improvement in adaptive difficulty prediction is marginal (typically 1–2%). This suggests that increasing the reference set beyond 256 offers diminishing returns in prediction performance while incurring higher computational cost.

Finally, we agree that dynamically adjusting the reference set size based on question types is an interesting direction. We leave this exploration to future work.

Table: Average Pearson correlation ($\rho$) between predicted and ground-truth adaptive difficulties under different reference set sizes.

Q4 Part 2: Generalization to non-math domains

Re.: We would greatly appreciate clarification on which aspect of generalization you are referring to:

(1) If the concern is about the adaptive difficulty prediction framework, we note that cross-domain generalization is not required in our setting. Instead, for each domain, we can train a separate MLP adapter, which is both justified and practical.

• Justification: This adapter is trained specifically for the data domain used in the subsequent LLM RL fine-tuning, which is known in advance. If data from a particular domain will be used during RL fine-tuning, it is naturally included in the adapter training data. Therefore, there is no need to generalize to out-of-domain data. This setting is fundamentally different from standard supervised learning, where models are expected to perform well on unseen distributions.

• Practicality: The training cost is low, taking about 11.5 minutes on a single NVIDIA A100 GPU. This makes per-domain retraining entirely manageable in practice.

(2) If the question instead concerns the generalization of our full data-efficient RL fine-tuning pipeline beyond math, we have conducted an additional experiment in the science domain using the SCP-25K dataset. Evaluation results on the science subsets of MMLU confirm that our method continues to significantly improve data efficiency, supporting its broader applicability. Please refer to our response to Q2 & L1 of reviewer NvgW for detailed results due to space limits here.

---

[1] Lee, Bruce W., Hyunsoo Cho, and Kang Min Yoo. "Instruction Tuning with Human Curriculum." In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 1281-1309. 2024.

[2] Team, Kimi, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li et al. "Kimi k1. 5: Scaling reinforcement learning with llms." arXiv preprint arXiv:2501.12599 (2025).

[3] Bae, Sanghwan, Jiwoo Hong, Min Young Lee, Hanbyul Kim, JeongYeon Nam, and Donghyun Kwak. "Online difficulty filtering for reasoning oriented reinforcement learning." arXiv preprint arXiv:2504.03380 (2025).

[4] Vojnovic, Milan, and Se-Young Yun. "What is the Alignment Objective of GRPO?." arXiv preprint arXiv:2502.18548 (2025).

[5] Shi, Taiwei, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. "Efficient reinforcement finetuning via adaptive curriculum learning." arXiv preprint arXiv:2504.05520 (2025).