Reviewer 4 YMHP

We would like to thank reviewer YMHP for their feedback and the time they dedicated to our review. We worked to address each of their comments and hope they will not hesitate to let us know of any further suggestions or questions.

responses

1. I didn't see an ablation study for the two losses. [...] I think an ablation study for the impact of the two losses would be the most useful.

We would like to clarify that we did perform an in-depth ablation of our reward terms in Section D of our Appendix.

1. First, we examined ablating the think tokens KL reward term $r^{KL}$, quantifying whether the think tokens $t_{o_i}$ themselves are interpretable logical continuations from the student's perspective as compared with the teacher's.
2. Second, we examined ablating the min/max reductions terms, which serve to ensure the rewards do not forego any individual token, as detailed in Section 3.3.

We provided these results in Table 7 and summarize them again in the Table below for easier reference:

Furthermore, beyond performance, in Figures 8-12, we also compared the lengths of the produced teacher explanations and provided several qualitative, concrete examples showing the effects of each reward term. As detailed in Appendix D.1, ablating $r^{KL}$ leads to a teacher that only learns to repeat the solution tokens themselves in its explanation to exploit the repetition tendency of pretrained student LMs, with the distillation performance and average length of its output reasoning traces dramatically dropping. Furthermore, ablating the min/max reward reduction terms makes the rewards effectively biased by the length of the reasoning trace. While this second ablation only minimally influences performance, it makes the number of think tokens in the teacher's reasoning traces almost double, unnecessarily increasing the downstream student distillation cost.

Following the reviewer’s feedback, we extended the end of Section 3.3 to reference more explicitly the presence of this set of important results validating our reward design, and avoid readers potentially missing them. In particular, we modified lines 198-199 to:

“For further discussion, we refer to Appendix D, where we empirically analyze and validate each design choice in our reward function in terms of final performance (Table 7) and concrete qualitative differences of the resulting explanations (Figures 8-11).”

We note that we did not consider ablating $r^{SS}$ to devolve our compute resources to the other experiments, as without this term our RLT would simply collapse to the student’s initial distribution to minimize $r^{KL}$ with no incentive to make its explanations useful. In fact, this is precisely what we also observed in early runs where the weighting coefficient $\lambda$ of $r^{KL}$ was too high. However, we hope the reviewer will not hesitate to let us know in case they believe we should still collect these results and add them to our ablation study in future revisions.

2. Distillation is something very data dependent and conclusions might change a lot with scale and where questions + answers are being sourced from [...] I'm interested in seeing, for example, how the performance of a student distilled from the teacher changes.with / without the RLT teacher over differing amounts of NuminaMath data.

We definitely agree with the reviewer that distillation performance can be very dependent on the scale and quality of the initial question dataset. As described in Sections 4.1 and 4.2, this is precisely the reason that we distilled our students using the same starting set of 17K question-solution pairs from the recent state-of-the-art distillation baselines used in our comparison [1, 2]. This allowed us to remove potential confounding factors other than the quality of the reasoning traces between the ones generated from our RLT and much more expensive LLMs such as DeepSeek R1 [3] (we note that we also matched our baselines’ exact distillation hyperparameters and evaluation code rather than making our own).

Following the reviewer’s feedback, we collected and added new results in Appendix C, changing the amount of student distillation data, focusing on the NuminaMath dataset. While we did not have time to experiment with hundreds of thousands of samples, we constructed new datasets with 1K, and 10K samples from the AIME, MATH, and Olympiads subsets of NuminaMath as done in [1], enabling us to directly compare with the available reasoning traces from our best baseline using postprocessed reasoning traces from DeepSeek R1 (i.e, Bespoke-7B [2]):

As summarized in the Table above, we find that the performance of the RLT and R1 traces from the Bespoke pipeline positively scales with the amount of student training data. While we removed all science questions from STILL-2 present in our original data [4], we find performance on GPQA does not decrease substantially, evidencing positive transfer across different reasoning domains. We note these results are consistent with our previous random subsampling of the questions from [1] (from Table 1 of our paper), with the RLT traces outperforming the postprocessed R1 traces across all tested data sizes and configurations.

We hope these new results will provide further insights to our future readers regarding the scalability and robustness of our new framework.

3. The Related Works section includes few discussions or mentions of papers that are working on "better teachers". Examples are Montessori-Instruct and DataEnvGym [...] I think this paper is different enough that it doesn't need to justify it's novelty, but it should situate itself relative to prior work better.

Following the reviewer’s feedback, we extended our Related Works section to include prior methods working toward similar goals of designing/training “autonomous data generation agents,” including the two referenced works [5, 6] and others such as [7]. We made sure to better contextualize the contributions of our paper in relation to these papers in terms of its different methodology and its application to the domain of LM reasoning.

references

[1] Li, Dacheng, et al. "LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!."

[2] Bespoke Labs. Bespoke-stratos: The unreasonable effectiveness of reasoning distillation.

[3] Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning."

[4] Min, Yingqian, et al. "Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems."

[5] Li, Xiaochuan, Zichun Yu, and Chenyan Xiong. "Montessori-instruct: Generate influential training data tailored for student learning."

[6] Khan, Zaid, et al. "Dataenvgym: Data generation agents in teacher environments with student feedback."

[7] Ning, Xuefei, et al. "Can LLMs learn by teaching for better reasoning? A preliminary study."

We would like to thank reviewer UdmA for their feedback and the time they dedicated to our review. We worked to address each of their comments and hope they will not hesitate to let us know of any further suggestions or questions.

responses

1. A clear cost analysis is missing [W1]. Could you provide a more direct comparison of the total computational cost [...]? [Q1]

Following the reviewer’s suggestion, we added a subsection to Appendix A, reporting the GPU cost of training and collecting distillation data with our RLT (recorded using a single H100 node with 8 GPUs, see Appendix A.1 for full hardware specs). We compared these times with an estimated minimum cost of traditional RL post-training and data collection using the DeepSeek R1 model. To compute this estimate, we used official figures for the GPUs used to train this DeepSeek model (2,048 NVIDIA H800 GPUs [1, 2]) and online communication from the DeepSeek team. Moreover, we also calculated a lower bound of the GPU hours for generating the distillation dataset by multiplying the minimum amount of GPU resources required to host DeepSeek R1 with half precision and the “roof inference speed” for generating each completion in our dataset, without taking into account slowdowns from the growing context size.

However, as we do not have the resources to train/host DeepSeek R1 ourselves, we note that both these estimates should be considered large underestimations (we did not even consider time/resources for multiple generations, filtering, and post-processing stages conducted in our baselines [3, 4]). We provide a summary of this cost comparison in the Table below:

We would also like to note that the training time of RLTs is largely dominated by the autoregressive generation loop during the RL phase, with the initial SFT and the extra forward passes through our student network adding a negligible amount of compute cost. This is also precisely the case in the traditional RL reasoning framework, as autoregressive generation comprises a large number of forward passes up to the maximum output context length specified for each single batch sample (16384 in our case).

We hope that adding this new direct comparison and analysis to our paper will help better contextualize the trivial overheads and great potential of our new RLT framework to cut future requirements for training and distilling RL reasoning models.

2. Recent work [5] provides empirical evidence that teaching a diverse set of students is more effective [...] This can be implemented by either teaching a diverse group of students concurrently or by teaching different students at different training stages. [Q2]:

Following the reviewer’s suggestions, we added a direct reference to [5] and collected two new sets of experiments in Appendix D to increase student diversity for computing the RLT rewards:

1. First, we considered using both the 7B and the 32B students, averaging their probabilities, thus introducing student diversity into the RLT reward.
2. Second, we considered conducting training in two stages, each optimizing both teacher and student for half the total number of optimization steps. In particular, we paused the teacher RL optimization midway, distilled a student with this preliminary RLT’s explanations, and resumed our teacher optimization for the remaining steps with this updated student to compute the RLT reward.

We note that this second extension has the added benefit of better aligning the teacher rewards to the student’s learning dynamics, and can be considered an initial step toward online optimization of both models (a future direction outlined in Section 6, lines 361-363).

As summarized in the Table above, both these extensions provide some moderate improvements to the RLT distillation performance. However, we also note that these came at a substantial additional training cost, taking approximately 1.4x and 1.8x more GPU hours than our original pipeline.

3. The paper [...] does not explore the scaling behavior of the teacher model. [W2] Have you ever tried the training of larger RLTs, for example, a 14B RLT? [Q3]

As detailed in our previous response, we ran all our experiments for a 7B RLT using a single H100 node. However, as the cost of training and deploying LLMs does not scale linearly with the number of parameters, our same experiments with a 14B RLT would have taken several weeks and required expensive multi-node setups, beyond the project's allocated resources.

Thus, to still try to address the reviewer’s feedback, we instead focused on decreasing the size of the RLT teacher down to 3B parameters. We ran these experiments and added these new results to Appendix D:

As summarized in the Table above, we indeed confirm that larger teachers yield better explanations and downstream results. However, most of the performance gap between our 3B and 7B teachers appears to occur only when distilling the 32B student LM, where the mismatch between teacher and student capabilities tends to the extreme. However, even in this case where the student is over 10x larger than the teacher, our 3B RLT is still able to provide considerable improvements to the initial Qwen 32B's performance.

We believe these results further validate how the RLT framework enables even small, inexpensive models to be effective teachers, but also confirm that larger teacher models are indeed able to provide better explanations, highlighting the potential future implications of scaling with more training compute.

4. Why is $r^{KL}$ computed under the RLT format of the teacher? [Q4]

The $r^{KL}$ reward term is computed with the KL divergence over the distribution of think tokens outputted by our RLT during training (thus, under the RLT’s format with both the question and solution in context) and the student’s distribution over the same think tokens (under the distillation format with only the question in context). The purpose of $r^{KL}$ is to ensure that the think tokens in the explanations outputted by the teacher are interpretable logical continuations from the student’s perspective.

Instead, if $r^{KL}$ was computed using the same student format also for our teacher (thus, seeing only the question in context), there would be no regularization preventing our teacher from outputting explanations simply copying and repeating the solution tokens to increase likelihood with the RLT format. As shown by our experiments in Appendix D.1, this is precisely what happens empirically when only optimizing the $r^{SS}$ term: resulting in a teacher that only learns to repeat the solution tokens themselves rather than providing general examples of reasoning steps that can be applied to new problems.

Following the reviewer’s comments, we tried to make our motivation clearer by extending Section 3.3 as summarized above, explicitly explaining what would occur if we did not use the RLT format for $r^{KL}$:

“[...] If we instead compared two distributions conditioned on the question alone, the KL would vanish and would fail to penalize the RLT steps that the teacher could not have generated without the solution in context.”

5. Minor: Some sentences are too long to easily comprehend [W3.1]

Following the reviewer’s feedback, we scanned through the text and broke down overly long sentences to improve readability. Two examples include:

Lines 57-59 in our introduction, which we modified to:

“We train RLTs with dense rewards using the student’s log probabilities. Intuitively, our rewards measure both the student’s” understanding of each problem’s ground-truth solution from the teacher’s explanations, and the interpretability of the logical leaps in the explanations themselves.”

Lines 254-256 in Section 4, which we modified to:

“We also find the effectiveness of the RLT traces stays consistent across different data sizes. In contrast, the R1 traces from the Bespoke pipeline appear significantly less effective when subsampled.”

If the reviewer has any other specific example where text clarity could be improved that we might have missed, we hope they will not hesitate to let us know.

6. Minor: In Figure 1, unclear whether it is the student size or teacher size [W3.2]

Following the reviewer’s feedback, we added a second sentence to the caption in Figure 1 to explicitly refer to the teacher/student sizes in each subplot:

“A 7B RLT provides better distillation and RL cold-starts [...] We show this holds both when distilling 7B students of the same size (Left) and also 32B students, much larger than the RLT itself (Right).”

references

[1] Liu, Aixin, et al. "Deepseek-v3 technical report."

[2] Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning."

[3] Li, Dacheng, et al. "LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!."

[4] Bespoke Labs. Bespoke-stratos: The unreasonable effectiveness of reasoning distillation.

[5] Ning, Xuefei, et al. "Can LLMs learn by teaching for better reasoning? A preliminary study."

We would like to thank Reviewer Esvg for their feedback and the time they dedicated to our review. We worked to address each of their comments and hope they will not hesitate to let us know of any further suggestions or questions.

weaknesses

1. During the testing phase, the student model still relies on explanations from the teacher to produce answers, lacking independent problem-solving ability. Additionally, the teacher's explanations may already include the answer, effectively revealing the solution to the student model—naturally leading to better performance.

We would like to clarify that the teacher’s explanations are only used during the training of our students. During testing, our student models do not rely in any way on the explanation produced by the teacher and can effectively solve problems from scratch independently. In fact, our procedure for distilling and testing our students is exactly the same as prior distillation approaches [1, 2]. The only difference is how we obtain the distillation dataset:

• Prior approaches [1, 2] using expensive LLMs not only make use of the ground-truth solutions to verify the correctness of their reasoning traces but also rely on filtering and post-processing stages to improve the quality of their distillation data.
• In contrast, we show RLTs are more effective without requiring any filtering and post-processing stages, using the raw outputs from orders-of-magnitude smaller 7B teacher LMs to generate the distillation data.

Following the reviewer’s feedback, we expanded the text in Section 4.1 by including the above explicit comparison with the traditional RL distillation pipeline, which we hope will make it clearer to readers that our RLT framework actually has lower requirements and that our students do possess stronger independent problem-solving ability.

2. [...] The student model remains fixed during the teacher's training [...] This may lead to issues such as the reward function overfitting [...] or failing to align in real time [...]

Throughout our experiments in Section 4, we found that keeping the student model fixed was sufficient for our 7B teacher to outperform distillation/cold-starting with LLMs such as DeepSeek R1. While optimizing both models at the same time would be more costly in terms of computation, we do agree with the reviewer that concurrent optimization is a potential further direction to scale up our framework. In fact, we mention this exact idea when discussing future extensions in Section 6:

“One example [of future directions] is training RLTs and their students in tandem, allowing the teacher's explanations to adapt to the student's learning dynamics live” (lines 361-363).

Nonetheless, to still try to address the reviewer’s feedback, we collected and analyzed two new sets of more expensive experiments in Appendix D, taking a first step in this direction:

1. First, we considered conducting training in two phases, each optimizing both teacher and student for half the total number of optimization steps. In particular, we paused the teacher's RL optimization midway, distilled a student with this preliminary RLT’s explanations, and resumed our teacher optimization for the remaining steps with this updated student to compute the RLT rewards. The objective of this experiment is precisely to lessen the risk of potential overfitting to the student’s initial parameters, as suggested by the reviewer.
2. Second, we also considered using both the 7B and the 32B students for computing the RLT reward, averaging their probabilities. The objective of this experiment was instead to lessen the risk of potential overfitting to a single model.

As summarized in the Table above, both these extensions provide some moderate additional improvement to the RLT's distillation performance. However, we also note that these improvements came at a substantial additional cost, taking approximately 1.8x and 1.4x more GPU hours than our original RL training pipeline.

3. No ablation study was conducted to evaluate the independent contributions and sensitivity of different reward terms.

We would like to clarify that we did perform an in-depth ablation of our reward terms in Section D of our Appendix:

1. First, we examined ablating the think tokens KL reward term $r^{KL}$, quantifying whether the think tokens $t_{o_i}$ themselves are interpretable logical continuations from the student's perspective.
2. Second, we examined ablating the min/max reductions terms, which serve to ensure the rewards do not forego any individual token, as detailed in Section 3.3.

We provided these results in Table 7 and summarize them again in the Table below for easier reference:

Furthermore, beyond performance, in Figures 8-12, we also compared the lengths of the produced teacher explanations and provided several qualitative, concrete examples showing the effects of each reward term. As detailed in Appendix D.1, ablating $r^{KL}$ leads to a teacher that only learns to repeat the solution tokens themselves in its explanation to exploit the repetition tendency of pretrained student LMs, with the distillation performance and average length of its output reasoning traces dramatically dropping. Furthermore, ablating the min/max reward reduction terms makes the rewards effectively biased by the length of the reasoning trace. While this second ablation only minimally influences performance, it makes the number of think tokens in the teacher's reasoning traces almost double, unnecessarily increasing the downstream student distillation cost.

Following the reviewer’s feedback, we extended lines 198-199 at the end of Section 3.3 to reference more explicitly the presence of this set of important results that validate our reward design, and avoid readers potentially missing them:

“For further discussion, we refer to Appendix D, where we empirically analyze and validate each design choice in our reward function in terms of final performance (Table 7) and concrete qualitative differences of the resulting explanations (Figures 8-11).”

questions

1. Could you please clarify the connection between these aspects and the concept of "Test-Time Scaling" as highlighted in the title?

By “test-time scaling,” we are referring to the RLT’s ability to produce explanations for training students and providing them the ability to use additional compute to generate lengthy reasoning traces and improve performance. We note that this same nomenclature is used in the abstracts, introductions, and titles of many prior important papers in the LM post-training field, such as DeepSeek-R1 [3], THUDM’s T1 [4], and "s1: Simple test-time scaling." [5].

Following the reviewer’s comment, we tried making the connection more evident for our readers by modifying lines 26-29 and adding direct citations to the above example works:

“With the rise of RL for open-ended reasoning (RL reasoning), inducing a new form of language model (LM) test-time scaling [3, 4, 5]...”

2. Could you provide a more detailed explanation as to why the proposed method is considered scalable? What specific properties or mechanisms enable its scalability?

By simplifying the RL task and directly aligning the RL objective with downstream student learning, we show how a 7B RLT provides better distillation and cold-starting data than the 671B DeepSeek R1 model and other closed-source LMs. These results provide concrete evidence that the RLT framework has the potential to become a more scalable framework by greatly reducing the cost of training advanced models. Instead of relying on large models at every stage, our framework allows training small, specialized teachers that can be used to teach much larger models efficiently. This flips the traditional scaling paradigm: the heaviest work (RL training) can now be handled by compact, more affordable models.

To better put the scalability and efficiency of our framework into perspective, we added a section comparing the GPU cost of training and collecting reasoning traces for distillation with our RLT model compared with an estimated lower-bound cost of RL training and data collection using DeepSeek R1 (see our first response to reviewer UdmA for details):

We hope that adding this new direct comparison and discussion to our paper will help better contextualize the scalability and potential of our new RLT framework, allowing for cheaply distilling large reasoning models while performing expensive RL training with much smaller and efficient ones to counteract the exponentially growing training costs from the scaling laws of LLMs [6].

references

[1] Li, Dacheng, et al. "LLMs Can Easily Learn to Reason from Demonstrations Structure, not content, is what matters!."

[2] Bespoke Labs. Bespoke-stratos: The unreasonable effectiveness of reasoning distillation.

[3] Guo, Daya, et al. "Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning."

[4] Hou, Zhenyu, et al. "Advancing language model reasoning through reinforcement learning and inference scaling."

[5] Muennighoff, Niklas, et al. "s1: Simple test-time scaling."

[6] Hoffmann, Jordan, et al. "Training compute-optimal large language models."

We would like to thank Reviewer YrSD for their feedback and the time they dedicated to our review. We worked to address each of their comments and hope they will not hesitate to let us know of any further suggestions or questions.

weaknesses

1. The authors state that the RLT framework can be used with any RL algorithm, but the experiments performed solely use GRPO. Although I find this believable, the paper could benefit from an experiment to demonstrate and verify this claim.

Following the reviewer’s suggestion, we extended our implementation and added a new set of experiments training our 7B RLT model using the RLOO [1] algorithm instead of GRPO [2]. We added these new results and analysis to Appendix C, which we summarize below:

In line with recent empirical findings about the similar empirical effectiveness of the different reasoning RL algorithms [3], the performance of our RLT trained with RLOO appears very close to our previous results obtained using GRPO. We believe the small gap is mostly due to the fact that we did not re-adjust any of the main hyperparameters from our GRPO implementation (e.g., batch size, number of generations per question). We hope these new results will provide concrete evidence confirming that the RLT paradigm's performance is not tied to any specific RL algorithm.

2. The authors do not discuss what happens in cases where there is a large mismatch in the abilities of the teacher and student LM [...] An ablation on "small teacher LM", with increasingly more capable student LMs (beyond 32B) would greatly improve the analysis of the paper.

While our current experiments do show that even with a significantly smaller 7B teacher, our framework remains effective for teaching a much larger student, we understand the reviewer’s interest in examining the effect of pushing the gap between teacher and student to the extreme. However, while we ran all our experiments on a single H100 node (8 GPUs), we note that training models beyond 32B would require multi-node training and be computationally too expensive for this project’s resources.

Thus, to still try to address the reviewer’s suggestion, we instead focused on further decreasing the size of the RLT teacher down to 3B parameters. We added these new results to Appendix D of our current revision, which we summarize in the Table below:

As shown by our new set of results, we expectedly find that larger teachers yield better explanations and downstream results. However, we find that most of the performance gap between our 3B and 7B teachers only occurs when distilling the 32B student LM, where the mismatch between teacher and student capabilities tends to the extreme. However, even in this case where the student is over 10x larger than the teacher, our 3B RLT is still able to provide considerable improvements to the initial Qwen 32B student.

The reason that we did not observe performance degradation appears to be due to the role of the $r^{KL}$ reward term plays during the optimization. Even in cases where the initial 3B teacher is unable to provide logical explanations to a specific question and answer, optimizing $r^{KL}$ will naturally guide the teacher's output distribution (with both question and answer in context) toward converging to what was most likely from the student's distribution (with only the question in context). We added in Appendix D an analysis of this phenomenon, which we validated by inspecting at different training stages the 3B teacher's output explanations for challenging questions with low values of $r^{SS}$.

We believe these results further validate how the RLT framework enables even small, inexpensive models to be effective teachers, but also confirm that larger teacher models are indeed able to provide better explanations, highlighting the potential future implications of scaling with more training compute.

We are open to collecting additional results for future revisions of our work by pushing the gap between teacher and student even further by training and evaluating a 0.5B teacher model, in case the reviewer finds this further necessary.

3. In (i) of section 3.3, [...] I interpreted that the logprobs are normalized (divided by) the avg and min logprobs. [...] Similar comments apply to also (ii) in section 3.3.

Following the reviewer’s feedback, we went through the paper and extended the text in several cases where we previously used concise language (for saving space) to improve clarity. For example, in the specific instances flagged by the reviewer, we rewrote lines 175-176 and 178-181 to:

“This first reward term is computed with the student’s log probabilities over the solution tokens. We reduce this vector to a scalar by applying both average and minimum operations over the different per-token values:” (ex lines 175-176)

“This second reward term is computed with the KL divergence over the same think tokens [...]. Similarly to $r^{SS}$, we reduce this vector to a scalar by applying both average and maximum operations over the different per-token values:” (ex lines 178-181)

questions

1. The authors state that the paradigm of solely correctness-based rewards does not allow the model to explore beyond its initial latent abilities (line 38). With the RLT's dense reward paradigm, can there be exploration beyond the initial LM's latent abilities? To me, this answer still seems to be "no" -- but since the teacher does not have to solve the task, there are no sparse-reward problems [...]

Following the reviewer's question, we realized that the term "exploration" could have different interpretations and added an explicit explanation in Section 1 and Appendix E for our statement that “correctness-based rewards do not allow the model to explore beyond its initial latent abilities,” which we summarize below:

In traditional RL for robotics, neural network policies can learn to solve tasks from random initializations thanks to being guided by dense reward functions. Without this guidance, they would be faced with an infeasible exploration challenge made exponentially more difficult by the task horizon [4]. This is because dense rewards allow the policy gradient optimization to rank the relative progress obtained across its initial suboptimal actions, allowing the policy to bootstrap from partial solutions and extrapolate far beyond its initial capabilities.

However, in the context of LM reasoning, as correctness-based rewards are inherently sparse, this extrapolation is not possible. In particular, if an LM is too weak to provide the optimal solution for a task, all its rewards (and, thus, the policy gradient) will be zero. In contrast, this limitation would not apply to our RLT, which can make use of a dense reward function and obtain a learning signal to iteratively improve even when all sampled initial explanations are suboptimal, as with the traditional RL framework.

We hope this new discussion will clarify what we mean by “extrapolation beyond the LM’s initial latent abilities” (lines 37-38) and hope the reviewer will not hesitate to let us know if they believe anything else requires additional contextualization.

references

[1] Ahmadian, Arash, et al. "Back to basics: Revisiting reinforce style optimization for learning from human feedback in llms."

[2] Shao, Zhihong, et al. "Deepseekmath: Pushing the limits of mathematical reasoning in open language models."

[3] Hu, Jian, et al. "Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models."

[4] Devidze, Rati, et al. "Explicable reward design for reinforcement learning agents."