T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling

Thanks for your kind review and feedback!

We would first like to emphasize that the primary contribution of this work is to propose a framework designed to effectively scale the reinforcement learning (RL) training of large language models (LLMs). This approach significantly enhances reasoning capabilities and offers a unique perspective on understanding inference scaling. Extensive experimental results on math and reasoning benchmarks clearly demonstrate the effectiveness of our method, and we hope you recognize both our contribution and the efficacy of our proposed framework.

Additionally, we fully agree that comprehensive ablation studies are essential. To thoroughly validate each component's effectiveness, we have included extensive ablation analyses in our paper (see Tables 1, 2, 3, and Figure 2). For your convenience, we re-present these results here:

1. Effects of high temperature | Temperature | min-p | MATH500 | AIME | Omni-MATH-500 | |-------------|---------|---------|------|---------------| | 0.9 | 0 | 78.2 | 19.1 | 32.0 | | 1.1 | 0 | 84.6 | 29.0 | 37.8 | | 1.2 | 0 | 86.4 | 29.3 | 38.6 | | 1.3 | 0 | 84.6 | 24.3 | 36.4 | | 1.2 | 0.05 | 78.8 | 11.5 | 31.6 |

2. Effects of penalty design

3. Effects of long CoT for SFT. It can be observed that RL with LongCoT (T1) significantly outperforms RL with short CoT (Qwen2.5-Instruct).

For questions,

Q1: If critiques are incorporated into SFT, what is the learning signal?

• A1: In this work, we employ a prompt engineering pipeline (as detailed in Section 2.2.1) to generate responses accompanied by reflections. Critiques serve as intermediate outputs within this pipeline and are integrated into the final response rather than being presented separately. Consequently, our training employs traditional Supervised Fine-Tuning (SFT) with next-token prediction.

Q2: Is the model explicitly trained to generate critiques alongside answers?

• A2: No, critiques are inherently part of the generated response rather than explicitly trained as a separate output.

Q3: Is there a more powerful model used to generate critiques alongside answers?

• A3: We utilize Qwen-72B-Instruct and Gemini-1.5-Pro to initially generate Chains-of-Thought (CoT). Gemini-1.5-Pro is specifically employed to generate critiques. Finally, the o1-mini model (without access to its hidden CoT) integrates the generated CoT and critiques into the final response. Notably, except for o1-mini, these models demonstrate significantly poorer math reasoning performance compared to T1. Furthermore, since the o1-mini model's CoT is inaccessible, using responses from it without the explicit long CoT results in considerably weaker performance relative to T1.

Q4: What is the prompt used to generate the critiques?

• A4: The prompts used to generate the synthetic responses involve a pipeline of prompting with 3 prompts. We will add them into the appendix in the next version due to the space limit of rebuttal. To generate critiques, the key point is to ask the LLM to “ analyze what's wrong in every given wrong response and try to find the key point in every correct response”.

Q5: How do the authors ensure the correctness of the critiques?

• A5: We do not explicitly verify the correctness of critiques. Instead, we just expect the model to learn and initialize self-reflection behavior in the SFT stage and prioritize accuracy and effectiveness improvements during the Reinforcement Learning (RL) stage.

Thanks for your valuable feedback!

Q1: Have you tried other RL algorithms ?

We also conduct experiments using GRPO, and run the experiments using Qwen-14b with k=16 for efficiency. The results are shown as follows

It can be observed that T1 with RLOO and T1 with GRPO show similar performance, which demonstrates that the proposed techniques can bring consistent improvement based on all existing RL algorithms, including both RLOO and GRPO.

Q2: Is the warm-start needed?

A2: Yes, a warm-start is important for reinforcement learning (RL) training as it provides a strong initial foundation for reasoning patterns and encourages exploration. In comparison, for example, Qwen2.5-14b-Instruct, which is trained with RL but utilizes short-CoT, significantly underperforms compared to T1, an RL-trained model initialized from a LongCoT SFT model. Recent studies, such as DeepSeek-R1-Zero, have shown that RL training from a base model can also yield strong reasoning capabilities and self-reflection patterns. However, training directly from a base model typically requires an extensive training period to develop these cognitive patterns. A warm-start approach positions the model effectively from the outset, allowing continuous optimization and refinement of its reasoning abilities.

Thanks for your valuable feedback!

W1: Lack of comparison with other RL methods for LLMs (particularly GRPO)

A1: We appreciate the reviewer's concern regarding the comparison with other RL methods, particularly GRPO and PPO. However, our work, T1, is designed to be independent of any specific RL framework, including GRPO, PPO, and RLOO. The core contribution of T1 lies in developing techniques to scale RL for LLMs, which are agnostic to the underlying RL method. Specifically, the techniques we propose can be integrated with any RL framework to enhance the reasoning abilities of LLMs.

We chose RLOO as the primary RL framework due to its demonstrated stability and the strong performance observed in prior work. Notably, we find that RLOO and GRPO show similar results in our preliminary tests, reinforcing the idea that the techniques we introduce can be applied to both methods. Furthermore, the main contribution of our paper lies in how we leverage RL scaling to improve reasoning, a key area where existing methods, including GRPO and PPO, have not demonstrated substantial improvement in comparison to our findings. We also conduct experiments using GRPO, and run the experiments using Qwen-14b with k=16 for efficiency. The results are shown as follows

It can be observed that T1 with RLOO and T1 with GRPO show similar performance, which demonstrates that the proposed techniques can bring consistent improvement based on all existing RL algorithms, including both RLOO and GRPO.

W2: The contribution is similar to GRPO

A2: While both T1 and GRPO aim to improve RL for language models, our approach fundamentally differs from GRPO by directly targeting exploration scaling rather than refining the RL algorithm itself.

• GRPO focuses on decomposing and improving PPO (e.g., by removing the value network and using group-wise reward normalization), with little emphasis on boosting exploration.
• In contrast, T1 introduces strategies to help RL scaling for reasoning, such as trial-and-error learning with self-verification through chain-of-thought data, along with entropy bonuses and dynamic KL regularization—to systematically explore a broader reasoning space.

T1 offers orthogonal contributions to algorithms like GRPO or RLOO and provides new insights into how RL can directly enhance the reasoning capabilities of LLMs, including RL training scaling and inference scaling.

W3: Computational cost

A3: We employ 64 H800 GPUs for all training, with each iteration taking approximately 300 seconds for a 14B parameter model with K=64. Larger models require higher computational resources and smaller K can lead to faster training. Current open-source improvements in training and inference infrastructure optimization are expected to continuously reduce the compute requirements for RL training and enhance overall training efficiency.

W4: More details about reward modeling.

A4: We use rule-based reward with the correctness of response as the reward, 1 for correct and 0 for wrong. We ask the model to put the final answer within box and then use LLM to judge whether the model’s response equals to the groundtruth. You can find more details in Appendix A.

Thanks for your kind review and valuable suggestions!

Q1 & W1: All training and most evaluations focus on math word problems and quantitative reasoning.

Thank you for the question. We primarily conduct reinforcement learning (RL) training on math problems due to the following two reasons: Math problems can be reliably verified by rules, and thus the reward cannot be hacked during RL training. Math data are easily accessible in open-source communities. Data in other domains tends to be challenging either to verify reliably or to collect comprehensively. Nonetheless, our experiments indicate that training on mathematical problems generalizes effectively to other domains, and we actively evaluate our models in non-mathematical tasks as well, including GPQA and ZebraLogic. We found that training on math data can effectively lead to improvement in other reasoning tasks.

Q2 & W2: Critical training details, such as infrastructure configuration, compute costs, and detailed training data information, are missing.

We employ 64 H800 GPUs, with each iteration taking approximately 300 seconds for a 14B parameter model (larger models require more GPUs and higher computational resources). Smaller K can lead to faster training. While training larger models involves substantial costs, smaller models also significantly benefit from RL training. Furthermore, ongoing improvements in training and inference infrastructure optimization within the open-source community are expected to continuously reduce the compute requirements for RL training and enhance overall training efficiency. For the training data, we use open-source data, including MATH-train, and NuminaMath . We split around 12k for the SFT stage and the others for RL training. And we will open-source the data later. You can find more training details in Appendix A.

[1] MATH (train): Measuring Mathematical Problem Solving With the MATH Dataset [2] Numina-MATH: https://huggingface.co/datasets/AI-MO/NuminaMath-CoT/

Q3: Could you provide an estimate of GPU hours or FLOPs used for training?

We utilize 64 H800 GPUs, with each iteration requiring around 300 seconds for a 14B model. Since RL training combines generation and training processes, we have not recorded exact FLOP measurements.

Q4: Which models are used for synthetic data generation?

We leverage multiple models for synthetic data generation. Specifically, we use Qwen-72B-Instruct and Gemini-1.5-Pro to generate initial Chains-of-Thought (CoT), Gemini-1.5-Pro to generate critiques, and finally, the o1-mini model (without hidden CoT) to merge the CoT and critiques into the final answer. We experimented with several other models and discovered that different models excel in different aspects; thus, we utilize a combination of these models to produce high-quality final outputs.

Q5: Have you conducted RL training on smaller models?

A5: Currently, our experiments are mainly on 9B to 32B models and all of them shows significant performance improvement. We haven’t conducted RL training on smaller models. We think that we can also achieve observable improvement across different scale of models with RL.

Q6: Is there significant variance across different RL training runs?

In our experiments, we observed stable and consistent performance across multiple runs. Variance can be further reduced by increasing the sampling response (larger $K$). Overall, the improvements from RL training are consistent and reproducible across different runs.

Q7: How would model T1 compare with a Supervised Fine-Tuning (SFT) model trained on a larger dataset of human-annotated reasoning chains?

This is an insightful question and one that we are also exploring. Currently, nearly all available reasoning datasets are synthetically generated by models. Human-annotated datasets, such as GSM8k-train and MATH-train, usually contain short CoTs, leading to inferior performance. Open-source, high-quality, human-annotated reasoning datasets remain scarce, and it is unclear whether human-generated LongCoT data would outperform synthetic or RL-trained models. Annotating large-scale human reasoning data is challenging and costly, but we aim to investigate this further in future work.