rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking

Q1: No sensitivity analysis of MCTS parameters—it is unclear whether performance saturates beyond 64 rollouts.

Response: Thank you for your thoughtful review and for recognizing our contributions.

We sincerely appreciate your suggestions and have conducted additional analysis on MCTS parameters, specifically focusing on the number of candidate nodes per step and the number of rollouts, which we found to be the most influential factors.

1. number of candidate nodes per step: To further analyze this parameter, we conducted additional experiments on Math-500 and AIME, testing different candidate node settings (4, 8, 16, 32, 40) under 8 and 64 rollouts. Notably, our paper adopts node=32. As shown in the table below, increasing the number of candidate nodes generally improves accuracy, but beyond 32 nodes, performance saturates.

2. number of mcts rollouts: As mentioned in Section 4.2 (Scaling Up Test-Time Computation), different benchmarks exhibit different trends as rollout count increases. Specifically, Math, AIME, and Olympiad benchmarks saturate at 64 rollouts. For Gaokao and College Math, where performance showed signs of further improvement beyond 64 rollouts, we conducted additional 128-rollout experiments.

As shown in the following table, Pass@N scores consistently improve with more rollouts, but gains become marginal beyond 64 rollouts compared to the doubled search costs. On Gaokao, increasing rollouts from 64 → 128 resulted in only a slight improvement (81.3 → 81.6 for pass@1). On College Math, performance fully saturated at 128 rollouts.

We appreciate the reviewer’s constructive feedback and will incorporate these analyses into our revision.

Q2: Computational efficiency of self-evolution is unclear—training costs (weeks of GPU time) may be prohibitive for broader adoption.

Response: Thank you for your thoughtful feedback and for highlighting the importance of computational efficiency. We provide a detailed cost breakdown in the appendix and further clarify below.

Self-evolution primarily involves two stages: (1) Policy Model & PPM Training; (2) Training Data Generation via extensive MCTS rollouts. As shown in the following tables, training is efficient. Each round completes within a day.

Training data generation is the main cost, but it is scalable and affordable: (1) From Round 2 onward, since our policy model and PPM are both 7B, they can be served on a single 40GB A100. (2) To process 747K math problems efficiently, we used 15 groups of 4×40GB A100s, completing data generation in ~3 days. (3) Further speedup is feasible: This process scales linearly with more GPUs (i.e., reducing the number of problems assigned per GPU). Round 1 is the only costly stage, as it requires bootstrapping with DeepSeek-Coder-V2, which was done using 8×80GB H100s.

Overall, we believe the computational cost is reasonable and manageable, and the primary bottleneck—data generation—can be further optimized with additional GPUs. We will refine our explanation in the revision to provide a clearer analysis of efficiency and scalability.

Q1: The main issue with the paper is the overclaim regarding the "Self-Evolved" nature of the proposed method. The use of a 236B model (DeepSeek-Coder-V2-Instruct) in the initial round of self-evolution directly contradicts this claim.

Response: We appreciate the reviewer' feedback regarding our claim of "self-evolved deep thinking". We clarify our approach and demonstrate that self-evolution remains the primary driver of improvement. Notably, the effectiveness of self-evolution is acknowledged by all the other three reviewers.

1. Clarification of self-evolution: Our method relies on iterative improvement through MCTS-driven deep thinking. While we use DeepSeek-Coder-V2-Instruct-236B in Round 1 for bootstrapping, this does not constitute distillation, as the main role is to provide an initial dataset and plays no role in later rounds. The key novelty lies in Rounds2-4, where we progressively train stronger 7B policy and PPM to improve independently through self-evolution. The two models did not rely on a superior model during the learning, which is the key characteristics that defines model distillation. This is similar to a person who self-evolves the capability through solving problems with known answers (the initial dataset) without other people's help.

2. Empirical evidence: performance gains from self-evolution: Table 8 (Appendix A.2) details per-round performance. We also provide the key results below for reference. The results show that: the policy model (round 1) did not match the performance of 236B, even with SFT data. Performance gains primarily occur in Rounds 2-4, driven by self-play and MCTS. The final 7B model outperforms the 236B model and approaches O1-mini's performance.

3. The role of round 1: code-augmented CoT format induction: We use 236B in Round 1 primarily to ensure that our policy model can generate code-augmented CoT traces, not to transfer problem-solving ability.

4. Distinction from Distillation : Distillation typically involves a student model mimicking a teacher model throughout training. In contrast: (i) we use 236B only once to generate the initial dataset, (ii) self-evolution proceeds without further reliance on 236B, and (iii) later performance improvements arise purely from self-play. Our setup is analogous to AlphaGo, where an initial policy network is required for stability, but key performance gains stem from iterative self-evolution.

To clarify, we will revise the paper to emphasize that Round 1 is for data formatting (code-augmented CoT), not capability transfer. We hope this addresses your concerns and welcome further suggestions.

Q2: The paper proposes a code-based CoT approach to enhance math reasoning. it raises concerns about domain-specificity. The authors should address this limitation by discussing the potential generalizability of their method to other reasoning tasks beyond math.

Response: Clarification of Scope: focus on math reasoning: Thank you for raising this point. We would like to emphasize that the primary focus of our paper is math reasoning, as clearly stated in the title. While generalizing to other domains is an interesting direction, exploring those is outside the scope of the current work. As you correctly point out, the potential generalizabililty of our method beyond math is definitely a worth exploring direction in our future work, as we elaborate next.

1. Code-augmented CoT for code reasoning: Our code-augmented CoT approach could indeed be valuable for other reasoning tasks, such as code reasoning. Code reasoning shares similarities with math reasoning in terms of structured problem-solving, making this method applicable in such domains as well.

2. General Applicability of Self-Evolved Deep Thinking & PPM: Beyond code-augmented CoT, our work’s core contributions — Self-Evolved Deep Thinking and the PPM — potentially offer a more general framework for various reasoning tasks. A key challenge in general reasoning lies in providing feedback to verify whether a trajectory reaches the desired outcome at the end of an MCTS rollout. In code reasoning, this could involve extensive test cases, while in other domains, feedback might come from human annotation or mutual verification with another LLM. Future work will explore these directions.

Thank you again for your valuable feedback and suggestions. We hope these responses address your concerns and clarify any confusion, and we kindly ask you to consider re-evaluating our work.

Q1: Clarification on the evaluation benchmarks: "The authors’ experimental design generally appears sound, particularly in their ablation studies and comparisons on multiple benchmarks, but a closer look at smaller test sets, such as the 15-problem subset of AIME, raises concerns about cherry-picking or insufficient sample size."

Response: Thank you for your thoughtful feedback! We would like to clarify that for AIME 2024, we included both AIME I and II, resulting in 30 problems in total rather than 15. Moreover, our evaluation follows prior works [1,2,3] by selecting AIME 2024 and MATH-500 as key benchmarks, alongside six additional benchmarks, covering a total of 10,195 problems. The following table provides a detailed breakdown of the number of problems in each benchmark. Given this large-scale evaluation, we believe our results are robust and not based on cherry-picked subsets. We will revise the paper to clarify these details.

[1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning

[2] Kimi k1.5: Scaling Reinforcement Learning with LLMs

[3] OpenAI o1, learning to reason with LLMs: https://openai.com/index/learning-to-reason-with-llms/

Q2: the proposed code-augmented CoT data synthesis method has two main shortcomings: while it can detect computational and symbolic deduction errors, it cannot identify logical errors (since Python typically won’t produce an exception in those cases), and its verification is limited to tasks manageable via code execution rather than general math proofs, though this may still suffice for datasets like MATH.

Response: Thank you for your insightful comment! We would like to clarify that the design of our code-augmented CoT is specifically focused on reducing computational and symbolic deduction errors, rather than logical errors. To address logical errors, we primarily rely on our proposed PPM (process preference model), which is designed to select the optimal step-level reasoning step at each stage, such that the final trajectory reaches the correct answer.

Regarding math proofs, as discussed in Appendix A.1 (Generalization to theorem proving), our method shows great potential for application to math proofs, as demonstrated by examples in Appendix A.3. The current limitation is step-level proof verification, which we believe can be addressed with a proof-capable PPM. To overcome this, our future work will focus on collecting step-level proof data to train a dedicated math proof PPM.

Q3: In many RL settings, methods that rely heavily on pre-trained or trainable critic models can be easily hacked by LMs. How does your approach mitigate these risks? Could you explain any strategies you employ to ensure the policy cannot simply exploit the critic’s learned features?

Response: Thank you for your insightful question, which prompted deeper reflection and discussion on this important issue. Here are our thoughts:

1. MCTS-driven self-evolution adopts offline updates: Reward hacking is a known challenge in the general RL and RLHF, when the policy is updated online[1,2,3]. In such cases, the policy can exploit flaws in the reward model, leading to increasing reward scores without real performance gains. Our approach uses offline updates for both the policy and reward model. In each self-evolution round, we fix the policy model and PPM (from the previous round) and use MCTS to generate training data offline. This decoupling ensures controlled (real high-quality) data selection and significantly reduces reward hacking.

2. Robust process-level rewards: A key cause of reward hacking in general RL settings is the lack of reliably labeled response feedback, requiring a learned reward model for scoring. In contrast, math reasoning allows direct validation against ground truth, avoiding the need for a reward model to score final answers. To avoid potential unreliable reward label annotations during process-level reward, we incorporate: (i) MCTS Rollouts and Backpropagation: Initial step-level Q-values from the PPM are refined via MCTS rollouts, with backpropagation adjusting scores based on how often steps lead to correct answers. (ii) Process Preference Model: Instead of using raw Q-values — often imprecise even with extensive rollouts — we construct preference pairs and train the PPM accordingly, reducing noise and improving robustness.

Extensive experiments validate these strategies, showing our approach mitigates reward hacking while ensuring reliable process-level supervision. We welcome any further discussions!

[1] Defining and Characterizing Reward Hacking

[2] Scaling Laws for Reward Model Overoptimization

[3] Reward Shaping to Mitigate Reward Hacking in RLFH

Thank you for your thoughtful and positive feedback on our work. We sincerely appreciate your insights and your recognition of our contributions. Below, we address your specific comments.

Q1: I strongly encourage the authors to share code, models and/or especially the data generated on the usual platforms the foundation model community uses if not done already.

Response: Thank you for your suggestion. We are committed to releasing the code and the generated data as soon as possible.

Q2: P3: “Candidates that execute successfully are retained as valid nodes and scored by the PPM, which assigns a Q-value” it is not clear to me why the PPM assigns the Q value as page 2 indicates that the Q value is generated separately from the number of trajectories leading to the correct answer for a given node, while the PPM is trained from the Q value; could you clarify that point (here and ideally either in page 2 or 3 depending on your answer)? Currently the flow of p2 and fig 1c makes it seem like the code-augmented component and Q value assignment of your pipeline happens already before (and is a prerequisite for) training the PPM.

Response: Thank you for your question! We acknowledge that the current description may cause some ambiguity, and we will revise the paper to clarify this process.

To clarify: as shown in Fig. 1(c), in the first and second rounds of data generation, we did not have a PPM yet. Instead, we assigned Q-values to each candidate based on terminal-guided annotation. This Q-value-labeled data was then used to train the PPM. Starting from the third and fourth rounds, the PPM became available, enabling PPM-augmented Q-value annotation. Specifically, during MCTS, each newly generated node is initially assigned a Q-value predicted by the PPM (whereas in the first two rounds, the initial Q-value was set to 0). This Q-value is then refined through MCTS rollouts. We will update Page 3 to make this process clearer.

Q3: Fig 2: is it intended that the python code for step 2 repeats the python code for step 1 first?

Response: Yes, this design is to ensure that the Python code executes correctly. Specifically, Step 2's code may depend on variables or functions defined in Step 1. If a node corresponding to Step 2 were to execute only its own Python code without including the preceding steps, it could lead to syntax or execution errors due to undefined references.

Q4: P5: “In each selfevolution round, we perform 16 rollouts per math problem, which leads to 16 reasoning trajectories. “ considering multiple steps and multiple future trajectories per step in rollouts, wouldn’t that lead to more than 16 reasoning trajectories?

Response: Thank you for your question! Our definition of rollout follows prior works [1,2,3], where a full rollout refers to a complete reasoning trajectory from the root node (question) to a terminal answer node. Thus, each rollout consists of multiple reasoning steps and explores different paths, ensuring that performing 16 rollouts per math problem yields at least 16 full reasoning trajectories. We appreciate your careful reading and will revise the paragraph to make this clearer.

[1] Reasoning with language model is planning with world model, emnlp 2023

[2] Mutual reasoning makes smaller LLMs stronger problem-solvers, iclr2025

[3] LLaMA-Berry: Pairwise optimization for olympiad-level mathematical reasoning via o1-like monte carlo tree search

Q5: Tab6: could PQM be underperforming due to PPM being more attuned to the policy model since it stems from several rounds of iterating?

Response: Thank you for your insightful question! The primary reason PQM underperforms compared to PPM is that using score-based Q-values as direct labels for the process reward model inherently introduces imprecision. Although extensive MCTS rollouts help improve Q-value accuracy, these Q-values struggle to differentiate fine-grained quality levels. For example, distinguishing an optimal step (score=1.0) from a near-optimal step (score=0.9) is inherently challenging, whether through MCTS automatic annotation or human annotation. This inevitably introduces noisy scores in PQM's training data, affecting its reliability.

To fully address your question, we conduct an additional experiment. We use the generated data from SLM-r1 (our first trained 7B policy SLM from round 1) to train both PQM and PPM. As shown in the table, PPM consistently outperforms PQM from the early stages of self-evolution. This empirically demonstrates that PPM's advantage stems from its superior design, which effectively mitigates the impact of noisy Q-value scores.