Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search

1. Over-claim about "without external guidance".

We would like to clarify that “external guidance” specifically refers to guidance provided by another LLM verifier at inference time (see Abstract, lines 18-19). Many existing LLM reasoning methods rely on extensive sampling and guidance from a verifier model (e.g., PRM-guided tree search). Regarding the cost of the multi-agent system, we believe this is analogous to the inevitable cost of curating most LLM training datasets. We will clarify these points in the revision.

2. Over-claim about "superior performance".

We would like to highlight that the performance gains on challenging MATH benchmarks are substantial. For example, we achieved +5% on OlympiadBench, +10% on AMC2023, and +3.3% on AIME2024 compared to the best available 7B models. Moreover, on our out-of-domain generalization experiments, our method consistently outperforms baseline models with an average improvement of 8.3% (see Table 2). While we acknowledge that the paper review may carry some degree of subjectivity, we would also like to note that all three other reviewers appreciate the strong empirical performance. However, we would like to tone down the performance claims in the revision.

3. No theoretical results.

We acknowledge the reviewer’s concern regarding the lack of theoretical analysis, as our proposed training framework currently does not come with a rigorous theoretical justification. However, we would like to note that conducting a comprehensive theoretical analysis in the context of LLM, especially for complex RL training pipelines is highly non-trivial and beyond the scope of this work. We also agree that some components, particularly the proposed Restart and Explore (RAE) technique, merit further investigation. We view this as an important direction for future work and include some preliminary theoretical insights about RAE as a starting point for such analysis.

RAE modifies the initial state distribution by starting new rollouts not only from dataset-sampled prompts but also from random partial trajectories, i.e., prompts concatenated with partial responses generated by intermediate policies. This design can be justified by Theorem 6.2 of the Conservative Policy Iteration (CPI) paper [1], It states that if $\pi$ is an approximately optimal policy learned from one initial distribution $\mu$, and $\pi^*$ is the true optimal policy for a different initial distribution $\tilde{\mu}$, then

$$\eta_{\tilde{\mu}}(\pi^*) - \eta_{\tilde{\mu}}(\pi) \ \le \ \frac{\epsilon}{(1-\gamma)^2} \left\|\frac{d_{\pi^*, \tilde{\mu}}}{\mu}\right\|_\infty,$$

where $d_{\pi^*, \tilde{\mu}}$ is the stationary state distribution of $\pi^*$ under $\tilde{\mu}$, $\gamma$ is the discount factor, and $\epsilon$ is small approximation error. The concentrability coefficient $\bigl\|\frac{d_{\pi^*, \tilde{\mu}}}{\mu}\bigr\|_\infty$ measures how badly $\mu$ misses states that $\pi^*$ visits. If $\mu$ puts little or no mass on those important states, the policy learned from $\mu$ can perform poorly under $\tilde{\mu}$.

In the context of LLMs, the "initial state" corresponds to the prompt prefix. Training with a narrow or fixed set of prompts results in a limited $\mu$, which may place little to no mass on the key states an optimal policy should visit, thereby inflating the concentrability coefficient. RAE mitigates this issue by broadening $\mu$ through randomized partial rollouts, effectively augmenting the initial state distribution with more diverse and relevant prefixes encountered during training. While we do not have access to the true stationary distribution $d_{\pi^*, \tilde{\mu}}$, this augmentation strategy reduces distribution mismatch and narrows the theoretical performance gap between $\pi$ and $\pi^*$. This aligns with the conclusion in [1], which argues that more uniform state coverage leads to tighter bounds.

We plan to explore a more formal and comprehensive analysis of RAE in future work.

4. Question-1: If you perform SFT using only your synthesized data in stage 1, would the performance differ from the current results?

We kindly refer the reviewer to the “Large-scale FT vs. Large-scale RL” ablation study in Section 6, where we have demonstrated that performing SFT using only the large-scale synthesized data in Stage 1 is sub-optimal compared to large-scale RL training.

5. Question-2: Is there an R1-style comparison?

Please refer to our response to reviewer wWt8 for details.

Reference

• [1] Approximately optimal approximate reinforcement learning. ICML, 2002.

---

If these clarifications satisfactorily address the reviewer's concerns, we kindly ask if the reviewer would consider updating the score to reflect what we believe is a paper with noteworthy contributions to the community.

1. R1 uses a cold-start RL to develop reflection skills on its own (the “Aha moment”), but the paper relies on a specialized dataset to teach reflection-like abilities.

We would like to respectfully clarify several important points:

• R1 is a concurrent work and comparison should not be required: r1 was officially released around the same time as the ICML 2025 abstract deadline (end of January). According to the reviewer guidelines, we are not required to compare with or even mention r1. However, we chose to acknowledge r1 as a concurrent work because we believe it offers valuable context for the research community amidst the recent interest it has generated.
• Clarifying potential misunderstanding: We respectfully suggest that the reviewer may have some confusion regarding r1. (1) Deepseek introduces two models: r1-zero, which is trained using pure RL from a base model, and r1, the actual released model. R1 includes a SFT stage before RL, where the SFT data is explicitly designed with a reflection pattern (see Section 2.3.1 in [1]). Thus, r1 employs a similar strategy to our format-tuning stage, although its implementation details are not disclosed. (2) Rather than limiting capabilities, this SFT stage is helpful to achieve more effective RL. Indeed, the r1 authors acknowledge the limitations of r1-zero without SFT (see Section 2.2.4 in [1]), including poor readability and language mixing issues. Furthermore, recent work [2] supports the effectiveness of incorporating SFT before RL. Several studies have also questioned the validity of the “Aha moment” claimed by r1-Zero and suggest “Aha moment” could be a mirage (see [3]), which challenges the reviewer's claim that "r1 can develop reflection skills on its own".
• Comparison results provided: Please refer to our response to reviewer wWt8.

2. The performance still lags behind o1 and open-sourced r1 model.

• Unfair comparison: We respectfully argue that it is not reasonable to compare the performance of a research prototype with that of large-scale industry systems. R1 contains over 600B parameters, whereas our model is based on a 7B LLM. Furthermore, the scale of training data and infra used by o1 and r1 is not publicly disclosed but is far beyond what is accessible in a academic setting.
• R1 is not truly open-sourced: While o1 is certainly closed-source, we would also like to clarify that r1 is not fully open-sourced. Only the model weights have been released. The training data, codebase, and detailed pipeline remain proprietary. Over the past two months, several efforts have attempted to reproduce r1’s performance, but few—if any—have succeeded in matching it, which further suggests that important implementation details are missing.
• Research Objective: o1 and r1 are undoubtedly successful, but their performance is largely attributed to extensive resources. In contrast, the goal of academic research is not to achieve SOTA benchmark results, but rather to explore novel methodologies orthogonal to scaling up resources. For instance, our proposed RAE technique effectively mitigates the sparse reward issue in RL, which remains a major challenge for complex reasoning tasks and is under-explored in prior works. Moreover, RAE could potentially be integrated into r1’s training framework as well. We respectfully note that if benchmark performance were the sole criterion for evaluation, then almost no academic work (including LLM reasoning papers submitted to ICML 2025) could surpass current industry-developed models such as r1 or o1.

3. How does your approach mitigate the risk of critic model hacking?

• Request for clarification: In RL, the term “hacking” typically refers to reward hacking, where the model exploits flaws in a reward model to receive undeservedly high scores. By contrast, the critic model in PPO is used for estimating advantages via GAE, and does not directly influence the reward signal. We kindly ask the reviewer to clarify whether the concern is regarding to reward model or PPO’s critic model.
• Our approach: Satori utilizes a hybrid reward mechanism with both a reward model and rule-based reward. (see Section 4.2). While it is possible that the reward model may become less reliable during later stages of training, our design ensures that the rule-based reward remains dominant, mitigating the risk of reward hacking.

Reference

• [1] DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning, Jan, 2025.
• [2] Demystifying Long Chain-of-Thought Reasoning in LLMs, Feb 2025.
• [3] There May Not be Aha Moment in R1-Zero-like Training — A Pilot Study, Feb, 2025.

---

If these clarifications satisfactorily address the reviewer's concerns, we kindly ask if the reviewer would consider updating the score to reflect what we believe is a paper with noteworthy contributions to the community.

Compare with more naive approaches (e.g., the ones adopted in DeepSeek-R1 with only RL finetuning).

We appreciate the reviewers' interest in the comparison between our method and the RL-only approach (r1-zero). As noted in our response to reviewer WTyn, r1 is a concurrent work, and such comparisons are not expected according to ICML reviewer guidelines. Nonetheless, we are happy to provide this comparison and share the observations and insights we have gained, in the spirit of contributing to the broader research community.

The results and discussions can be found at https://docs.google.com/document/d/e/2PACX-1vRk1wLF9fsEVPFP2ijxNUd82LPnWZ2w-zTxVNGkkrof9yy36BNrM47JiCr6r8bga9c2Sr3q0a-S0oFF/pub

• Apple to apple performance comparison of r1-zero and Satori:

In order To fairly compare our method with r1-zero, we keep all experimental settings consistent, including the base model, training data, and RL training framework. The only differences lie in the algorithms used, our method employs PPO with RAE, while r1-zero uses GRPO, and whether an additional SFT stage is incorporated (ours includes Format Tuning and r1-zero starts RL training from the base model). We evaluated r1-Zero on the same in-domain and out-of-domain benchmarks, and the results show no advantage over our method. This suggests that the perceived strong performance of the released DeepSeek-r1 model is likely due to significant engineering investment, such as larger-scale in-house data, a much larger base model, and more powerful infrastructure. Moreover, we observed several challenges and unexpected behavior of training r1-zero, which we detail below.

• Training of r1-zero could be unstable:

We found that training r1-zero can be quite unstable. For instance, rewards may drop sharply in later training stages, and the model sometimes begins generating repetitive random sequences. This instability is especially common when the KL penalty is set to zero (i.e., no regularization). We hypothesize that without an SFT stage to anchor the model's behavior, the base model may produce harmful outputs early on, which can lead to a vicious cycle during optimization.

• R1-zero's behavior is hard to control:

Even when training converges, R1-Zero exhibits undesirable behaviors at inference time. Specifically, we observed two representative issues: (1) Repetitive responses, where the model generates repeated tokens or sentences mid-generation. (2) Language mixing, where responses may unexpectedly contain characters from other languages (e.g., Chinese), a phenomenon also noted by the r1 authors. These findings suggest that pure RL fine-tuning lacks model behavior constraints. While more carefully designed reward functions might help, we argue that a simpler and more effective solution is to incorporate a SFT stage, as we did with Format Tuning, to stabilize model’s initial behavior.

• R1-zero shows some "reflection pattern", especially using Python code verification:

We observe that r1-zero indeed demonstrates some reflection behavior after RL training, but we found it tends to use Python code to verify answers, even when this is unnecessary. For example, in a multiple-choice question from the MMLU-Pro dataset (which may not require logical verification), the model attempts to verify the answer using Python, which is a redundant step in that context. This behavior suggests that without appropriate constraints, cold-start RL can introduce unexpected patterns, beyond the desired problem-solving skills. Moreover, a recent study has also questioned the validity of the "Aha moment” claimed by r1-zero and suggests “Aha moment” could be a mirage [1].

• Our final takeaway: SFT warm ups, RL improves.

Our view is that neither pure SFT nor pure RL (as in r1-Zero) is sufficient for effective LLM post-training. Pure SFT can overly constrain the model and limit its ability to generalize beyond demonstrations. Pure RL can lead to unexpected and unreasonable behaviors, as observed with r1-zero. Instead, we advocate for an appropriate combination of both. This is the key insight we propose: a small-scale warm-up SFT helps the base model to get familiar with certain reasoning patterns, and a large-scale RL helps the model to utilize such reasoning patterns to incentivize its actual reasoning capability. This idea is also consistent with practices in the classical RL literature. For example, [2] demonstrates that a small amount of demonstration data can accelerate RL training by improving policy initialization. Moreover, the released DeepSeek-r1 model (as opposed to r1-Zero) also incorporates a SFT stage before RL (though the details of such SFT stage are not disclosed), which aligns with our insight.

Reference

• [1] There May Not be Aha Moment in R1-Zero-like Training — A Pilot Study, Feb, 2025.
• [2] Deep Q-learning from Demonstrations, AAAI, 2018.

1. Ablations and analysis are limited: Some design choices (meta-actions, RAE parameters, reward function) could be more thoroughly justified and explored.

We thank the reviewer for the suggestion. We have conducted additional ablation studies to further analyze our design choices at https://docs.google.com/document/d/e/2PACX-1vQ5lTnuQ5x6bx5Qh87cVdEq5iyrRIMLi5DYVHQUbrqB3f2Gye6mn0bHLwMcVosqddg_wp6P2JdEMfJM/pub.

• Meta-actions: Please refer to rebuttal response 6.
• RAE parameters: We clarify that RAE does not introduce any new tunable hyperparameters. However, to evaluate its impact, we perform an ablation study by training a model without applying RAE. The results show significant performance degradation, confirming that RAE is crucial for RL optimization.
• Reward function: We have already ablated the reflection bonus in Appendix E. In addition, we include another ablation study that removes the preference bonus provided by the outcome reward model. The performance degrades in this setting, indicating that the preference bonus plays an important role in mitigating the sparse reward issues.

2. Analyze common failure modes and how COAT helps address them.

We thank the reviewer for the suggestion. We have conducted a failure mode analysis (see provided link above). Specifically, we find that COAT effectively identifies and mitigates five common failure modes in reasoning tasks: (1) overly complicated solution that misleads the problem-solving process; (2) numeric simplification and calculations error; (3) replacing variables with incorrect numerical values; (4) lack of comprehensive consideration; (5) wrong interpretation of the problem.

3. Benchmark scope is limited.

Since our training data consists of open-sourced math problems, it is reasonable to evaluate the model primarily on math-related benchmarks. However, we would like to emphasize that our evaluation goes beyond the math domain with six OOD datasets logical reasoning, code reasoning, commonsense reasoning, tabular reasoning, and domain-specific reasoning. This type of OOD evaluation is rarely explored in prior works. We would appreciate it if the reviewer could clarify in what sense the benchmark scope is considered limited, and what types of additional evaluation would be most helpful.

4. Evaluation only focuses on final answer accuracy (pass@1).

We would like to note that zero-shot pass@1 is a widely adopted evaluation metric in LLM reasoning literature, provided that the ground-truth final answer is usually available. We agree with the reviewer that analyzing the intermediate reasoning process could provide valuable insights. However, this remains challenging due to the lack of reliable automatic verifiers capable of evaluating intermediate steps at scale. Toward a deeper understanding of our model’s behavior, we have included a diverse set of demo examples in Appendix A that showcase different reasoning patterns of our model.

5. Novelty is more in the combination than in fundamentally new concepts.

While the novelty is the eye of the beholder, we respectfully argue that the contributions of this work go beyond a simple combination of existing ideas. The reviewer describes our approach as a “novel combination of the COAT framework, RAE algorithm, and two-stage training for reasoning.” To the best of our knowledge, none of these components have been introduced in prior LLM reasoning research:

• COAT differs substantially from classical CoT prompting in both design and motivations.
• RAE offers a new perspective (by changing initial state distribution) to tackle the sparse reward problem in RL, a challenge has not been resolved by prior works.
• Our two-stage training pipeline (small-scale Format Tuning + Online RL) also offers new insights for LLM post-training: SFT warm ups, RL improves.

6. Question-1: Have you tried to ablate <|reflect|> and <|explore|> meta-actions separately? What if <|explore|> is not used at all?

We believe that <|reflect|> and <|explore|> are fundamental meta-actions for reasoning tasks, and both are essential for effective problem-solving. Specifically, <|reflect|> prompts the model to evaluate its current reasoning, enabling it to identify potential errors or suboptimal steps, <|explore|> allows the model to propose alternative solutions, which is a natural follow-up when the model realizes (via <|reflect|>) that its current approach may be flawed.

---

If these clarifications satisfactorily address the reviewer's concerns, we kindly ask if the reviewer would consider updating the score to reflect what we believe is a paper with noteworthy contributions to the community.