Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs

This paper investigates the behaviors of two models Qwen2.5-3B and Llama3.2-3B when training with RL on the task of Countdown. It introduces an analysis framework based on four cognitive behaviors. It is found that Qwen naturally exhibits these behaviors, while LLaMA requires behavior priming via finetuning on specifically designed data. Further experiments show that amplifying LLaMA's behaviors in the math domain can also align its behavior development curve with that of Qwen.

The paper investigates why some language models (LMs) improve substantially under reinforcement learning (RL), while others plateau. Focusing on two models—Qwen-2.5-3B and Llama-3.2-3B—the authors identify four cognitive behaviors (verification, backtracking, subgoal setting, and backward chaining) as critical for enabling self-improvement. Qwen naturally exhibits these behaviors, leading to superior performance, while Llama does not unless primed. Through controlled experiments, the study demonstrates that even incorrect solutions containing the right reasoning patterns can boost performance, suggesting that exhibiting these behaviors is more important than correctness. Further, the authors show that pretraining on behavior-rich data enables Llama to match Qwen’s RL-driven improvement, establishing a causal link between initial cognitive behaviors and RL efficacy.

This paper investigates the fundamental mechanisms that enable language models to improve their reasoning capabilities, specifically learn self-improving, through reinforcement learning (RL). The authors focus on two 3B-parameter models, Qwen-2.5-3B and Llama-3.2-3B, for comparison when trained on the Countdown game. The research identifies four key cognitive behaviors critical to effective problem-solving: verification (error-checking), backtracking (abandoning failing approaches), subgoal setting (breaking down complex problems), and backward chaining (reasoning from desired outcomes to initial inputs). Surprisingly, the study finds that Qwen naturally exhibits these behaviors, while Llama does not, leading to significant performance disparities during RL training. Crucially, when Llama was primed with datasets emphasizing these cognitive behaviors or pretrained on OpenWebMath dataset that amplifies such behaviors, its performance approached that of Qwen and presents desired test-time scaling behaviors. This suggests that a model's initial reasoning capabilities fundamentally determine its potential for self-improvement.

This paper dives into a curious difference observed when training language models to improve using reinforcement learning: some models get significantly better, while others just sort of hit a wall. The authors noticed this particularly when comparing Qwen-2.5-3B and Llama-3.2-3B on a math game called Countdown, where Qwen improved dramatically and Llama didn't, even with the same training. They wondered what intrinsic qualities in a model allow for this kind of self-improvement.

We thank all reviewers for their thoughtful and constructive feedback. We are pleased that reviewers found our work "timely" (1iHv), providing "a rigorous ablation study to present a reproducible recipe on how to train a o1-like model" (CcUW), with "strong causal claims" (ZG7V) and “insightful” (UQGC). We're grateful for the reviewers' recognition that our paper is "well written, clearly structured and easy to follow" (1iHv).

In response to the reviewers' valuable suggestions, we will add:

New experiments and analyses:

• Larger model experiments (UQGC, ZG7V, 1iHv): We added results with Llama-3.1-8B and Qwen-2.5-14B models. Results show consistent patterns — Qwen-14b naturally exhibits target behaviors and improves while Llama-8b struggles and plateaus early in training — confirming our findings scale beyond 3B models.

• Instruct model analysis (UQGC, ZG7V, 1iHv): Tested Qwen-2.5-3B-Instruct and Llama-3.2-3B-Instruct to examine how instruction tuning affects cognitive behaviors. Instruct models are better at exploring these behaviors.

• Transfer of RL training on Countdown (1iHv): We demonstrate that cognitive behaviors developed during RL on Countdown transfer to other reasoning domains. After 100 steps of RL training on Countdown, Qwen-2.5-3B shows not only improved task performance (38%→50% with our prompt format, 4-shot 44%→0 shot-50% with Qwen's 4-shot format) but also exhibits amplified cognitive behaviors when evaluated on unrelated tasks. On MATH problems, we observe substantial increases in backtracking (0.15→0.59) and verification (0.02→0.29). Similarly, on GPQA, all four behaviors show meaningful increases: verification (0.008→0.03), backtracking (0.019 →0.114), subgoal setting (0.478→0.616), and backward chaining (0.128→0.313). These results suggest that RL training on Countdown instills reasoning strategies that transfer beyond the training domain.

Discussion on our choice of behaviors (addressing CcUW, ZG7V):

We acknowledge this concern and will expand Section 3.2 to provide principled rationale for our behavior selection:

• Theoretical grounding: These behaviors are well-established in cognitive science literature on expert problem-solving (Newell & Simon, 1971; Polya, 1945). Further, all our behaviors are drawn from classical planning Fikes, R. E., & Nilsson, N. J. (1971).
• Computational necessity: Each behavior addresses a fundamental limitation of linear reasoning: verification (error detection), backtracking (error correction, exploration), subgoal setting (problem decomposition), and backward chaining (goal-directed planning)
• Domain generality: Unlike domain-specific strategies, these behaviors apply across reasoning tasks
• Ease of identification: These behaviors can be reliably identified in text outputs.

We will update our draft to make this clearer.

Benefits and Limitations of Countdown (CcUW, ZG7V):

We chose Countdown as our primary testbed for a few reasons:

First, it provides a controlled environment to test our key hypothesis: what behaviors are necessary for self-improvement with RL? The game presents a challenging search problem with a large branching factor and an inherent look-ahead problem, capturing the complexity of planning tasks while remaining tractable.

Importantly, Countdown offers advantages over seemingly richer domains like MATH or code generation. Current models are often over-optimized for these tasks during pretraining, making it difficult to study post-training learning dynamics. In contrast, models have minimal exposure to Countdown, providing a cleaner experimental setting to observe emergent behaviors during RL. While we acknowledge that verification in Countdown (as pointed out by CcUW) is more straightforward than in open-ended domains, this simplicity is actually advantageous for our core investigation. It allows us to isolate the cognitive behaviors that enable self-improvement i.e., the "aha moment" in learning, without confounding factors from domain knowledge or ambiguous success criteria. The straightforward forward inference steps in Countdown (simple arithmetic) let us clearly observe when and how models develop more sophisticated search strategies.

We will expand the discussion to recognize this focused approach as a limitation and discuss paths toward broader domains in our revised discussion section. However, we believe establishing these fundamental principles in a controlled setting provides essential groundwork for understanding self-improvement in more complex reasoning tasks.

We hope that these revisions help to address the key points raised by the reviewers and strengthen the presentation of our work. Thank you again for the constructive feedback that has helped us to improve the paper.

The paper identifies that certain models are inherently more amenable to improved reasoning via RL finetuning due to the pretraining data or priming. The reviewers generally agree that the work is principled and systematic, providing a rigorous framework and axes on which to evaluate these reasoning capabilities that leads to the clear and convincing conclusions. The authors additionally addressed the concerns relating to two potential limitations (model sizes, categorization of reasoning), with only one (choice of task) which remains somewhat unaddressed. Nonetheless, the reviewers believe this study is already sufficiently well-structured and informative that I recommend acceptance.