Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model

We sincerely thank the reviewer for the positive assessment and thoughtful questions about our work. We appreciate the recognition of our framework's value and the insightful analysis on critic and advantage estimation. We address each concern below.

On Reward Hacking Concerns

We appreciate the reviewer's concern about potential reward hacking. However, our experimental setup has taken this into consideration, and greatly reduce the potential space for reward hacking, fundamentally different from RLHF settings where this issue commonly arises:

• We focus exclusively on reasoning tasks with verifiable ground truth answers
• Our reward function is a simple exact-match criterion against these ground truths
• Unlike RLHF with learned reward models that can be exploited, our binary exact-match reward cannot be "gamed" through superficial response modifications The traditional reward hacking in RLHF manifests when models optimize for high reward model scores while producing qualitatively worse outputs. In our setting, achieving higher rewards directly corresponds to solving more problems correctly.

Evidence of Genuine Reasoning Improvement

Beyond the reward and response length curves, we want to share more evidence that ORZ develops genuine reasoning capabilities. First, the model shows consistent performance improvements across a wide range of held-out evaluation sets, indicating true generalized intelligence improvement. We also conducted careful manual inspection of model outputs throughout training. Here's a representative example from ORZ-32B:

• Question: Find the greatest integer value of $b$ for which the expression $\frac{9x^3+4x^2+11x+7}{x^2+bx+8}$ has a domain of all real numbers.
• ORZ-32B's approach (summarized): The model correctly identifies that the domain constraint requires the denominator to never equal zero → formulates this as requiring the discriminant $b^2-32<0$ → solves to get $|b|<\sqrt{32}=4\sqrt{2}\approx 5.656$ → systematically checks boundary cases to conclude $b=5$ is the maximum integer value → performs multiple rounds of self-reflection to verify the answer.

This example illustrates the model's ability to: (i) correctly interpret mathematical constraints, (ii) apply appropriate algebraic techniques, (iii) perform systematic case analysis, and (iv) engage in self-verification. These are hallmarks of genuine mathematical reasoning, not reward exploitation. We will include additional qualitative examples and more detailed analysis in the revised version.

Training Duration for Different Model Scales

The reviewer correctly observes that training steps are different across model sizes. We clarify that this was not a deliberate design choice but rather a practical constraint: experiments on smaller scale models are not of top priority in our plan, and it gets terminated early due to limited computation and time constraints. We are currently resuming experiments to explore longer training horizons for these smaller models and will update the paper with extended training curves.

Response Length Fluctuations in ORZ-32B

This is indeed an interesting phenomenon that we've investigated further since submission. Our additional experiments reveal that learning rate tuning is the key factor:

• When we reduced the 32B model's learning rate from 1e-6 to 5e-7, length growth became much more stable.
• Conversely, increasing the 7B model's learning rate from 1e-6 to 2e-6 induced similar fluctuations. We will include these findings in the appendix of the revised version.

Clarification on Figure 5 Visualization

We apologize for any confusion in the figure caption. We actually use a continuous color map rather than discrete thresholds in text visualization: values closer to 0 appear redder, and values closer to 1 appear greener. And as you can see in the original submission's Figure 5, the critic learns to assign lower values to repetitive/degenerate text and higher values to meaningful reasoning steps. We will revise the caption to make this color mapping logic clearer.

Thanks to the reviewer for the detailed assessment and for flagging areas where broader evidence would help.

Novelty

While the core algorithm is purposefully minimal, the contribution is the first fully open and efficient reproduction of DeepSeek-R1-Zero. Matching its accuracy with 10x fewer steps, dissecting which components are dispensable, and providing a reusable training harness help the field move beyond opaque, resource-intensive recipes. Our pipeline, while retaining the simplicity of vanilla PPO, introduces three scaling-critical components that depart markedly from originally widely-adpoted PPO implementations—and these deviations are uniquely critical for large-scale reasoning-oriented RL.

Extension to domains beyond math and reasoning

Our recipe transfers to new domains (e.g. code, puzzle, instruction-following dialogues) without any modifications. Note that the original submission has already included puzzle-style tasks such as table-join, table-reformat, zebra, and connections alongside math problems. To further demonstrate generalizability, we conducted additional experiments on Qwen-2.5-7B base model with a mixed-domain training distribution (math:code:puzzle:instruction-following = 40%:40%:15%:5%) using the identical ORZ recipe. Training progressed smoothly with consistent increases in both reward and response length. The table below compare benchmark performance between this mixed-domain model and our math-focused model across multiple evaluations, where the final column NAIVE_AVG is benchmark level arithmetic average for all listed benchmarks.

The enlarged task mix delivers substantial gains beyond pure mathematics: IFEVAL, LIVEBENCH, and the logic/code-heavy LCB suites improve sharply, yet the minimalist PPO recipe remains untouched, clear evidence of stronger reasoning transfer. This confirms that our recipe generalizes seamlessly to diverse domains without any algorithmic changes.

Generalization to non-Qwen model families

We successfully applied our ORZ recipe to Llama-3.1-8B base model with minimal adaptation (in the above mentioned mixed domain training setting, only adjusting the learning rate from 1e-6 to 5e-7). Training metrics show stable progression with both reward and response length increasing steadily throughout training, as shown in the table below:

Offline evaluations also show significant performance improvements across benchmarks as table below. The ORZ-Llama3.1-8B model in the table below corresponds to the checkpoint at iteration 500 (iter500) from applying our ORZ recipe to the Llama3.1-8B-Base model. The NAIVE_AVG is benchmark level of arithmetic mean of all listed benchmarks.

Alternative RL algorithms.

We have already benchmarked against GRPO—the most widely used variant in this setting—and found PPO to be both more sample-efficient and more stable. TRPO is seldom applied to LLMs because its second-order updates are computationally expensive, while REINFORCE++, a critic-free method akin to GRPO, shows similar convergence behaviors without offering clear advantages.

We hope these additions clarify the generality of the approach, justify our focus on PPO, and acknowledge its current limits.

Thanks to the reviewer for the encouraging evaluation and for highlighting areas where a deeper analysis could strengthen the paper. We respond to each point below.

General-scene quality

To go beyond reasoning benchmark scores and examine the quality of answer part (not the thinking part) of model output, we now include results on ArenaHard v2, an open-ended dialogue benchmark that exhibits strong correlation and separability with LMArena “judge-vote” rankings. Notably, ORZ-32B outperforms the strong Qwen-2.5-32B-Instruct baseline on ArenaHard v2—by +0.7 % in overall win rate (6.50 % vs 5.81 %) and an even larger +3.4 % in style-control win rate (11.40% vs 7.96%). These margins confirm that ORZ-32B generates generally fluent, well-controlled dialogue without mixed-language artifacts or over-compression in the answer part of model's output.

However, after careful investigation, we find that ORZ-32B develops highly flexible patterns in the thinking part of the model's output. The model sometimes exhibits free-form thinking behaviors including: (1) dynamic use of multiple thinking markers (e.g., nested <think></think> blocks), and (2) spontaneous code-switching between languages when conducting reasoning.

Novelty of the recipe

Our pipeline, while retaining the simplicity of vanilla PPO, introduces three scale-critical components that depart markedly from prior PPO implementations—and these deviations are uniquely critical for large-scale reasoning-oriented RL.

Hallucination and prompt-length sensitivity

We benchmark ORZ-32B on MATHIF, which measures instruction-following in mathematical reasoning. Correctness is reported without and with explicit constraints (w/o const. vs. w/ const.); the final column gives the relative change when constraints are applied. By contrast, all other models in the table lose 4–40% under the same test, showing that ORZ’s alignment maintains and even improves correctness where others falter.

We hope these additions address the reviewer’s questions about broader behaviour, clarify the source of our efficiency gains, and further demonstrate ORZ-32B’s value to the field.

We sincerely thank the reviewer for the positive assessment and valuable feedback. We appreciate the recognition of our work's contribution to the community through open-source methodology and meaningful simplifications. We address each concern below.

Organization and Structure

Thanks for your valuable feedback on the structure and organization of our results. We fully agree that improving the alignment between figures, their explanations, and relevant sections is crucial for readability. We will reorganize the manuscript in the camera-ready version to ensure better flow between related content. Specifically, we will move the explanation of Figure 3 closer to Section 2.2 where the design choices are introduced, eliminating the need for readers to scroll between sections. We will also improve cross-referencing throughout the paper to enhance readability.

Understanding PPO's Superior Advantage Estimation on Repetitive Tokens

We appreciate the reviewer's interest in understanding why PPO achieve better repetitive token advantage estimation. Our analysis suggests the following mechanism: During training, we find that repetitive token sequences reliably signal wrong reasoning paths and failed solutions. The value function quickly learns to give such states low scores, so when the model slips into repetition, the critic usually flags the state with negative advantages and pushes the policy out of the lower value states. Critic-free methods like GRPO lack this salient feedback and miss the cue.

Novelty

While the core algorithm is purposefully minimal, the contribution is the first fully open-source, greatly simplified, efficient reproduction of DeepSeek-R1-Zero. Matching its accuracy with 10× fewer steps, dissecting which components are dispensable, and providing a reusable training harness collectively advance the field's understanding and accessibility of reasoning-oriented RL training.

Full open-source release.

We understand the reviewer's concern about code availability. Unfortunately, NeurIPS 2024 submission guidelines prohibit sharing URLs or de-anonymizing information during the review period. To ensure reproducibility, we commit to releasing our complete codebase, training data, model checkpoints, and detailed reproduction instructions immediately after double-blind review period. Our submission includes a clear reproducibility statement affirming this commitment.

We hope these clarifications address the reviewer's concerns and reinforce the value of our contributions to the community.

Thanks to the reviewer for the careful reading and constructive feedback. We appreciate the acknowledgement of our open-source philosophy and the paper’s practical value, and we address the raised points below.

Extension to domains beyond math and reasoning

Our recipe transfers to new domains (e.g. code, puzzle, instruction-following dialogues) without any modifications. Note that the original submission has already included puzzle-style tasks such as table-join, table-reformat, zebra, and connections alongside math problems. To further demonstrate generalizability, we conducted additional experiments on Qwen-2.5-7B base model with a mixed-domain training distribution (math:code:puzzle:instruction-following = 40%:40%:15%:5%) using the identical ORZ recipe. Training progressed smoothly with consistent increases in both reward and response length. The table below compares benchmark performance between this mixed-domain model and our math-focused model across multiple evaluations.

The enlarged task mix delivers substantial gains beyond pure mathematics: IFEVAL, LIVEBENCH, and the logic/code-heavy LCB suites improve sharply, yet the minimalist PPO recipe remains untouched, clear evidence of stronger reasoning transfer. This confirms that our recipe generalizes seamlessly to diverse domains without any algorithmic changes.

Generalization to non-Qwen model families

We successfully applied our ORZ recipe to Llama-3.1-8B base model with minimal adaptation (in the above mixed-domain training setting, only adjusting the learning rate from 1e-6 to 5e-7). Training metrics show steady increases in both reward and response length, as shown in the table below:

Offline evaluations show consistent performance improvements across benchmarks as table below. The ORZ-Llama3.1-8B model in the table below corresponds to the checkpoint at iteration 500 (iter500) from applying our ORZ recipe to the Llama3.1-8B-Base model. The NAIVE_AVG is benchmark level of arithmetic mean of all listed benchmarks.

Emergent patterns.

We observe two interesting emergent patterns not well documented previously. First, during the final phases of ORZ-32B training, we observe the emergence of a distinctive reflection pattern where the model spontaneously revisits the problem statement from the beginning when encountering difficulties. Unlike local reflection patterns (e.g., correcting a single step), this global revisiting behavior allows the model to escape local minima by fundamentally reconsidering its approach. Second, while our training focuses on mathematical reasoning, ORZ-32B demonstrates unexpected transfer to logical reasoning in unrelated domains. Without explicit training, the model exhibits structured reasoning patterns in tasks such as academic writing, poetry composing, and general domain quesion and answering.

Multilingual data

Our training data includes multilingual content, with approximately 5% Chinese material from CN K12 datasets in Numina.

Theoretical perspective

We acknowledge the reviewer's interest in theoretical foundations. Crafting a rigorous framework that explains why our minimalist recipe scales reinforcement learning so effectively is an important—but substantial—undertaking. A full treatment would require dedicated analysis beyond the present submission, and we view it as a promising direction for future work.

We will include expanded ablation results and full training-curves in the revision. We hope these additions address the reviewer’s concerns and clarify the generality, empirical strength, and transparency of Open-Reasoner-Zero.