A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility

We thank the reviewer for their thoughtful comments and positive evaluation of our work. We appreciate the recognition of the value of our standardized framework and are grateful for the constructive suggestions. Below, we address the specific concerns raised:

Missing Citations: In our revised manuscript, we added a dedicated Related Work section -- adding citations from NLP literature, including the ones suggested by the reviewer alongside from RL literature -- both of which were relevant to this work for thorough coverage.

• We provide a reference image: https://imgur.com/a/l4BcKEy

FastCurl: Thanks for catching this! We agree that FastCuRL similarly shows significant gains on MATH500 and Olympiadbench benchmarks. We should formulate this more nuanced and have revised this in the manuscript as follows:

Only DeepscaleR and FastCuRL demonstrate robust, significant improvements across MATH500 and Olympiadbench benchmarks.

However, they do not show significant improvements across AIME (AIME'24 for DeepScaleR and AIME'25 for FastCuRL), AMC or Minerva unlike the SFT variants in contrast. In this context, we say that a reliable and scalable RL training recipe is still lacking.

Missing Models in Table 1: We agree that aligning the models across Table 1 and Table 3 would improve clarity. We will update Table 1 to include metadata for all models where this information could be reliably recovered. However, in some cases, the original papers do not provide sufficient detail about their evaluation configurations (e.g., decoding parameters or frameworks), which limits our ability to populate Table 1 uniformly. In our best practices, we recommend that future work publish full evaluation configurations to improve reproducibility and comparability.

We sincerely thank the reviewer for their time and positive assessment of our work. We appreciate their recognition of the importance of our findings and their practical value for the community. Below, we address the reviewer’s questions in detail:

1. Statistical Significance Testing: We agree that formal significance testing is a valuable addition. We have updated Table 3 and indicate statistical significance relative to the base model for $p<0.01$ and $p<0.001$ with $^{*}$ and $^{**}$ respectively using a one-sided paired t-test.
   We provide a reference image for showcasing the update: https://imgur.com/a/EIPqfHb

2. Guidelines for Benchmark Construction: We fully agree that small benchmark sizes contribute significantly to the instability of results. While our variance estimates suggest that running evaluations with at least ten random seeds can mitigate this instability, the root issue remains the lack of statistical power due to the small number of test samples. We believe that establishing minimum design criteria -- such as larger sample sizes and more diverse problem distributions -- would benefit the field. That said, we also acknowledge that many widely used benchmarks in reasoning, such as AIME or AMC, are derived from human exams and competitions, which naturally impose size constraints. As models become increasingly capable and data contamination concerns require stricter temporal filtering, it may be necessary to embrace and adapt evaluation protocols to work effectively with these small-scale but high-value benchmarks.

We thank the reviewer for their detailed and constructive feedback. We appreciate the reviewer’s positive assessment of Section 2 and the importance of rigorous evaluation. We agree with the concerns raised and have revised the manuscript accordingly.

Related Work: We have added a dedicated “Related Work” section for more thorough coverage.

• We provide a reference image: https://imgur.com/a/l4BcKEy

Table 3 Formatting: We have separated Table 3 into sections for RL-based/SFT-based models as per request (alongside expanding number of seeds, and using default chat template). This does improve readability, thanks -- providing the clear interpretation: Accuracies of methods can be compared to the base in each group, and the reference improvements traced to RL/SFT algorithms. We removed the highlight as identifying SOTA is not the intention here and might distract.

• We provide a reference image for showcasing the update: https://imgur.com/a/EIPqfHb

We hope referencing this new table will make it far easier to corroborate with the text and judge claims. Thanks for this suggestion!

Other fixes in errors/typos & readability

• Section 3.3 Readability: We made minor edits to Section 3.3 to improve its narrative structure, providing better clarity.
• Line 232: We removed it to avoid confusion. The next paragraph under Overfitting and Generalization already repeats the point.
• Open Reasoner Zero: Thanks, we corrected it to "Open Reasoner-Zero-7B was the only RL-trained model to outperform the instruct-tuned baseline by significant margins across most benchmarks."
• Broken Reference: Thanks! Line 124 refers to Figure 3. Fixed!

We hope we have addressed the major concerns of the reviewer, and are happy to answer any further questions/concerns. We look forward to a fruitful reviewer-author discussion phase.

Thank you for getting back to me, especially on the day of the deadline. I’m fine being overruled by the other reviewers. I’ve updated my score (from 4 to 6, which is the highest I’m willing to go) to acknowledge your responses and efforts during the response period.

But I’ll also say this for the meta-review period: these are all the kinds of details that should be explicitly mentioned in the paper so that the reader can assess if the systems were treated equally, among other things, given how sensitive they can be. Otherwise, it’ll strongly undermine your message.