Response

Thanks for supporting the acceptance of our work. Please find our responses to your questions below.

Q1: Does the conclusion drawn from the paper still hold when we scale up the model size? some priliminary results would be great enough

Thank you for the insightful question. We also care deeply about whether scaling up model size affects our conclusions. After careful investigation, we found that only three models over 70B parameters have both base and RLVR-trained versions publicly available: DeepSeek-R1-Zero, Tulu-3-70B [1], and Magistral [2]. We summarize them in the table below. For many other large models, isolating the effect of RLVR is not feasible. For example:

• GPT-o1: The base model is not accessible, and training details are unclear; the final model may not rely solely on RLVR.
• Qwen3-235B: The reasoning model undergoes multiple stages, including RLVR and long-CoT SFT, making it impossible to isolate the effect of RLVR alone.
• LLaMA-3-70B: no corresponding RLVR version exists.

*: undisclosed size but reasoning performance close to DeepSeek-R1-671B

For Deepseek-R1-Zero, we have done our best to make it run with limited resources (two 8xH100 GPU nodes). However, without expert engineering optimization, the throughput remains very low, around 50 tokens/s at a maximum sequence length of 32k. Runnings on AIME24 with 1024 samplings would take approximately 4 months, which makes it currently infeasible. We are actively seeking additional resources and hope to resume the process later.

Thus, we turned to Tulu-3-70B and Magistral.

For Tulu-3-70B, which we hosted locally with 8 H100 GPUs, we conducted evaluations on the first 100 subsampled problems from the MATH500 benchmark due to high computational cost. The results below are consistent with our main conclusion.

Tulu3 on Math500

For Magistral, one of the most advanced reasoning models whose. reasoning performance is colse to Deepseek-R1, we queried the API using a maximum context length of 40k. Again, we observed RLVR's great improvements at low k, but little to no gain at larger k. Specifically, at k = 1, the RLVR model solves approximately 7 more problems than the base on AIME24 and 8 more on AIME25. However, as k grows, the performance gap steadily narrows. The trend observed in this preliminary experiment further supports that our conclusion holds even for a frontier-level reasoning model.

Magistral on AIME24

Magistral on AIME25

We will include these results in the revision and explicitly discuss the scalability limitation. We also appreciate your suggestions, which help us strengthen the paper. A more comprehensive evaluation on very large models is currently constrained by compute resources, but this is an important direction for future work.

[1] Allen Institute for AI. "Tulu 3: Pushing frontiers in open language model post-training."

[2] Mistral AI. "Magistral." arXiv:2506.10910.

Q2: how the gap in the pass@k curves changes w.r.t. the difficulty of the evaluation tasks.

Thank you for the thoughtful question. To explore this, we analyzed the MATH500 benchmark, which includes difficulty annotations (Levels 1–5) based on AoPS (Art of Problem Solving) criteria. These levels are based on knowledge domain, solution technique, and complexity. Math500 dataset contains 43, 90, 105, 128, and 134 problems at Levels 1 to 5, respectively. Results are as follows:

level 1

level 2

level 3

level 4

level 5

Preliminary observations:

• At k=1, the gap between base and RLVR increases with difficulty level. Easy problems are often solved by the base on the first try. For harder problems, it's more difficult for the base to sample the correct CoT initially, while RLVR better optimizes sample efficiency. As a result, the crossover point—where the base overtakes RLVR—shifts with difficulty: around k=2 for Level 1, and k=16 for Levels 4–5. These aligns well with your intuition that more difficult tasks require more samples to retrieve correct CoTs from the prior for base model.
• However, at very large k, the gap becomes less correlated with difficulty level. The gap depends on how many problems the base model can eventually solve while the RLVR model cannot, due to reduced output diversity during RLVR training. How this relates to task difficulty is still unclear and warrants further investigation.

We appreciate this suggestion and will include this analysis in the revision.

Q3: Most pass@k curves reported in the paper are the results on some hold-out test sets that are not used for RLVR training right? If so, how would the pass@k curves look like on the training tasks?

Yes, the most pass@k curves are reported on hold-out test sets that were not used during RLVR training.

In Sections 4.3 and 4.4, we present pass@k curves on both training and test sets as fig.14 shows, and we find the trends to be highly consistent to the test set.

To further support this, we subsampled 100 examples from the training set for SimpleRLZoo-7B and plotted the pass@k curves. The results on the training set mirror those on the test set: RLVR outperforms the base model at small k, but is overtaken by the base model at larger k.

Training set

Test set (Math500)

Q4: further add a figure to show how the perplexity score evolves during the RLVR training process

Thank you for the thoughtful suggestion. To analyze how perplexity evolves over the course of RLVR training, we evaluated three RLVR checkpoints—early, middle, and final. For each checkpoint, we sampled 32 responses per problem, computed the median among 32 perplexity values, and reported the average over the first 10 problems in the table (since images are not allowed in rebuttals).

As expected, we observed that $PPL_{base}(Y_{RL})$ gradually decreases as RL training progresses, indicating that RLVR mainly sharpens the distribution within the base model’s prior rather than expanding beyond it.

For $PPL_{RL}(Y_{base})$, we observed the emergence of two clusters over time, consistent with your expectation. One cluster shows a slow decrease in perplexity, corresponding to base samples that remain within RLVR’s sharpened distribution. The other cluster exhibits a rapid increase in perplexity, reflecting base samples that fall outside the RLVR’s sharpened distribution.

Since this dynamic behavior is well aligned with the prediction from our conclusion, we believe this analysis further supports our claims.

conclusion

We hope our responses, especially regarding the first point on scaling to larger model sizes, have addressed your concerns and strengthened your confidence in the paper. If you have any further questions or suggestions, please don’t hesitate to let us know.

Dear authors,

Thanks for your thorough rebuttal! The new results and analysis address most of my concerns. So I'm happy to further raise my score to 6.

One further question that I have is that could the authors share some thoughts on this paper: Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs, which draws an opposite conclusion as yours by using a more fine-grained evaluation metric called CoT-Pass@k. Your reply to this question will not change my score on the submission, as this new paper is online after NeurIPS submission deadline. But I think it would be a very helpful discussion to the community to discuss about these two papers that drawer opposite conclusions.

Thank you sincerely for your timely follow-up and continued support of our work. We have carefully read the paper you mentioned. We believe the difference may stem from several key factors:

1. Verifier Limitations: The CoT-Pass@k metric in that paper relies on an 8B verifier model (DeepSeek-R1-0528-Qwen3-8B), which—despite being distilled—remains relatively weak. Its official report shows error rates of 14.3% (AIME24) and 18.5% (AIME25) on final answers alone, suggesting it may lack sufficient capability for reliably evaluating some benchmark problems. Moreover, smaller models often have limited conceptual understanding and can misjudge issues related to logic, calculation, or reasoning completeness. As a result, the verifier may inaccurately assess CoT correctness in some cases.

2. Verifier Bias: From our prior manual CoT inspections, we found that base models—being only pretrained—often generate “raw” CoTs that, while sometimes containing minor local flaws, still demonstrate correct high-level reasoning and lead to correct final answers. However, based on the verifier prompts provided, the verifier seems to apply very strict criteria and judge such CoTs wrong. In contrast, RLVR outputs tend to be more polished and syntactically aligned, potentially leading to more favorable evaluations—even when the underlying reasoning is not significantly improved. This may introduce bias in favor of RLVR under CoT-Pass@k, when the verifier lacks flexibility.

3. Trends at Larger k: Even using CoT-Pass@k, their results show that base models catch up to or nearly match RLVR at large k. For example, on AMC23 and Minerva, RLVR is fully matched by the base at high k (Fig. 2). On Math500, the performance gap shrinks from ~22% to ~3% at k=128 and continues to close. AIME24 shows a similar narrowing (~33% → ~20%) without saturation. This aligns well with our core observation.

4. Coding Tasks as a Robust Case: In coding tasks—where reasoning must pass unit tests—Pass@k and CoT-Pass@k effectively coincide. There, we still find that base models outperform RLVR at large k, providing a strong, verifier-free sanity check supporting our conclusion.

While we have not yet independently verified the accuracy of the verifier due to time constraints, we believe the above points offer possible explanations for the divergence in conclusions. We plan to conduct a more thorough check of this aspect in future.

Response

Thanks for supporting the acceptance of our work. Please find our responses to your questions below.

W1: no theoretical explanation. The paper includes some intuitive descriptions, such as the repeated reference to base models’ boundaries, but they remain informal.

Thank you for the thoughtful comment. While our paper focuses on empirical findings, we provide an explanation for findings in Appendix C, summarized here:

Current RLVR methods (e.g., PPO, GRPO) are on-policy, meaning they rely on the model to generate its own training samples. Given the vast language space (roughly $V^L \approx 15000^{4096}$, where $V$ is the vocab size and $L$ the avg response length), training from scratch is infeasible because the model would rarely encounter any positive reward under outcome-based reward settings.

As a result, RLVR relies on the pre-trained prior to produce responses likely to earn positive reward. This dependence restricts exploration: most on-policy samples stay within the prior’s region, and every rewarded trajectory further reinforces this behavior through gradient updates. Soft-max sampling can occasionally generate out-of-prior trajectories, but they are exceedingly rare; when they do appear, they are usually invalid and receive negative outcome reward. This feedback loop amplifies the exploration bottleneck.

Interestingly, we find that the findings, and intuitive explanations presented in our paper have been formalized into a theoretical framework in a recent work [1]. It provides theoretical support for the limitation we identify: under both KL-constrained and KL-free RLVR settings, the probability mass assigned by RLVR-trained models outside the base model’s support is theoretically bounded, and this upper bound is typically very small in practice. (Theorem 2.5, Corollary 2.7). This theory work further supports our findings, and the proof approach closely reflects the intuition we provide in Appendix C.

We will clarify this explanation for our findings further in the revision.

[1] The Invisible Leash: Why RLVR May Not Escape Its Origin. 2025 07.

W2: mainly focus on the Qwen model series.

Thank you for the comment. In addition to Qwen models, we also include LLaMA-3.1 and DeepSeek-R1-Distill-Qwen-14B as a starting model for RLVR in our paper.

To further demonstrate the generality of our findings, we additionally conducted experiments on other base models and their RLVR-trained versions, including: Gemma-2-2B [1], Mimo-7B [2], Tulu-3-70B [3].

The results are as follows:

Gemma-2-2B on Math500

Mimo-7B on Math500

Tulu-70B on Math500

For Tulu-3-70B, we evaluate on the first 100 problems of MATH500 due to the high computational cost.

*: Tulu-3-RLVR is RL-trained based on Tulu-3-SFT.

These results further support the robustness and broader applicability of our conclusions across model families.

[1] Gemma 2: Improving Open Language Models at a Practical Size.

[2] MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining

[3] Tulu 3: Pushing Frontiers in Open Language Model Post-Training

W3: The evaluation is too centered on pass@k curves. I would appreciate more analyses like the one in the “Perplexity Analysis” paragraph.

We appreciate your recognition of the perplexity analyses beyond pass@k. In our paper, we also include accuracy distributions, reasoning coverage, and comparisons with distilled models, intended to provide a more comprehensive view of RLVR’s capabilities and limitations.

During the rebuttal, we further expanded the perplexity analysis, adding the dynamics of perplexity throughout training, which will be included in the revision (see details in our response to Reviewer KdU6’s Q4 for details if you are interested).

Additionally, we plan to incorporate more plausible explanations and potential solutions for RLVR's limitations to deepen the discussion. We believe these additions will strengthen the paper beyond pass@k curves and help enrich understanding of the findings. Thank you again for your constructive suggestion.

Q1: What are the specific policy optimization parameters used during RLVF? For example, what values did you set for ppo_mini_batch_size and ppo_epochs in VeRL?

Thank you for the question. For broad experiments, we use publicly released checkpoints from existing RLVR works such as SimpleRLZoo, DAPO, OAT, and DeepCoder. The RL training hyperparameters for these models can be found in their respective papers.

For our in-depth analysis, we trained RLVR models ourselves using the following settings:

This results in 8 gradient updates per rollout step, calculated as: (Prompt batch size × Rollout number × PPO epochs) / PPO mini-batch size.

Q2: Why does the "Perplexity Analysis" paragraph only analyze two samples from AIME?

Due to space limitations, we included only two examples in the paper as representative cases—the remaining problems exhibit similar patterns. In fact, we conducted perplexity analysis on all 30 AIME problems, and in most cases, the base model assigns lower perplexity to RLVR responses than to its own, and other cases comparable.

We will include the full set of figures in the appendix of the revision.

conclusion

We hope our responses have addressed your concerns and strengthened confidence in our paper. If you have any further questions, please feel free to reach out.

Thank you for the rebuttal, appreciate your effort. In addition to those 'more experiments' stuff to support the claims, I would also like to revisit the core claim of the paper itself.

I believe that similar insights have been discussed in earlier works. For example, Section 5.2.2 of DeepSeekMath [1], released in February 2024, also employed similar phrasing, such as “fundamental capabilities.” I suspect there may be other works that made similar observations even earlier, though perhaps not explored this specific point as thoroughly as in your paper.

Do the authors have any comments on the potential minor novelty in the main claim?

[1] Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv:2402.03300, 2024.

We sincerely thank you for your time, thoughtful feedback, and continued support. Regarding the concern about potential overlap with DeepSeekMath, we would like to respectfully clarify our contribution.

Since the release of DeepSeek-R1 and its demonstration of “aha moments,” RLVR has received considerable attention, with a widely held belief that it could enable LLMs to develop novel reasoning abilities and support autonomous self-improvement. While sec 5.2.2 of DeepSeekMath offered the early experiment, its analysis was limited to the pass@k curve of a single 7B instruction-tuned model, two benchmarks (GSM8k and math), and one RL method (GRPO). We believe such a narrow setup and limited discussion is insufficient to rigorously address the broader and important question of whether current RLVR methods truly expand reasoning capability coverage. Indeed, despite Deepseek-math's earlier release, the belief in RLVR's effectiveness of expanding new capacity remained widespread in the community.

To the best of our knowledge, our paper is the first to formally pose this question and provide a large-scale, systematic evaluation across diverse settings to answer the critical question. As noted in Lines 347–352, our contributions include:

• Problem Framing: Clearly formulating the question of whether current RLVR methods expand reasoning capability coverage, challenging a widely held belief and offering insight for the community.
• Breadth: Evaluation across multiple model families (Qwen, LLaMA, Olmo, Mistral), sizes (7B–32B, including frontier models like Magistral-Medium), and RL algorithms (PPO, GRPO, RLOO, Reinforce++), covering various reasoning benchmarks.
• Novel Depth Analysis: Beyond pass@k, we analyze perplexity dynamics, accuracy distributions, entropy and KL ablations, and comparisons with distilled models.

As Reviewer 79qN summarized:

While the main claim has been observed already in prior literature as the authors acknowledged, this paper makes a significant contribution by providing a thorough analysis and stronger evidential basis... across a large number of training domains, models, and RL algorithms.

Athough DeepSeekMath surfaced a relevant experiment early on, our work is the first to formally frame and systematically answer this important question. We will clarify this more clearly in the revision.

As for the concern that “there may be other works with similar observations,” we have carefully investigated and, aside from the brief mention in Section 5.2.2 of DeepSeekMath, we are not aware of any prior work that directly and comprehensively studies the reasoning capacity coverage relationship between RLVR and base models. As acknowledged in our related work section, one rare relevant paper is the concurrent work [1], which focuses on how RLVR struggles to recover the pass@k drop introduced by SFT. However, it does not directly analyze or systematically compare base vs. RLVR reasoning capabilities as our paper does.

We hope this helps clarify our distinction from DeepSeekMath. Thank you again for your constructive engagement.

[1] Dang, Xingyu, et al. "Assessing diversity collapse in reasoning." March, 2025.

Response

Thanks for supporting the acceptance of our work. Please find our responses to your questions below.

W1: Limited theoretical insight: Lacks explanation for why RLVR fails to expand reasoning boundaries

Thank you for the thoughtful comment. While our paper focuses on empirical findings, we provide an intuitive explanation in Appendix C, summarized here:

Current RLVR methods (e.g., PPO, GRPO) are on-policy, meaning they rely on the model to generate its own training samples. Given the vast language space (roughly $V^L \approx 150000^{4096}$, where $V$ is the vocab size and $L$ the avg response length), training from scratch is infeasible because the model would rarely encounter any positive reward under outcome-based reward settings.

As a result, RLVR relies on the pre-trained prior to produce responses likely to earn positive reward. This dependence restricts exploration: most on-policy samples stay within the prior’s region, and every rewarded trajectory further reinforces this behavior through gradient updates. Soft-max sampling can occasionally generate out-of-prior trajectories, but they are exceedingly rare; when they do appear, they are usually invalid and receive negative outcome reward. This feedback loop amplifies the exploration bottleneck.

Interestingly, we find that the findings, and intuitive explanations presented in our paper have been formalized into a theoretical framework in a recent work [1]. It provides theoretical support for the limitation we identify: under both KL-constrained and KL-free RLVR settings, the probability mass assigned by RLVR-trained models outside the base model’s support is theoretically bounded, and this upper bound is typically very small in practice. (Theorem 2.5, Corollary 2.7). This theory work further supports our findings, and the proof approach closely reflects the intuition we provide in Appendix C.

We will clarify this explanation for our findings further in the revision.

[1] The Invisible Leash: Why RLVR May Not Escape Its Origin. 2025 07.

W2&Q2: Narrow scope: Findings may not generalize to future RL paradigms or different reward structures; Could different reward structures (e.g., process rewards vs. outcome rewards) change the conclusions about RLVR's effectiveness?

Thank you for the insightful question. As stated in the abstract and introduction, our study primarily aims to clarify the limitations of current RLVR.

By identifying what today’s RLVR can and crucially cannot achieve, we hope to motivate future RL research toward unlocking genuinely novel reasoning abilities.

Process-level rewards are indeed a promising direction for addressing current RLVR's limitation: in principle, assigning credit to intermediate reasoning steps could help alleviate the exploration inefficiency. However, current process rewards tend to be noisy and vulnerable to hacking, which is why most practical implementations still rely on outcome-based rewards—where the limitations we expose remain significant.

History has shown that RL can elicit new capabilities (e.g., AlphaZero in Go). However, RLVR for LLM faces a different set of challenges, most notably, the exponentially larger action space and the strong influence of the pretrained prior, as discussed in detail in Appendix C. We believe that with further advances such as improved process rewards, RL may eventually overcome these limitations (we elaborate on potential directions in the next question). However, before that, clearly identifying the current gap, as our paper does, is a critical first step toward that goal.

W3: Missing solutions: Identifies problems clearly but provides limited improvement suggestions;

Thank you for the comment. Pinpointing the problem is only the first step; finding viable remedies is equally important for advancing LLM reasoning models. We see several promising research directions:

1. Finer-grained reward structures: step-wise rewards guide intermediate reasoning and reduce exploration bottlenecks.
2. Improved exploration: Instead of naive softmax sampling, introduce structured or hierarchical search to enhance exploration efficiency.
3. Better long-horizon credit assignment: Use techniques to propagate reward more effectively over long CoT chains and enabling the model to assign credit to crucial intermediate steps instead the whole response
4. Scaling up RL training: Match RLVR compute and data scale to that of pre-training
5. Multi-turn tool use & external knowledge: Allow the agent to interact with tools or retrieve external facts, broadening the reasoning space beyond single-pass generation

While Appendix C already hints at several possible solutions, we will make these directions clearer and more concrete in the revision. Ultimately, addressing this limitation will require collective effort, and we hope our paper and the directions outlined above can help catalyze progress in this important area.

W4: Base model dependency: Conclusions may be influenced by specific base models tested

Thank you for the comment. In our paper, we include Qwen, LLaMA-3.1 and DeepSeek-R1-Distill-Qwen-14B as a starting model for RLVR.

To further demonstrate the generality of our findings, we additionally conducted experiments on other base models and their RLVR-trained versions, including: Gemma-2-2B [1], Mimo-7B [2], Tulu-3-70B [3].

The results are as follows:

Gemma-2-2B on Math500

Mimo-7B on Math500

Tulu-70B on Math500

For Tulu-3-70B, we evaluate on the first 100 problems of MATH500 due to the high computational cost.

*: Tulu-3-RLVR is RL-trained based on Tulu-3-SFT.

These results further support the robustness and broader applicability of our conclusions across model families.

[1] Gemma 2: Improving Open Language Models at a Practical Size.

[2] MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining

[3] Tulu 3: Pushing Frontiers in Open Language Model Post-Training

Q1: How do the findings extend to larger base models (e.g., 70B+ parameters)?

Thank you for the insightful comment. We also care deeply about whether scaling up model size affects our conclusions. During the rebuttal, we conducted extensive investigations and experiments on this topic, detailed in our response to Reviewer 79qN’s Weakness 1 (W1).

Our results show that the findings hold even for 70B+ models and state-of-the-art frontier models. Due to space constraints and the absence of a global response section, we kindly refer you to that response for the full discussion.

conclusion

We hope our responses have addressed your concerns and strengthened confidence in our paper. If you have any further questions, please feel free to reach out.

Response

Thanks for supporting the acceptance of our work. Please find our responses to your questions below.

W1: The biggest limitation of the paper is that results are all on relatively small models (7-32B parameters) and low training data diversity; Q1: Did you run any experiments on the models closer to the frontier?

Thank you for the insightful comment. We also care deeply about whether scaling up model size affects our conclusions.

After careful investigation, we found that only three models over 70B parameters have both base and RLVR-trained versions publicly available: DeepSeek-R1-Zero, Tulu-3-70B [1], and Magistral [2]. We summarize them in the table below.

For many other large models, isolating the effect of RLVR is not feasible. For example:

• GPT-o1: The base model is not accessible, and training details are unclear; the final model may not rely solely on RLVR.
• Qwen3-235B: The reasoning model undergoes multiple stages, including RLVR and long-CoT SFT, impossible to isolate the effect of RLVR.
• LLaMA-3-70B: no corresponding RLVR version exists. | Model Pair | Size | API Access | Weights Access | | - | - | - | - | | DeepSeek-V3-Base / R1-Zero | 671B | ✗ | ✓ | | Tulu-3-SFT / Tulu-3-RLVR | 70B | ✗ | ✓ | | Mistral-Medium-3 / Magistral-Medium | undisclosed* | ✓ | ✗ |

*: undisclosed size but reasoning performance close to DeepSeek-R1-671B

For Deepseek-R1-Zero, we have done our best to make it run with limited resources (two 8xH100 GPU nodes). However, without expert engineering optimization, the throughput remains very low, around 50 tokens/s at a maximum sequence length of 32k. Runnings on AIME24 with 1024 samplings would take approximately 4 months, which makes it currently infeasible. We are actively seeking additional resources and hope to resume the process later.

Thus, we selected Tulu-3-70B and Magistral.

For Tulu-3-70B, which we hosted locally with 8 H100 GPUs, we conducted evaluations on the first 100 subsampled problems from the MATH500 benchmark due to high computational cost. The results below are consistent with our main conclusion.

Tulu3 on Math500

For Magistral, one of the most advanced reasoning models, we queried the API using a maximum context length of 40k. Again, we observed the RLVR model has great improvements at low k, but little to no gain at larger k. Specifically, at k = 1, the RLVR model solves approximately 7 more problems than the base on AIME24 and 8 more on AIME25. However, as k grows, the performance gap steadily narrows. The trend observed in this preliminary experiment further supports that our conclusion holds even for a frontier-level reasoning model.

Magistral on AIME24

Magistral on AIME25

We will include these results in the revision and explicitly discuss the scalability limitation. A more comprehensive evaluation on very large models is currently constrained by resources, but this is an important direction for future work.

[1] Allen Institute for AI. "Tulu 3: Pushing frontiers in open language model post-training."

[2] Mistral AI. "Magistral." arXiv:2506.10910.

W2: Random guessing explanation: Example of something that would be more assuring: report % of trajectories containing flawed CoTs, use these measurements to estimate the bound on performance attributable to random guessing

Thanks for your suggestion. If we understand correctly, by "% of trajectories containing flawed CoTs," you’re referring to the proportion of incorrect reasoning paths across multiple samples per problem. While this is a useful metric, it serves a different purpose from our analysis.

As stated in Section 2.2, our goal is to assess the coverage of model capacity—specifically, whether the model can generate at least one correct CoT for a given problem. This is why we manually check whether at least one correct CoT appears per problem. In contrast, measuring the percentage of flawed CoTs aligns more with average-case evaluation, which, while valuable, does not directly capture capacity coverage.

To further address concerns about random guessing, we refer to the coding task in Section 3.2, where success requires passing unit tests—making random guessing highly unlikely. The same trends hold in that setting, which supports our conclusions and helps rule out random guessing as an explanation.

W3: Over-fitting to test set

Thank you for raising this point. If we understand correctly, by "overfitting to the test set," you are referring to potential test data leakage during RLVR training. We will add an explicit discussion to clarify in revision, as follows:

For the RLVR checkpoints in the main experiments in Section 3, the RL training data is publicly documented and does not overlap with the evaluation benchmarks. Further, in our deep analysis (e.g., RLVR train with OmniMath data), we further ensure a clean train/test split, so there is no data leakage.

If we have misunderstood your concern, please let us know—we would be happy to clarify further.

Q2: do you have any experiments with SFT pre-training, does it modify the conclusions?

If we understand correctly, you're asking whether our conclusions hold when RLVR is applied to models already fine-tuned via SFT or distillation. If it's yes, we have included such experiments in our paper

In coding task in Section 3.2, we evaluated DeepCoder-14B, which is trained using RLVR on top of DeepSeek-R1-Distill-Qwen-14B, an SFT model based on the pretrained Qwen-2.5-14B. As shown in Figure 3 in the paper and summarized below, as k increases, base models consistently catch up to and eventually surpass RL-trained models:

To further support our conclusion on SFT models, we add results on math task and will add it later in revision. Here, the RLVR model is trained on DeepSeek-R1-Distill-Qwen-1.5B (an SFT model from Qwen-2.5-1.5B). Evaluated on AIME24. Similar trends are observed.

In summary, these results reinforce our main findings, even when RLVR is applied starting from an SFT model.

Q3: The term "reasoning capacity/boundary" seems like a bit of a misnomer; shouldn't it be solution-space/reasoning coverage or scope?

Thank you for pointing this out. You're right that our use of terms like reasoning capacity/boundary may have been somewhat inconsistent and potentially confusing. "Reasoning coverage" or "reasoning scope" more accurately reflect the concept we intended. We will revise the terminology in the revision.

Minors

Thank you for your careful review and valuable suggestions. We address each point below and will incorporate the corresponding changes in the revision.

• Table 1: We currently omit training set details due to space limitations. In the revision, we will adjust the layout to make the training data clearly identifiable for each experiment.
• Figure 6: The comparison with the perplexity of o1 responses serves as a baseline, illustrating how "surprising" responses from other reasoning models (like o1) appear to the base. We will add a sentence to clarify this in the revision.
• Figure 8: The original purpose of using a folded y-axis was to highlight the ordering differences between algorithms at both k=1 and k=256. However, we acknowledge the readability concerns of fig.8, and we will replace it with a more reader-friendly version in the revision.
• Figure 16 (AIME24): Our choice of T=0.6 follows the evaluation protocol of DeepSeek-R1 and provides stable performance across benchmarks. We found empirically that the optimal temperature for RLVR model varies by task (e.g., higher T benefits AIME24, while lower T helps AMC23), and T=0.6 offers a reasonable overall trade-off. As shown in Figure 17, we also experimented with increasing T for RLVR. While this improves performance on AIME24, on most other benchmarks it performs comparably to RLVR at T=0.6. Even on AIME24, RLVR with higher T still lags behind the base model at large k, reinforcing our main conclusion.
• Tables 4 & 5: We appreciate your suggestion and will revise the tables accordingly. Specifically, we will revise the tables to show the fraction of problems in four categories: (1) both models solve, (2) only base solves, (3) only RLVR solves, and (4) neither solves. Results are as follows:

The results highlight key observations:

The overlap in correct solutions (type 1) is dominant. There are many cases where the base model solves a problem but RLVR fails (type 2), and very few cases where RLVR succeeds while the base does not (type 3). Even in rare type 3 cases (e.g., 1% or 5 problems in MATH500), when sampling 1024 times, the base model can solve all of them.

This supports our findings that RLVR rarely solve problems the base cannot, and generally achieve lower pass@k at large k, indicating reduced coverage.

conclusion

We hope our responses have addressed your concerns and strengthened confidence in our paper. Should there be any further questions, feel free to reach out.